JP5768172B2 - Image display control method and apparatus, and image pickup apparatus - Google Patents

Description

  The present invention relates to an image display control method and apparatus for displaying a wide-angle image such as a panoramic image and a 360-degree omnidirectional image, and an image capturing apparatus.

  For example, the following Patent Documents 1 and 2 describe a camera system in which a plurality of cameras can be installed in a radial manner to capture 360-degree omnidirectional images. With such a camera system, a panoramic image with an ultra-wide angle can be taken. However, a display device that can display this panoramic image at a time becomes large in size, and it is not easy to install in a camera system.

  Some cameras can take panoramic images with a wide angle while panning with a single digital camera. However, panoramic images cannot be displayed at a time on a small display screen with an aspect ratio of 4: 3 mounted on a small digital camera.

  Therefore, as described in, for example, Patent Documents 3 and 4 below, the image portion to be displayed is manually scrolled in the panoramic image, and the partial portions of all the images are sequentially displayed on a small screen.

  On the other hand, in recent digital cameras, a face detection function is usually installed. As described in the following Patent Documents 5, 6, and 7, "Face" ”Is detected, and when there are a plurality of faces, only the faces are displayed in order or each face is enlarged and displayed.

Japanese Patent Laid-Open No. 11-98342 JP 2004-61808 A JP 2000-101991 A JP-A-11-146243 JP 2008-28955 A JP 2007-295183 A JP 2007-318393 A

  Display screen with an aspect ratio of 4: 3 when a camera system that can shoot a super-wide-angle panoramic image on a surveillance camera, fixed-point camera, etc., or when an individual shoots a pan-wide-angle panoramic image while panning one camera It is not easy to view a panoramic image on the screen, and it is troublesome to scroll the screen manually.

  Therefore, it is conceivable to apply the face detection function described in Patent Documents 5, 6, and 7 to detect “faces” in the panoramic image and display only the faces in order or zoom in on the faces. .

  However, the panoramic image contains a lot of information, and if only the face is selected and displayed, information other than the face may be lost and important information may be missed. For example, when a camera system that captures a panoramic image is applied to a surveillance camera, even if an intruder's face is known, an image of the intrusion location is lost.

  An object of the present invention is to miss important information when viewing a panoramic image (in this specification, an image having an aspect ratio larger than the aspect ratio of a display image displayed on the display screen is called a panoramic image) on a small screen. It is an object of the present invention to provide an image display control method and apparatus and an image pickup apparatus that can view a panoramic image without any problem.

  The image display control method of the present invention is an image display control method in which the display image is cut out from a panorama image having an aspect ratio larger than the aspect ratio of the display image displayed on the display screen, and is displayed on the display screen. Detecting a main subject in the panoramic image, determining a moving route for moving the video frame from one end side to the other end of the panoramic image, and moving the video frame along the moving route The display image cut out by a video frame is displayed on the display screen, and the video frame is reduced when the video frame moving along the moving route approaches the position of the main subject, thereby reducing the main subject. An enlarged image of the main subject is reduced by enlarging the image frame and enlarging the video frame after the enlarged display, and an initial size of the video frame is determined in advance. Together, and determining the size of the video frame for each of the main subject at the time of the larger image.

  The image display control apparatus of the present invention is an image display control apparatus that cuts out the display image from a panorama image having an aspect ratio larger than the aspect ratio of the display image displayed on the display screen, and displays the image on the display screen. Subject detection means for detecting a main subject in the panoramic image, movement route determination means for determining a movement route for moving the video frame from one end side to the other end side of the panoramic image, and along the movement route Scroll display means for displaying the display image cut out in the video frame while moving the video frame on the display screen, and the video frame moving along the moving route approaches the position of the main subject. The image of the main subject is enlarged by reducing the image frame, and the image frame of the main subject is enlarged by enlarging the image frame after the enlarged display. Video frame size changing means for reducing and displaying a large display image, wherein the video frame size changing means predetermines an initial size of the video frame and sets the size of the video frame when the enlarged display is performed. It is determined for each main subject.

  An image pickup apparatus according to the present invention includes the above-described image display control apparatus.

  According to the present invention, the panoramic image itself can be viewed and monitored without missing the main subject in the panoramic image.

It is a functional block block diagram of the wide-angle image imaging device which concerns on one Embodiment of this invention. It is a functional block block diagram of the wide-angle image imaging device which concerns on another embodiment of this invention. It is a functional block block diagram of the wide-angle image imaging device which concerns on another embodiment of this invention. It is a functional block block diagram of the wide-angle image imaging device which concerns on another embodiment of this invention. It is a table block diagram of the recording means used with the wide angle image pick-up device shown in FIG. It is explanatory drawing of the wide-angle (panoramic) image display control method which concerns on one Embodiment of this invention. It is a flowchart which shows the process sequence of the 1st step of the wide-angle image display control method which concerns on embodiment of this invention. It is a figure explaining the specific example of the to-be-photographed object detection process of FIG. It is a flowchart which shows the process sequence of the 2nd step of the wide-angle image display control method which concerns on embodiment of this invention. It is a specific explanatory view of a wide-angle image display control method according to an embodiment of the present invention. It is explanatory drawing of the zoom-up display and zoom-out display of the main subject which concern on another embodiment of this invention. It is a flowchart which shows the process sequence which performs scroll display of a panoramic image, performing zoom-up display shown in FIG. FIG. 3 is an explanatory diagram of a movement route (basic route) of a panoramic image and automatic scroll display by the wide-angle image pickup device shown in FIG. 2. is there. It is correction explanatory drawing which concerns on another embodiment of the movement route which replaces FIG. It is a flowchart which shows the correction process procedure of the movement route shown in FIG.

  Hereinafter, an embodiment of the present invention will be described with reference to the drawings. Hereinafter, the image display control device of the present invention is referred to as a wide-angle image display control device, the image display control method of the present invention is referred to as a wide-angle image display control method, and the image imaging device of the present invention is referred to as a wide-angle image imaging device. Also say.

  FIG. 1 is a functional block diagram of a wide-angle image capturing apparatus according to an embodiment of the present invention. This wide-angle image pickup device 1 is a recording medium that uses a single camera (an image pickup unit including a shooting optical system and a solid-state image pickup device, or a digital signal processing of a picked-up image signal acquired by the solid-state image pickup device to obtain a JPEG image or the like. (Hereinafter, the same applies) 2), a rotating pan head 3 on which the camera 2 is mounted and the camera 2 is rotated, for example, by 360 degrees, and a photographing instruction means comprising a CPU and the like. 4, a rotation amount control unit 5 that controls the rotation amount of the rotary head 3 according to an instruction from the imaging instruction unit 4, and a rotary head drive that rotates the rotary head 3 in response to a command from the rotation amount control unit 5. Means 6, a camera locus recording means 7 for recording the movement locus of the camera in response to a command output from the rotation amount control means 5 to the rotary head drive means 6, and the position of the image taken by the camera 2 is calculated from the camera locus. Image position calculating means 8 , And a photographing control unit 9 to perform photographing control of the camera 2 based on an instruction from the imaging instruction means 4.

  The wide-angle image capturing apparatus 1 further includes an image compositing unit 10 that captures image data captured by the camera 2, for example, JPEG images, and combines a plurality of captured images to synthesize a panoramic image, and a main subject in the panoramic image. Subject detection means 11 to detect, movement route determination means 12 for determining a “movement route” to be described later based on the detection result of the subject detection means 11 and the calculation result of the image position calculation means 8, and the main subject on the panoramic image Trimming area determining means 13 for determining the detected position as a trimming target area (area to be zoomed up), and display in the panoramic image based on the determination result of the moving route determining means 12 and the determination result of the trimming area determining means 13 Automatic image scrolling means 14 for automatically scrolling a video frame to be performed at a predetermined speed; And an image display unit 15 such as a liquid crystal display device for displaying an image.

  In the wide-angle image pickup apparatus 1 having such a configuration, if the shooting angle of view of the camera 2 is 45 degrees, for example, the rotation amount control means 5 rotates the rotating pan head 3 of the camera 2 by 45 degrees, for a total of 8 images. The image synthesizing unit 10 combines the eight images to synthesize a 360 degree panoramic image, and the panoramic image is displayed while being automatically scrolled on the image display unit 15 by the display control of the image automatic scroll unit 14. To do. Details of the automatic scroll display will be described later.

  FIG. 2 is a functional block configuration diagram of a wide-angle image capturing apparatus according to another embodiment of the present invention. The wide-angle image capturing apparatus 20 is an image capturing apparatus that synthesizes a panoramic image while the user holds the camera 21 by hand and moves the camera 21. The image capturing instruction unit 4, the camera locus recording unit 7 described in FIG. Image position calculation means 8, photographing control means 9, image composition means 10, main subject detection means 11, movement route determination means 12, trimming area determination means 13, image automatic scroll means 14, and image display means 15 is the same as the wide-angle imaging device 1 of FIG.

  However, in the wide-angle image capturing apparatus 1 of FIG. 1, the camera 2 is installed on the rotary head 3 and is driven to rotate, whereas the wide-angle image capturing apparatus 20 of the present embodiment uses the handheld camera 21, and thus moves. In order to detect the locus, the difference is that the motion vector calculating means 22 is provided.

  The motion vector calculation means 22 calculates a motion vector by a well-known method, cuts out a part of a certain frame image captured in a moving image state, and where this part of the image part exists in the next frame image. The amount of movement and the direction of movement are calculated as a motion vector. Based on the calculated motion vector, the camera locus recording means 7 records the camera locus.

  The motion vector calculation means 22 obtains the motion vector in software by image matching processing. For example, the camera 21 is equipped with an electronic compass or an acceleration sensor, and the movement direction and movement angle of the camera 21 are detected by hardware. Then, the motion vector may be calculated from this.

  FIG. 3 is a functional block configuration diagram of a wide-angle imaging device according to still another embodiment of the present invention. In this wide-angle image pickup device 25, a hyperbolic mirror 27 is arranged in front of the photographing lens of the camera 26 so that an image of 360 degrees around can be obtained in one shot.

  The wide-angle image pickup device 25 also includes the shooting instruction unit 4, the shooting control unit 9, the main subject detection unit 11, the movement route determination unit 12, the trimming area determination unit 13, and the image automatic scroll unit described with reference to FIG. 1. 14 and the image display means 15 are the same as the wide-angle image pickup apparatus 1 of FIG. 1, but the camera locus recording means 7 and the camera are moved in order to take a 360-degree panoramic image in one shot. There is no image position calculation means 8 that causes blurring.

  However, in order to take a distorted image with the hyperbolic mirror 27, an image deforming means 28 that eliminates the distortion of the distorted image is required, and the main subject detecting means 11 takes in an image whose distortion has been corrected by the image deforming means 28, Detect the main subject.

  FIG. 4 is a functional block configuration diagram of a wide-angle image capturing device according to still another embodiment of the present invention. This wide-angle image pickup device 30 is different from the wide-angle image pickup device of FIGS. 1 to 3 in that an omnidirectional image pickup camera 31 in which a plurality of cameras are arranged radially is used, and the shooting instruction means 4 described in FIG. An image position calculation unit 8, a shooting control unit 9, an image synthesis unit 10 that combines the captured images of the cameras to synthesize a panoramic image, a main subject detection unit 11, a movement route determination unit 12, and a trimming The area determination unit 13, the automatic image scrolling unit 14, and the image display unit 15 are the same as the wide-angle image capturing apparatus 1 in FIG. 1.

  Since this wide-angle image pickup device 30 uses a plurality of cameras, instead of the camera locus recording means 7 of FIG. 1, means 32 for recording the arrangement conditions of each camera and means 33 for recording the photographing conditions of each camera; The outputs of these means 32 and 33 are output to the image position calculating means 8.

  FIG. 5 is a diagram showing an example of a data table stored in the camera arrangement condition recording unit 32 and the camera photographing condition recording unit 33 described in FIG. The rotation angle position with respect to the reference position of each camera is stored as an arrangement condition, and the focal length of each camera is stored as an imaging condition.

  As described above, four different wide-angle image capturing devices 1, 20, 25, and 30 have been described. However, in order to display a panoramic image photographed and synthesized by each wide-angle image capturing device on the small display means 15, a device is devised. Necessary. Even if, for example, only a person is cut out from a vast panoramic image as shown in Patent Documents 5, 6, and 7, and sequentially displayed on the display unit 15, most of the information of the panoramic image having a huge amount of information is lost. End up.

  FIG. 6A is a simple explanatory diagram of a panoramic image display control method according to an embodiment of the present invention. This panoramic image display control is processed by, for example, the movement route determination means 12, the trimming area determination means 13, and the automatic image scrolling means 14 shown in FIGS.

  In the panoramic image 41 shown in FIG. 6A, it is assumed that the main subject “A” and the main subject “B” are shown at positions separated from each other. Even if only the images “A” and “B” are cut out from the panoramic image 41 and displayed in an enlarged manner, the positions and directions where “A” and “B” exist, what is reflected in the background, the main subject Information such as what kind of image is displayed next to the image is lost.

  Therefore, in the present embodiment, first, the movement route 42 from the start position (the left end side in FIG. 6A) to the end position (the right end side in FIG. 6A) of the panoramic image 41 is obtained. The moving route 42 in the illustrated example has a route that passes through the main subject “A” and then passes through the main subject “B” as the moving route.

  The main subjects “A” and “B” are detected by the subject detection unit 11 described with reference to FIGS. 1 to 4, and subject frames 43A and 43B are determined in accordance with the sizes of the main subjects “A” and “B”. .

  Each of the rectangular frames 44 in FIG. 6A indicates a video frame. An image in the rectangular frame 44 is cut out from the panoramic image 41, and an image in the rectangular frame 44 is displayed on the display unit 15. It is displayed fully in the frame. That is, the rectangular frame 44 becomes an image trimming region, and when the rectangular frame 44 becomes smaller, the image inside is enlarged and displayed.

  First, an image at the start point position is cut out and displayed. The rectangular frame 44 at this time uses a large rectangular frame set as the initial size, and increases the amount of information of the display image.

  The rectangular frame 44 of the initial size shown in FIG. 6A has a maximum size that makes the vertical dimension the same as the vertical dimension of the panoramic image. However, if the rectangular frame 44 of the initial size is made too large, if the main subject is present near the upper and lower edges of the panoramic image, it can be moved along the moving route 42 with the rectangular frame 44 of the initial size remaining unchanged. It will disappear.

  In such a case, the rectangular frame 44 is reduced and moved. For example, as shown in FIG. 6B, the initial-sized rectangular frame 44 may be made somewhat small. When a panoramic image capturing camera system is used as a fixed point camera, a surveillance camera, or the like, most of the background image is a known image, so there is no problem even if the initial size is small.

  When the rectangular frame 44 having the initial size is moved along the movement route 42, the rectangular frame 44 is automatically moved at a predetermined speed, and an image cut out from the panoramic image 41 by the rectangular frame 44 after the movement is displayed. That is, the display image is automatically scrolled. By providing many overlapping portions between the display image and the next display image, a person looking at the screen can easily recognize the moving direction of the image.

  In this way, the partial image of the panoramic image 41 is automatically scroll-displayed, and when approaching the main subject “A”, the rectangular frame 44 from which the image is gradually cut out is made smaller. Thereby, the image is gradually enlarged and displayed.

  When the scroll advances and the center of the moved rectangular frame 44 coincides with the center of the subject frame 43A of the main subject “A”, the subject frame 43A is used as the rectangular frame 44. In FIG. 6A, the subject frame 43A is shown separated from the left and right rectangular frames 44, but this is for avoiding difficulty in understanding the drawing. Needless to say, it should be overlapped with the subsequent rectangular frame 44.

  When the image to be displayed moves away from the main subject “A”, the rectangular frame 44 is gradually enlarged again, and when approaching the main subject “B”, the rectangular frame 44 is reduced, and the main subject “B” The display is enlarged at 43B. As the scroll advances and moves away from the main subject “B”, the rectangular frame 44 is enlarged and displayed to the end of the panoramic image 41.

  In this way, when displaying the panoramic image 41 while automatically scrolling on a small display screen, the image of the portion where the main subject does not exist increases the amount of information such as a background image as an image including a wide range, and the closer the main subject is, the closer the main subject is. Zoom in so that you don't miss the main subject. Thereby, it is possible to automatically display the panorama image 41 on a small screen without leaving the information of the panorama image 41 and without overlooking the main subject. In addition, the screen is scrolled and the screen is zoomed in gradually when approaching the main subject, and then zoomed out, so there is no sudden screen change and the viewer feels uncomfortable. Nothing will happen.

  FIG. 7 is a flowchart showing a processing procedure in the first stage of the display control method for executing the automatic scroll display explained in FIG. First, in step S1 of FIG. 7, a panoramic image is obtained. In the case of the wide-angle image capturing apparatus 1 in FIG. 1, the images taken in each direction are combined into one panoramic image. In the case of the wide-angle image capturing apparatus 20 of FIG. 2, a panoramic image is synthesized by superimposing overlapping portions of the images captured as the camera 21 moves. With the wide-angle image capturing device 25 of FIG. 3, a panoramic image without distortion is generated by performing image conversion processing on the distorted image. In the case of the wide-angle image capturing device 30 in FIG. 4, panoramic images are synthesized by pasting together captured images of a plurality of cameras.

  In the next step S2, the main subject in the panoramic image is extracted. For example, when the face of a person or animal is the main subject, whether or not the face exists is detected by face image pattern matching, and if it exists, the coordinate position is detected. Even when a building or the like is a main subject, the shape may be detected by pattern matching. It is up to the user to select what subject is the main subject.

  In the next step S3, a trimming area for each main subject is calculated. In the example described with reference to FIG. 6, since the sizes of the main subjects “A” and “B” are different, trimming areas (subject frames 43A and 43B in FIG. 6) in which the main subjects are inscribed are calculated.

  FIG. 8 is a table showing an example of the subject detection result when the main subject is “face”. Five main subjects are detected, the coordinates of the trimming area (subject frame) are calculated based on the respective position coordinates, and the size of the trimming area is calculated based on the respective sizes.

  In the illustrated example, an evaluation value as a “face” image is obtained. The evaluation value is the probability of the subject as the main subject, and the face image is evaluated as the maximum value “1”. Further, in FIG. 8, a determination result as to whether or not it is a priority target is entered. Even if it is a face image, it can be identified by the pattern whether it is a human face, a dog face, or a cat face, so if the user sets the priority target to human Then, it is determined whether the detected main subject is a human or an animal.

  In the next step S4 in FIG. 7, the movement route is calculated. The moving route is calculated as a route that passes through all the detected main subjects, for example. Alternatively, referring to the table of FIG. 8, the detection evaluation value is calculated as a route passing through a main subject having a user-specified or default predetermined value (for example, 0.7) or more, or a person who is a priority subject among main subjects is selected. Calculated as the route taken.

  In the next step S5, it is determined whether or not there are a plurality of moving routes. When there are a plurality of main subjects, there may be a plurality of calculated movement routes. Therefore, if there are a plurality of travel routes, that is, if the determination result in step S5 is affirmative (Yes), the process proceeds to step S6, and the maximum route evaluation value among the route evaluation values of each travel route is selected as the optimum route. The process of FIG. 7 is terminated.

  In step S6, for example, when each of the plurality of moving routes is not a moving route that passes through all of the main subjects, the total value of the detection evaluation values shown in FIG. 8 for the main subjects on the route is used as the route evaluation value. . In addition, when there are a plurality of travel routes, a configuration may be adopted in which candidates for the travel route are presented to the user and the user is allowed to select.

  If there is one moving route, that is, if the determination result in step S5 is negative (No), the process in FIG. 7 is terminated in step S5.

FIG. 9 is a flowchart showing a processing procedure in the second stage of the display control method following FIG. First, in step S11, the trimming frame (rectangular frame 44 in FIG. 6) is set to the initial size (H ₀ , W ₀ ). In the next step S12, a start coordinate position (X ₀ , Y ₀ ) for starting scroll display is set. In the next step S13, the trimming area coordinates (x _n , y _n ) and the trimming area size (h _n , w _n ) of the first main subject in the moving route are set. In the next step S14, the image at the start coordinate position of the panoramic image is cut out with the trimming frame of the initial size and displayed on the display screen.

  In the next step S15, it is determined whether or not the coordinate position of the current trimming frame is the end coordinate position of the panoramic image. If the coordinate position is the end coordinate position, the scroll display is ended.

When the coordinate position of the current trimming frame is not the end coordinate position, the process proceeds to step S16, and the trimming frame is moved along the movement route. In the next step S17, a distance Rn to the next main subject is calculated. In the case of this embodiment, calculation is performed using only the X coordinate, but the distance may be calculated in consideration of the Y coordinate. In the next step S18, the distance _{R n} is equal to or a predetermined distance _{T 1} (e.g., 500 pixels) or less.

Current trimming frame is present in a predetermined distance T ₁ within a distance from the main subject when (the result of the determination in step S18 is affirmative (Yes)), the main subject of the size of the trimming frame the current (H, W) in step S19 According to the trimming area size and the distance to the subject.

That is,
H = h _n + (H−h _n ) * R _n / T ₁
W = w _n + (W−w _n ) * R _n / T ₁
And

  Thereafter, in step S20, the trimming area size is actually changed, and the process returns to step S14 to display the inside of the trimming frame.

If the determination result in step S18 is negative (No), the process proceeds to step S21 to determine whether the coordinates are the same as or passed through the main subject. If the result of this determination is affirmative (Yes), the process proceeds to step S22, where the current trimming frame size is set to the trimming area size (h _n , w _n ) of the main subject. Further, since the main subject has been passed, the trimming area coordinates (x _{n + 1} , y _{n + 1} ) and trimming area size (h _{n + 1} , w _{n + 1} ) of the next main subject are set in the next step S23. Thereafter, the process proceeds to step S20, where the trimming area size is actually changed, and the inside of the trimming frame is displayed in step S14.

  If the determination result of step S21 is negative (No), the process proceeds to step S24, where it is determined whether there is a subject that has passed so far. If the determination result is affirmative (Yes), it is further determined in step S25 whether or not the distance Rn-1 to the subject that has passed immediately before is within a predetermined distance T2. If the determination result is affirmative (Yes), the object is moved away from the subject, so that the current trimming frame size (H, W) is enlarged according to the initial size of the trimming region and the distance to the subject, contrary to step S19. (Step S26).

That is,
H = H + (H ₀ −H) * | R _n−1 | / T ₂
W = W + (W ₀ −W) * | R _n−1 | / T ₂
And

  Thereafter, the process proceeds to step S20 to actually change the trimming area size and display the trimming frame.

  If the determination results in steps S24 and S25 are negative (No), the process returns to step S14 without changing the size of the trimming area, and the trimming frame is displayed as it is.

  As described above, according to the present embodiment, the trimming frame is gradually reduced when approaching the main subject and the image of the main subject is zoomed up, and the trimming frame is enlarged when moving away from the main subject. The image of the main subject is zoomed out and a panoramic image at a position outside the main subject is displayed with a trimming frame of the initial size.

  In FIG. 6, the reduction of the video frame (trimming frame) 44 is started at a position considerably away from the main subject A so that the present embodiment can be easily understood. If this is the case, the background image portion of the main subject A may not be displayed in the scroll display before and after the main subject A is zoomed up.

  However, in the present embodiment, such a concern does not actually occur. In recent years, solid-state imaging devices mounted on digital cameras have been increased in number of pixels, and it has become common to mount 10 million pixels or more. For example, in the case of a solid-state imaging device equipped with 12 million pixels, the captured image is 4000 pixels wide × 3000 pixels long.

  Therefore, if the reduction of the video frame 44 is started from a position where the center of the video frame 44 and the center of the subject frame are separated by 500 pixels, the enlarged display is not performed until then. That is, since the enlarged display starts from the state in which the image of the main subject A in the video frame 44 of the initial size is scrolled to about 1/3 of the display screen, the background image of the main subject A until then is displayed. Will enter the video frame 44 of the initial size.

  In the present embodiment, when the automatic scroll display position reaches the end coordinate position, the process ends in step S15. When the end coordinate position is reached, the automatic scroll display from the start coordinate position is started again. As long as there is no end instruction from a user, it is good also as a structure which continues displaying by an infinite loop.

Further, although the predetermined distance T ₁ (step S18) when the trimming frame approaches the main subject and the predetermined distance T ₂ (step S25) when leaving the trimming frame are described as different distances, they may be the same. When leaving, the zoom-up display of the main subject has already been completed, so steps S24 to S26 may be omitted, and after the zoom-up display, the trimming frame may be suddenly set to the initial size.

  Furthermore, although the moving speed of the trimming frame 44 has been described as being constant, the moving speed of the trimming frame 44 may be slowed as it approaches the main subject and returned to the original speed as it moves away. In this way, the image display is easier to see, and the main subject can be observed carefully.

  FIG. 10 is a diagram illustrating a main subject to be enlarged and a moving route in an actual panoramic image. The table of the main subject explained in FIG. 8 corresponds. At the start coordinate position of the panoramic image, the image is cut out and displayed by a rectangular frame (trimming frame) 44 having a large initial size, and this rectangular frame 44 is moved along the moving route 42 to automatically scroll the display.

  In the illustrated panoramic image, there are five main subjects, and the subject frames 43A, 43B, 43C, 43D, and 43E of the respective main subjects are detected by the subject detection unit 11 of FIGS. When scrolling the panoramic image, the trimming frame 44 is moved along the moving route 42 while the panoramic image is cut out and displayed by the initial size trimming frame 44, and the first dog is zoomed by the subject frame 43A. The image is displayed up (= zoom in), and when the dog is separated from the dog, the trimming frame 44 is enlarged and moved.

  Next, when entering a region where there are many main subjects, the face of the first child is zoomed up and displayed in the subject frame 43B in the order shown along the moving route 42 in the example shown in the figure. Subsequently, the parent face is zoomed up in the subject frame 43C, the second child's face is zoomed up in the subject frame 43D, and the second dog face is then zoomed in the subject frame 43E. Zoom up display. When moving away from the subject frame 43E, the trimming frame 44 is returned to the initial size and moved. When the panoramic image end position is reached, the scroll display is ended.

  Such scroll display is possible by processing according to the flowchart of FIG. However, scroll display in an embodiment different from the embodiment of FIG. 9 is also possible.

  FIG. 11 is an explanatory diagram of zoom-up display and zoom-out display according to another embodiment of the present invention. In the processing procedure of FIG. 9, when the main subject is approached, the trimming frame is moved and reduced in parallel, but in this embodiment, the trimming frame 44 is moved and reduced separately. ing.

  That is, in this embodiment, the panoramic image is cut out and displayed with the initial size trimming frame 44 while moving the initial size trimming frame 44 (image a). When the main subject image in the image cut out by the trimming frame 44 of the initial size comes to substantially the center (image b), the main subject is gradually enlarged and displayed while reducing the trimming frame for cutting out the main subject.

  After the main subject is displayed in the subject frame 43A (image c), the trimming frame is enlarged and zoomed out (image d), the trimming frame is returned to the initial size, and then the trimming frame is moved (image). e).

  In this way, it is possible to perform zoom-up display for each main subject individually and to scroll the entire panoramic image on the display screen without leaving the entire panoramic image.

FIG. 12 is a flowchart showing a scroll display processing procedure for performing the zoom-up display described in FIG. First, in step S31, the trimming frame is set to an initial size (H ₀ , W ₀ ). In the next step S32, a start coordinate position (X ₀ , Y ₀ ) for starting scroll display is set. In the next step S33, the trimming area coordinates (x _n , y _n ) and trimming area size (h _n , w _n ) of the first main subject in the moving route are set. In the next step S34, an image at the start coordinate position of the panoramic image is cut out with the trimming frame of the initial size and displayed on the display screen.

In the next step S35, it is determined whether or not the coordinate position of the current trimming frame is the end coordinate position of the panoramic image. If the coordinate position is the end coordinate position, the scroll display is ended. When the coordinate position of the current trimming frame is not the end coordinate position, the process proceeds to step S36, and the trimming frame is moved along the movement route. In the next step S37, it calculates the distance R _n to the next main subject. In this embodiment, although to calculate the distance R _n only X-coordinate may calculate the distance in consideration of the Y coordinate. Distance _{R n} at the next step S38 is equal to or a predetermined distance _{T 3} (e.g., 5 pixels) or less.

If the determination result of step S38 is affirmative (Yes), the size of the trimming frame is reduced by a predetermined reduction rate α (step S39). That is,
H = H * α
W = W * α
And If the determination result of step S38 is negative (No), the process returns to step S34.

In step S40 following step S39, it is determined whether or not the reduced trimming frame size is larger than the trimming area size (h _n , w _n ) of the main subject. If the determination result is affirmative (Yes), the size of the trimming frame is changed in step S41, and the reduced trimming frame is temporarily displayed in the next step S42, and then the process returns to step S39 again to continue the reduction process.

If the determination result in step S40 is negative (No), the trimming frame size is set to (h _n , w _n ) in step S43, the trimming frame size is changed in step S44, and the trimming frame size is set in step S45. Is displayed. Thereafter, in step S46, the process waits for a predetermined time t seconds.

  When the wait ends in step S46, the trimming frame enlargement process is performed in step S47. This enlargement process enlarges the size of the trimming frame by a predetermined enlargement ratio β.

That is,
H = H * β
W = W * β
And

In the next step S48, it is determined whether the size of the enlarged trimming frame is smaller than the initial size (H ₀ , W ₀ ). If it is smaller (judgment result is affirmative (Yes)), the size of the trimming frame is changed in step S49, the image in the trimming frame is once displayed in step S50, and the process returns to step S47 again to enlarge or display the trimming frame Perform image reduction processing.

If the determination result in step S48 is negative (No), that is, if the trimming frame size is not smaller than the initial size, the trimming frame size is set to the initial size (H ₀ , W ₀ ) in the next step S51, and trimming is performed in step S52. The frame size is actually changed, and the inside of the trimming frame is displayed in step S53.

Thereafter, in step S54, the trimming area coordinates (x _{n + 1} , y _{n + 1} ) and trimming area size (h _{n + 1} , w _{n + 1} ) of the next subject in the movement route are set, and the process returns to step 35.

  FIG. 13 is a diagram illustrating an example of a panoramic image captured by the wide-angle image capturing device 20 of FIG. Since the camera 21 is panned (moved) by hand, the panoramic image obtained by combining the individual images does not become a beautiful strip with a long side. The line indicated by reference numeral 50 in FIG. 13 is the movement locus of the camera 21, and the relative positional relationship between the individual images before being combined and synthesized is calculated by the motion vector, and is recorded by the camera locus recording means 7 in FIG. It is a movement trajectory. The subject frames 43A, 43B, 43C, 43D, and 43E are subject frames of the main subjects.

  The panoramic image taken by the wide-angle image pickup device 30 in FIG. 4 is also the same, and even when the individual picked-up images of the respective cameras are pasted together, it may not be a beautiful strip shape, and the upper edge and the lower edge are aligned. As a result, an unnecessary region (blacked portion in the figure) 51 is generated. The moving route 50 in such a case is obtained from the relative positional relationship between the individual captured images, that is, the positional relationship between the cameras.

  The unnecessary area 51 in the composite image shown in FIG. 13 should not be shown at the time of image reproduction. For this reason, an initial size trimming frame 44 that does not show the unnecessary area 51 as much as possible is determined. Then, the movement route of the trimming frame 44 of this initial size is determined along the camera movement locus 50.

  In this case, a positional deviation between the center coordinates of the main subject and the movement route 50 occurs. FIG. 14 is a diagram illustrating an embodiment of the correction route 60. In the present embodiment, a circle K having a predetermined radius d is obtained with respect to the center coordinates of each main subject. Then, it is determined whether or not the distance between the adjacent circles K is smaller than 2d. When the distance exceeds 2d, a corrected route is obtained for each main subject based on the moving route 50 in FIG. A route connecting the centers of the main subjects is set as a correction route. The dimension “d” may be set so that the trimming area size of the main subject is just within the circle K, for example.

  FIG. 15 is a flowchart showing the moving route correction processing procedure shown in FIG. 14, which is the detailed procedure of step S4 in FIG. In the present embodiment, first, in step S61, the camera movement locus is set as a basic route of the movement route. In the next step S62, a circle K having a radius d centered on the center coordinates of each main subject is obtained. In the next step S63, as the order of route correction (bypass processing), it is defined that processing is performed from a subject close to the start coordinate in the basic route.

  In step S64, it is determined whether or not there is an intersection between the basic route and the circle K. When there is no intersection (determination result is negative (No)), no bypass route is provided for the subject. If the determination result is affirmative (Yes), it is further determined in step S65 whether or not a straight line connecting the center coordinates of the subjects has two or more intersections with the circle K.

  This is synonymous with determining whether or not the subjects are very close to each other. If the determination result in step S65 is affirmative (Yes), the process proceeds to step S66, and a straight line connecting the center coordinates of the subjects is set as a bypass route (between the second and third main subjects in FIG. 14, third and fourth). Between the second main subject, and between the fourth and fifth main subjects).

  If the determination result in step S65 is negative (No), the process proceeds to step S67, and a straight line connecting the intersection of the basic route and the circle and the center coordinate of the subject is used as a bypass route (first main subject, second main subject in FIG. 14). Refer to the subject).

  After step S66 or step S67, the process proceeds to step S68, where it is determined whether or not the determination in step S64 has been performed for all main subjects, that is, all the circles K. If the determination in step S64 has not been completed for all main subjects. Returning to step S64, if the processing is completed, the processing in FIG. 15 is terminated.

  By scroll-displaying according to the procedure of the flowchart of FIG. 9 along the movement route 60 obtained in this way, the entire panoramic image display and the zoom-up display of the main subject can be made compatible. It becomes possible to prevent oversight of important information.

  FIG. 14 shows the panoramic image display control by the scroll display processing described in the flowchart of FIG. 9, but the entire panorama image display and the zoom-up display of each main subject are also performed by the procedure of the flowchart described in FIG. Both can be achieved. In this case, there is no need to obtain the correction route, and the trimming frame 44 of the initial size may be moved along the camera movement locus 50 (basic route) shown in FIG.

  In the above-described embodiment, all the main subjects on the panoramic image are enlarged or displayed, or the main subject having the specified characteristics is enlarged, but the main subject that is a predetermined distance away from the moving route is ignored. Only the main subject that is not more than a predetermined distance from the moving route may be enlarged and displayed.

  Furthermore, the above-mentioned movement route is calculated by considering not only a plurality of factors and the position of the main subject but also the relative positional relationship of individual images before synthesis, each image position, camera movement locus, etc. The calculation accuracy is improved, there is no useless movement when the automatic scroll display is performed, and the scroll display is easy to see.

  In the above-described embodiments, the wide-angle image capturing apparatus has been described as an embodiment in which a panoramic image is displayed on its own image display unit 15. However, the panoramic image is taken out to an external personal computer, a digital photo frame, etc. The present invention can be similarly applied to display a panoramic image by executing the above display control.

As described above, the following items are disclosed in this specification.
(1)
An image display control apparatus that cuts out the display image from a panoramic image having a larger aspect ratio than the aspect ratio of the display image displayed on the display screen, and displays the image on the display screen.
Subject detection means for detecting a main subject in the panoramic image; movement route determination means for determining a movement route for moving the video frame from one end side to the other end side of the panoramic image; and Scroll display means for displaying the display image cut out in the video frame while moving the video frame on the display screen, and when the video frame moving along the moving route approaches the position of the main subject Image frame size changing means for enlarging and displaying the image of the main subject by reducing the video frame and reducing the enlarged display image of the main subject by enlarging the video frame after the enlarged display;
An image display control device, wherein the video frame size changing means determines an initial size of the video frame in advance and determines a size of the video frame for the enlarged display for each main subject.
(2)
(1) The image display control apparatus according to (1), wherein the subject detection unit detects a position and a size of a main subject by performing matching processing with a main subject pattern prepared in advance.
(3)
The image display control device according to (1) or (2), wherein the subject detection unit detects a face as the main subject.
(4)
The image display control device according to any one of (1) to (3), wherein the movement route determination unit determines a line connecting the positions of main subjects having specified characteristics as a movement route. apparatus.
(5)
(1) The image display control device according to any one of (4), wherein when there are a plurality of travel routes determined by the travel route determination means, the travel route candidates are displayed on the display screen to the user. An image display control device to be selected.
(6)
The image display control device according to any one of (1) to (4), wherein the moving route determination means indicates an evaluation value indicating the main subject likeness of the main subject on each moving route when there are a plurality of moving routes. An image display control device that selects a movement route that maximizes the sum of the two.
(7)
The image display control device according to any one of (1) to (6), wherein the moving speed of the video frame is slowed when the video frame is within a predetermined distance from the main subject.
(8)
The image display control device according to any one of (1) to (6), further comprising an image composition unit that combines a plurality of images to compose the panorama image, and the movement route determination unit includes the panorama image An image display control apparatus that determines the movement route from a relative positional relationship between individual images before image synthesis.
(9)
(8) The image display control device according to (8), wherein the relative positional relationship between the individual images is obtained from the movement trajectory of the camera that captured the individual images.
(10)
The image display control device according to (9), wherein the movement trajectory is obtained from a motion vector between the captured images.
(11)
(8) The image display control device according to (8), wherein the relative positional relationship between the individual images is obtained from the relative positional relationship between a plurality of cameras that individually photographed the individual images.
(12)
(8) The image display control device according to any one of (11), wherein a main subject within a predetermined distance is given priority among the main subjects separated from the moving route, and a main subject separated from the predetermined distance. An image display control device that sets the video frame while ignoring a subject.
(13)
(8) The image display control device according to any one of (12) to (12), wherein when the main subject exists within a predetermined distance from the movement route obtained from the relative positional relationship of the individual images, the main subject is selected. An image display control device that corrects the travel route with a correction route that passes through.
(14)
(1) The imaging device which mounts the image display control apparatus in any one of (7).
(15)
An image display control method for cutting out the display image from a panoramic image having an aspect ratio larger than the aspect ratio of the display image to be displayed on the display screen, and displaying the image on the display screen.
A main subject in the panoramic image is detected, a moving route for moving the video frame from one end side to the other end side of the panoramic image is determined, and the video frame is moved while moving the video frame along the moving route. The display image clipped in step (b) is displayed on the display screen, and when the video frame moving along the movement route approaches the position of the main subject, the image of the main subject is reduced by reducing the video frame. The enlarged image is enlarged and the image frame is enlarged after the enlarged display, the enlarged display image of the main subject is reduced, the initial size of the image frame is determined in advance, and the image frame for the enlarged display is displayed. An image display control method for determining the size of each main subject.
(16)
(15) The image display control method according to (15), wherein the main subject pattern prepared in advance is subjected to matching processing to detect the position and size of the main subject.
(17)
(15) The image display control method according to (16), wherein a face is detected as the main subject.
(18)
(15) The image display control method according to any one of (17) to (17), wherein a line connecting the positions of main subjects having specified characteristics is determined as a movement route.
(19)
(15) The image display control method according to any one of (18) to (18), wherein when there are a plurality of moving routes, the moving route candidates are displayed on the display screen and selected by the user.
(20)
(15) The image display control method according to any one of (18), wherein when there are a plurality of movement routes, the movement is such that the sum of evaluation values indicating the main subject-likeness of the main subject on each movement route is maximized. An image display control method for selecting a route.
(21)
(15) The image display control method according to any one of (20) to (20), wherein the moving speed of the video frame is slowed when the video frame is within a predetermined distance from the main subject.
(22)
(15) The image display control method according to any one of (20), wherein the panorama image is synthesized by combining a plurality of images, and the relative positional relationship between the individual images before the panorama image is synthesized. An image display control method for determining the movement route.
(23)
(22) The image display control method according to (22), wherein the relative positional relationship between the individual images is obtained from the movement trajectory of the camera that captured the individual images.
(24)
(The image display control method according to 23, wherein the movement trajectory is obtained from a motion vector between the captured individual images.
(25)
(The image display control method according to claim 22, wherein the relative positional relationship between the individual images is obtained from the relative positional relationship between a plurality of cameras that individually photograph the individual images.
(26)
The image display control method according to any one of (22) to (25), wherein a main subject within a predetermined distance is prioritized among the main subjects that are separated from the moving route, and a main subject that is separated from the predetermined distance. An image display control method for setting the video frame while ignoring a subject.
(27)
(22) The image display control method according to any one of (26) to (26), wherein when a main subject exists within a predetermined distance from a movement route obtained from the relative positional relationship of the individual images, the main subject is selected. An image display control method for correcting the travel route with a correction route to be passed.

  The image display control method according to the present invention can be mounted as a display function of an imaging apparatus that can display a panoramic image without missing important information when the panoramic image is displayed on a screen smaller than the panoramic image. This is useful when panoramic images are captured and displayed on a personal computer or workstation.

1, 20, 25, 30 Wide-angle image pickup device 2, 21, 26, 31 Camera (digital camera)
3 rotating pan head 4 photographing instruction means 5 rotation amount controlling means 6 rotating pan head driving means 7 camera locus recording means 8 image position calculating means 9 photographing control means 10 image synthesizing means 11 main subject detecting means 12 moving route determining means 13 trimming area Determination means 14 Automatic image scrolling means 15 Image display means 22 Motion vector calculation means 27 Hyperbolic mirror 28 Image deformation means 41 Composite panoramic image 42 Moving route 43 Rectangular frame (video frame)
50 Camera movement path (movement route)
55 Relocated travel routes 56, 57, 58, 59, 60 Correction route steps S15, S17 Image frame size changing means

Claims (27)

An image display control apparatus that cuts out the display image from a panoramic image having a larger aspect ratio than the aspect ratio of the display image displayed on the display screen, and displays the image on the display screen.
Subject detection means for detecting a main subject in the panoramic image; movement route determination means for determining a movement route for moving the video frame from one end side to the other end side of the panoramic image; and Scroll display means for displaying the display image cut out in the video frame while moving the video frame on the display screen, and when the video frame moving along the moving route approaches the position of the main subject Image frame size changing means for enlarging and displaying the image of the main subject by reducing the video frame and reducing the enlarged display image of the main subject by enlarging the video frame after the enlarged display;
An image display control device, wherein the video frame size changing means determines an initial size of the video frame in advance and determines a size of the video frame for the enlarged display for each main subject.   The image display control device according to claim 1, wherein the subject detection unit detects a position and a size of the main subject by performing matching processing with a main subject pattern prepared in advance.   The image display control device according to claim 1, wherein the subject detection unit detects a face as the main subject.   4. The image display control device according to claim 1, wherein the movement route determination unit determines a line connecting the positions of main subjects having specified characteristics as a movement route. 5. apparatus.   5. The image display control apparatus according to claim 1, wherein when there are a plurality of travel routes determined by the travel route determination means, the travel route candidates are displayed on the display screen to the user. An image display control device to be selected.   5. The image display control device according to claim 1, wherein the moving route determination unit indicates an evaluation value indicating the main subject likeness of the main subject on each moving route when there are a plurality of moving routes. 6. An image display control device that selects a movement route that maximizes the sum of the two.   7. The image display control device according to claim 1, wherein the moving speed of the video frame is slowed when the video frame is within a predetermined distance from the main subject.   7. The image display control apparatus according to claim 1, further comprising an image composition unit that combines a plurality of images to compose the panorama image, and the movement route determination unit includes the panorama image. An image display control apparatus that determines the movement route from a relative positional relationship between individual images before image synthesis.   The image display control apparatus according to claim 8, wherein the relative positional relationship between the individual images is obtained from a movement trajectory of a camera that captured each image.   The image display control device according to claim 9, wherein the movement trajectory is obtained from a motion vector between the captured images.   9. The image display control device according to claim 8, wherein the relative positional relationship between the individual images is obtained from the relative positional relationship between a plurality of cameras that individually photographed the individual images.   12. The image display control device according to claim 8, wherein a main subject within a predetermined distance is given priority among the main subjects separated from the moving route, and a main subject separated from the predetermined distance. An image display control device that sets the video frame while ignoring a subject.   13. The image display control device according to claim 8, wherein when a main subject exists within a predetermined distance from a moving route obtained from a relative positional relationship between the individual images, the main subject is selected. An image display control device that corrects the travel route with a correction route that passes through.   An image capturing apparatus equipped with the image display control apparatus according to claim 1. An image display control method for cutting out the display image from a panoramic image having an aspect ratio larger than the aspect ratio of the display image to be displayed on the display screen, and displaying the image on the display screen.
A main subject in the panoramic image is detected, a moving route for moving the video frame from one end side to the other end side of the panoramic image is determined, and the video frame is moved while moving the video frame along the moving route. The display image clipped in step (b) is displayed on the display screen, and when the video frame moving along the movement route approaches the position of the main subject, the image of the main subject is reduced by reducing the video frame. The enlarged image is enlarged and the image frame is enlarged after the enlarged display, the enlarged display image of the main subject is reduced, the initial size of the image frame is determined in advance, and the image frame for the enlarged display is displayed. An image display control method for determining the size of each main subject.   16. The image display control method according to claim 15, wherein matching processing is performed with a main subject pattern prepared in advance to detect a position and a size of the main subject.   The image display control method according to claim 15 or 16, wherein a face is detected as the main subject.   The image display control method according to any one of claims 15 to 17, wherein a line connecting positions of main subjects having specified characteristics is determined as a movement route.   The image display control method according to any one of claims 15 to 18, wherein when there are a plurality of moving routes, the moving route candidates are displayed on the display screen and selected by the user.   The image display control method according to any one of claims 15 to 18, wherein when there are a plurality of movement routes, the movement is such that the sum of evaluation values indicating the main subject-likeness of the main subject on each movement route is maximized. An image display control method for selecting a route.   21. The image display control method according to claim 15, wherein the moving speed of the video frame is slowed when the video frame is within a predetermined distance from the main subject.   21. The image display control method according to claim 15, wherein the panoramic image is synthesized by pasting a plurality of images, and the relative positional relationship between the individual images before the panoramic image is synthesized. An image display control method for determining the movement route.   23. The image display control method according to claim 22, wherein the relative positional relationship between the individual images is obtained from a movement trajectory of a camera that captured each image.   24. The image display control method according to claim 23, wherein the movement locus is obtained from a motion vector between the captured images.   23. The image display control method according to claim 22, wherein the relative positional relationship between the individual images is obtained from the relative positional relationship between a plurality of cameras that individually photograph the individual images.   The image display control method according to any one of claims 22 to 25, wherein a main subject within a predetermined distance is given priority among the main subjects separated from the moving route, and a main subject separated from the predetermined distance. An image display control method for setting the video frame while ignoring a subject.   27. The image display control method according to claim 22, wherein when a main subject is present within a predetermined distance from a movement route obtained from a relative positional relationship between the individual images, the main subject is selected. An image display control method for correcting the travel route with a correction route to be passed.

Images