JP7073092B2 - Image processing equipment, image processing methods and programs - Google Patents

Description

The present invention relates to a technique for setting parameters related to a virtual viewpoint.

There is a virtual viewpoint image technology as a technique for reproducing an image from a camera (virtual camera) that does not actually exist and is virtually arranged in a three-dimensional space by using images taken by a plurality of real cameras. In the generation of the virtual viewpoint image, there is a region where the image from the virtual camera cannot be reproduced due to the blind spot between the images taken by the real camera. Further, when the virtual camera is closer to a person or the like (object) in the shooting scene than the actual camera, the resolution of the object is lowered and the image becomes blurred. If the path of the virtual camera (movement of the position of the virtual camera along the time axis) is set without considering these, the obtained virtual viewpoint image will have low image quality. Therefore, there is a possibility that the completed virtual viewpoint image will be confirmed on the preview screen or the like, and the route setting of the virtual camera will be redone (reset) many times.

In this regard, for example, a technique has been proposed in which information on blind spots between images taken by an actual camera is presented to a user (Patent Document 1). In the technique of Patent Document 1, by visualizing a blind spot area or an area that the observer does not want to see on a 2D map, it is possible to confirm the blind spot area in advance without actually generating a virtual viewpoint image. I was doing it.

Japanese Unexamined Patent Publication No. 2011-172169

However, in the method of Patent Document 1, it was not possible to confirm the correspondence between the position of the virtual viewpoint and the quality of the virtual viewpoint image before setting the virtual viewpoint. Therefore, in the present invention, it is possible to confirm the correspondence between the position of the virtual viewpoint and the quality of the virtual viewpoint image before setting the virtual viewpoint. The purpose of this is to suppress the work of re-setting the route of the virtual camera.

The image processing apparatus according to the present invention has a first common area in which imaging areas captured by a plurality of imaging devices included in the first imaging device group overlap, and a second image processing apparatus different from the first imaging device group. A second common area in which the shooting areas shot by a plurality of shooting devices included in the shooting device group overlap, and the resolution of the shot image is higher than that of the shot image corresponding to the first common area. The specific means for specifying the above, the display control means for causing the display means to display information based on the first common area and the second common area specified by the specific means, and the display means for the display means. A virtual viewpoint image generated based on a plurality of captured images obtained by a plurality of imaging devices included in the first imaging device group and the second imaging device group based on user input based on the displayed information. It is characterized by having a determination means for determining information representing a virtual viewpoint according to the above.

According to the present invention, the user can grasp in advance the resolution of the object in the completed virtual viewpoint image in the route setting of the virtual camera. As a result, the work of re-setting the route of the virtual camera is suppressed.

It is a figure which shows an example of the structure of a virtual viewpoint video system. (A) is a diagram showing an example of camera arrangement, and (b) is a diagram showing a shooting area of cameras belonging to each camera group. It is a flowchart which showed the whole flow until the virtual viewpoint image is generated. It is a flowchart which shows the process of generating a GUI screen used for parameter setting of a virtual camera. It is a figure explaining the derivation method of the common shooting area for each camera group. It is a figure which shows an example of the result that the common shooting area for each camera group is projected on a field map. It is a flowchart which shows the flow of the parameter setting process of a virtual camera. It is a figure which shows an example of the GUI screen for parameter setting of a virtual camera. It is a flowchart which showed the detail of the height adjustment process of the virtual camera which concerns on Embodiment 2. It is a figure which shows an example of the sample image of a virtual viewpoint image. (A) to (c) are diagrams for explaining a method for creating a sample image. It is a flowchart which showed the details of the virtual camera and the gaze point path designation process which concerns on Embodiment 3. It is a figure explaining the intersection determination.

Hereinafter, embodiments of the present invention will be described with reference to the drawings. It should be noted that the following embodiments do not limit the present invention, and not all combinations of features described in the present embodiment are essential for the means for solving the present invention. The same configuration will be described with the same reference numerals.

Embodiment 1

FIG. 1 is a diagram showing an example of the configuration of a virtual viewpoint video system in the present embodiment. The virtual viewpoint video system shown in FIG. 1 includes an image processing device 100 and two types of camera groups 109 and 110. The image processing device 100 includes a CPU 101, a main memory 102, a storage unit 103, an input unit 104, a display unit 105, and an external I / F 106, and each unit is connected via a bus 107. First, the CPU 101 is an arithmetic processing device that collectively controls the image processing device 100, and executes various programs stored in the storage unit 103 or the like to perform various processes. The main memory 102 temporarily stores data and parameters used in various processes, and also provides a work area to the CPU 101. The storage unit 103 is a large-capacity storage device that stores various data necessary for displaying various programs and GUIs (graphical user interfaces), and for example, a non-volatile memory such as a hard disk or a silicon disk is used. The input unit 104 is a device such as a keyboard, a mouse, an electronic pen, and a touch panel, and accepts various user inputs. The display unit 105 is composed of a liquid crystal panel or the like, and performs GUI display for setting a route of a virtual camera at the time of generating a virtual viewpoint image. The external I / F unit 106 is connected to each camera constituting the camera groups 109 and 110 via a network (here, LAN 108), and transmits / receives video data and control signal data. The bus 107 connects each of the above-mentioned parts and performs data transfer.

The above two types of cameras are a zoom camera group 109 and a wide-angle camera group 110, respectively. The zoom camera group 109 is composed of a plurality of cameras equipped with a lens having a narrow angle of view (for example, 10 degrees). The wide-angle camera group 110 is composed of a plurality of cameras equipped with a lens having a wide angle of view (for example, 45 degrees). Each of the cameras constituting the zoom camera group 109 and the wide-angle camera group 110 is connected to the image processing device 100 via the LAN 108. Further, the zoom camera group 109 and the wide-angle camera group 110 can start and stop shooting, change the camera settings (shutter speed, aperture, etc.), and transfer the shot video data based on the control signal from the image processing device 100. conduct.

Regarding the system configuration, there are various components other than the above, but since it is not the main subject of the present invention, the description thereof will be omitted.

FIG. 2A is a diagram showing an example of camera arrangement in an imaging system including two types of camera groups, a zoom camera group 109 and a wide-angle camera group 110, in a stadium where, for example, soccer is played. A player as an object 202 exists on the field 201 in which the competition is performed. The 12 zoom cameras 203 constituting the zoom camera group 109 and the 12 wide-angle cameras 204 constituting the wide-angle camera group 110 are arranged so as to surround the field 201. The area 213 surrounded by the dotted line in FIG. 2B shows the shooting area of the zoom camera 203. Since the zoom camera 203 has a narrow angle of view, the shooting area is narrow, but the zoom camera 203 has a characteristic that the resolution when the object 202 is shot is high. Further, the area 214 surrounded by the dotted line in FIG. 2C shows the shooting area of the wide-angle camera 204. Since the wide-angle camera 204 has a wide angle of view, the shooting area is wide, but the wide-angle camera 204 has a characteristic that the resolution when the object 202 is shot is low. Although the shape of the photographing area is shown as an ellipse in FIG. 2B for convenience, the shape of the actual photographing area of each camera is generally rectangular or trapezoidal as described later.

FIG. 3 is a flowchart showing the entire flow until the virtual viewpoint image is generated in the image processing device 100. This series of processes is realized by the CPU 101 reading a predetermined program from the storage unit 103, expanding it into the main memory 102, and executing this by the CPU 101.

In step 301, shooting parameters such as exposure conditions at the time of shooting and a signal for starting shooting are transmitted to the zoom camera group 109 and the wide-angle camera group 110. Each camera belonging to each camera group starts shooting according to the received shooting parameters, and holds the obtained video data in the memory in each camera.

In step 302, the multi-viewpoint video data taken by each zoom camera 203 belonging to the zoom camera group 109 and the multi-viewpoint video data taken by each wide-angle camera 204 belonging to the wide-angle camera group 110 are acquired. The acquired video data of the plurality of viewpoints (here, 12 viewpoints each) is expanded in the main memory 102.

In step 303, the three-dimensional shape estimation process of the object is executed using the multi-viewpoint video data acquired in step 302. As the estimation method, a known method such as a Visual-hull method using the contour information of the object or a Multi-view stereo method using triangulation may be applied.

In step 304, various parameters such as the movement path of the virtual camera necessary for generating the free start point image are set based on the estimated object shape data. The details of the processing related to the parameter setting of the virtual camera will be described in detail later.

In step 305, a virtual viewpoint image is generated according to the set virtual camera parameters. The virtual viewpoint image can be generated by using computer graphics technology to generate an image viewed from a set virtual camera with respect to the estimated object shape.

In step 306, it is determined whether or not to change the parameter settings of the virtual camera to generate a new virtual viewpoint image. This process is performed based on an instruction from the user who has confirmed the image quality and the like by looking at the virtual viewpoint image displayed on the preview screen (not shown). If the user wants to regenerate the virtual viewpoint image, the parameters related to the virtual camera are set again (return to step 304). Then, when the parameter is changed, the virtual viewpoint image is generated again with the changed content. On the other hand, if there is no problem with the generated virtual viewpoint image, this process ends. The above is a rough flow until the virtual viewpoint image is generated according to the present embodiment.

Subsequently, the preparation process of the GUI screen used in the parameter setting process of the virtual camera in the above-mentioned step 304 will be described. FIG. 4 is a flowchart showing a process of generating a GUI screen used for setting parameters of a virtual camera. The process of FIG. 4 may be executed prior to each process shown in FIG. 3, or may be executed at the timing when step 304 is executed.

In step 401, the camera information of all the camera groups (here, two types of the zoom camera group, 109, and the wide-angle camera group 110) is acquired. The camera information includes information such as the installation location of the cameras belonging to each camera group, the gazing point position, and the angle of view. In the present embodiment, the gazing point position is a position that is the center of photography for all 24 cameras. These information may be acquired by reading out what is stored in the storage unit 103 in advance, or by accessing each camera (or one camera representing each camera group) via LAN 108 and acquiring the information. You may.

In step 402, a region (common shooting region) in which the shooting regions of a plurality of cameras having substantially the same angle of view overlap is derived for each camera group based on the acquired camera information of each camera group. Here, the expression "substantially the same" is due to the fact that the difference in the distance from the camera to the gazing point causes minute fluctuations in the angle of view between the cameras. The angles of view of the cameras belonging to the same camera group do not have to be exactly the same. FIG. 5 is a diagram illustrating a method of deriving a common shooting area for each camera group. Now, two cameras 501 and 502 belonging to the same camera group are shooting with the same gaze point with respect to the field 201. At this time, the shooting area of the camera group to which the camera 501 and the camera 502 belong is the intersection 511 of the quadrangular pyramid and the field 201 directed from the camera 501 toward the gazing point, and the quadrangular pyramid and the field 201 directed from the camera 502 toward the gazing point. It is obtained as a region 513 that overlaps with the intersecting surface 512 with. Here, for convenience of explanation, two cameras 501 and 502 have been used, but the same idea applies to three or more cameras. In this way, a common shooting area for each camera group is derived.

In step 403, each of the derived common shooting areas for each camera group is visualized and projected in a recognizable manner on a bird's-eye view (field map) of the entire shooting space including the field 201. FIG. 6 is a diagram showing an example of a result in which a common shooting area for each camera group is recognizablely projected on a field map. In FIG. 6, the rectangular frame 600 shows the field map. The broken line ellipse 601 indicates a common shooting area of the zoom camera group 109 (hereinafter referred to as a zoom camera group shooting area). Within the zoom camera group shooting area 601 it is possible to generate a virtual viewpoint image using the multi-viewpoint image taken by the zoom camera group 109. Since the resolution of the object is relatively high in the zoom camera group shooting area 601, the image quality of the virtual viewpoint image should be maintained even if the virtual camera is brought closer to the object (even if the height of the virtual camera is lowered). Is possible. In the case of the present embodiment, the zoom camera group shooting area 601 is an area where the shooting areas of all 12 zoom cameras 203 belonging to the zoom camera group 109 overlap. The ellipse 602 of the alternate long and short dash line indicates the common shooting area of the wide-angle camera group 110 (hereinafter, the wide-angle camera group shooting area). Within the wide-angle camera group shooting area 602, it is possible to generate a virtual viewpoint image using the multi-viewpoint image taken by the wide-angle camera group 110. In the wide-angle camera group shooting area 602, the resolution of the object is relatively low, so if the virtual camera is brought closer to the object than a certain level (the height of the virtual camera is lowered below a certain level), the image quality of the virtual viewpoint image deteriorates. Will be done. In the case of the present embodiment, the wide-angle camera group shooting area 602 is an area where the shooting areas of all 12 wide-angle cameras 204 belonging to the wide-angle camera group 110 overlap. Further, in the area 603 indicated by the diagonal line in the field map 600, the shooting areas of the wide-angle cameras 204 belonging to the wide-angle camera group 110 having a wide angle of view do not overlap in the present embodiment, and it is not possible to generate a virtual viewpoint image of a certain quality. The possible area (hereinafter, the area where the virtual viewpoint image cannot be generated) is shown. In the example of FIG. 6, the outer edges of the zoom camera group shooting area and the wide-angle camera group shooting area are shown by broken lines and alternate long and short dash lines, respectively, and the virtual viewpoint image generation impossible area is shown by diagonal lines, but the present invention is not limited to this. The common shooting area for each camera group capable of generating a virtual viewpoint image may be displayed so as to be recognizable by the user, and each area may be indicated by, for example, color coding. In FIG. 6, for convenience of explanation, the shape of the common shooting area of each camera group is shown by an ellipse, but it is actually a polygon. In this way, the GUI screen used in the parameter setting process of the virtual camera is ready.

Next, the parameter setting process of the virtual camera according to the present embodiment will be described with reference to the flow of FIG. 7. The process of FIG. 7 corresponds to the process of step 304 of FIG. First, in step 701, the GUI screen for setting the parameters of the virtual camera is displayed on the display unit 105. FIG. 8 is a diagram showing an example of a GUI screen. In the GUI screen 800 shown in FIG. 8A, the field map 600 obtained in the above-mentioned preparation process is displayed on the left side of the screen, and the object shape 810 estimated in the above-mentioned step 303 is mapped on the field map 600. ..

In step 702, the designation of the movement path (camera path) of the virtual camera and the movement path (gaze point path) of the gazing point is accepted via the displayed GUI screen. After pressing the button 801 or 803, the user operates the mouse or the like on the field map 600 to move the cursor, thereby designating the movement locus as the camera path or the gazing point path. In FIG. 8A, the thick arrow 811 indicates the designated camera path, and the dotted arrow 812 indicates the designated gaze point path. The camera path and the gaze point path specified in this way are associated with each other so as to match in the setting frame. That is, when the virtual viewpoint image is composed of, for example, 600 frames, each division point obtained by dividing each path by the number of frames is associated with each other in order from the start point of the arrow. When the user moves the cursor to an arbitrary position (coordinates) on the camera path 811 and clicks with the mouse, etc., the division point P0 closest to the cursor is selected and displayed, and the gazing point position Q0 corresponding to P0 is also displayed. indicate. At this time, the angle of view area 813 seen from the virtual camera is also displayed at the same time. Since the user can grasp the positional relationship between the gazing point Q0, the object 810, and the angle of view area 813, the gazing point Q0 and the object 810 cannot generate the zoom camera group shooting area 601 and the wide-angle camera group shooting area 602 and the virtual viewpoint image. It is possible to perform the designated work while confirming which area of the area 603 it fits in, that is, what kind of resolution the virtual viewpoint image has. Therefore, the camera path and the like can be set appropriately, and the number of times of redoing the designated work can be suppressed. It is also possible to correct the path by moving P0 and Q0 with a mouse or the like. Further, the height of the virtual camera path from the field 201 is set to a default value (for example, 15 m), and the height of the gazing point path is set to a default value lower than that of the virtual camera path (for example, 1 m).

In step 703, the processing is separated depending on whether or not the height of the virtual camera and the gazing point is adjusted. When the user presses the camera path height edit button 802 or the gaze point path height edit button 804, the process proceeds to step 704.

In step 704, a process of adjusting the height of the virtual camera and the gazing point (height adjustment process) is executed. Here, the process of adjusting the height of the virtual camera will be described as an example. The user moves the cursor to an arbitrary position (coordinates) on the camera path 811 and specifies the position (height edit point) of the virtual camera whose height is to be changed by performing a click operation with a mouse or the like. Here, as in step 703, the division point closest to the cursor is selected and displayed as the height edit point. In FIG. 8B, a portion indicated by a cross on the camera path 811 indicates a height edit point specified by the user. It is possible to set a plurality of height edit points. In the example of FIG. 8B, two height edit points P1 and P2 are set, and points Q1 and Q2 on the gazing point path 812 corresponding to P1 and P2 are displayed accordingly. Also, the distance connecting P1 and Q1 and P2 and Q2 is displayed on the GUI. This is because the size of the object in the virtual viewpoint image changes depending on the distance between the virtual camera and the gazing point. The closer the distance, the larger the size of the object, and the more likely it is that the image will be out of focus. In the case of FIG. 8B, the distance between P1 and Q1 is 24 m, and the distance between P2 and Q2 is 20 m, and P2 and Q2 are easier to blur. When the height edit point is set, the height setting window 820 is displayed in the GUI screen 800, and the user can use an arbitrary value (unit: m) in the input field 821 corresponding to each edit point in the height setting window 820. ) Can be entered to change the height of the virtual camera at that position. In this case as well, the user can specify an arbitrary value while checking the zoom camera group shooting area 601 and the wide-angle camera group shooting area 602, and the distance between the virtual camera and the gazing point, so that the height of the virtual camera can be appropriately set. Can be set. The height other than the place where the altitude is changed by the height editing point is adjusted by interpolating from the height editing point at the adjacent position or the default value so that the height does not change suddenly.

The above is the content of the parameter setting process of the virtual camera according to this embodiment. In the GUI screen of this embodiment, each camera group shooting area is displayed in two dimensions, but it may be displayed in three dimensions.

As described above, according to the present embodiment, the user can set parameters such as the movement path and height of the virtual camera while grasping the shooting area of each camera group. This makes it possible to suppress redoing the parameter setting work when generating a virtual viewpoint image.

Embodiment 2

In the first embodiment, the common shooting area for each camera group is visualized on the GUI screen for parameter setting of the virtual camera and projected on the field map so that the camera path and the like can be set appropriately. Next, an embodiment in which the user can directly confirm the resolution of the object in adjusting the height of the virtual camera will be described as the second embodiment.

Since the basic system configuration and the general flow of the self-viewpoint video generation process are the same as those in the first embodiment, the description thereof will be omitted, and the differences will be mainly described below.

FIG. 9 is a flowchart showing the details of the height adjustment process of the virtual camera (step 704 described above) according to the present embodiment.

In step 901, the designation of the height edit point is accepted. Here, too, it is assumed that two points, P1 and P2, are designated as shown in FIG. 8B described above. In the following step 902, the lower limit of the height at the specified height edit point is acquired. The lower limit of height indicates the height of the virtual camera corresponding to the minimum acceptable level for the resolution of the object, the distance between the virtual camera and the gazing point, the angle of view of the virtual camera, and the note. It is determined in advance by the combination of the camera group shooting areas where the viewpoint is located. That is, when the gazing point is located in the high-resolution camera group shooting area, the distance between the virtual camera and the gazing point can be shortened, so that the lower limit of the height is low. On the other hand, when the gazing point is located in the low-resolution camera group shooting area, the lower limit of the height becomes high. Further, when the angle of view is wide, the lower limit of height is low, and when the angle of view is narrow, the lower limit of height is high. Whether or not the resolution is acceptable is determined based on criteria such as whether or not the resolution is lower than that of the object in the actual camera. To obtain the lower limit of height, first, the gazing point coordinates of the specified height editing point are acquired, the classification information of each camera group shooting area corresponding to the coordinates, and the distance between the virtual camera and the gazing point. And the information of the angle of view of the virtual camera is acquired. The classification information means information such as a flag indicating that the zoom camera group shooting area 601 or the wide-angle camera group shooting area 602 is assigned to each coordinate position. Then, based on these acquired information, the height lower limit value of the virtual camera at the specified height edit point is acquired.

In step 903, a sample image of the virtual viewpoint image corresponding to the acquired lower limit of height is displayed. The sample image is an object cut out from the image of the actual camera. The cutting method will be described with reference to FIGS. 11 (a) and 11 (b). FIG. 11A shows an image 1101 taken by a certain real camera. Image 1101 shows objects A to E corresponding to five athletes. By identifying the faces and uniform numbers of the objects A to E with respect to the image 1101 using the existing recognition technique, a sample image obtained by cutting out the area for each object can be obtained. The data of the obtained sample image is stored in the storage unit 103 together with the information on the number of pixels (width and height). FIG. 11B shows an image 1102 taken by another real camera. The same processing is performed on the image 1102 to acquire a sample image. Then, if the number of pixels of the newly created sample image exceeds the number of pixels of the previously created sample image, the process of overwriting (updating) with the newly created sample image is repeated for each object. By executing this process on all real cameras, the sample image with the highest resolution can be prepared for each object. When displaying the sample image, the sample image is reduced according to the distance between the virtual camera and the object and the angle of view of the virtual camera. The method of this reduction will be described with reference to FIG. 11 (c). First, the object closest to the gazing point is detected, and the data of the sample image of the object is read from the storage unit 103 into the main memory 102. Next, the number of pixels in the object area in the virtual viewpoint image is calculated from the distance between the virtual camera and the object and the angle of view of the virtual camera. Specifically, it can be calculated by arranging a three-dimensional model 1122 of a player's standard height size at the position of the detected object and drawing a preview image 1123 when viewed from the virtual camera 1121. Then, the sample image is reduced so that the object area in the preview image 1123 and the number of pixels (width and height) of the sample image match or approximate. After reduction, it is enlarged by simple interpolation to an appropriate size for display on the GUI screen 800. The sample image is displayed on the GUI screen 800 in a format such as the sub-window shown in FIG. 10 (a). This allows the user to check the resolution of the object in the actual image when the height is lowered to the lower limit. At this time, as shown in FIG. 10A, the corresponding lower limit of height may be displayed together with the sample image.

In step 904, the height change input via the height setting window 820 is accepted for the specified height edit point. In the following step 905, it is determined whether or not the height after the change is below the lower limit of the height. If the height after the change is below the lower limit, the process proceeds to step 906. On the other hand, if the height after the change is equal to or higher than the lower limit, this process is exited.

In step 906, the sample image is updated with the content according to the height after the change, that is, the sample image having a low resolution exceeding the allowable limit. FIG. 10B shows an example of the updated sample image, and it can be seen that the sample image corresponding to the height adjustment point P2 is out of focus. The sample image after this update is a reduced / enlarged display of the sample image peculiar to the object according to the distance between the virtual camera and the gazing point after setting and the angle of view of the virtual camera, as in step 903 described above. Is. At this time, a message indicating that the height has fallen below the lower limit, for example, "It has fallen below the lower limit" may be displayed (warning display). In the following step 907, the process is switched depending on whether or not the height is changed again. If a new height is specified, the process returns to step 904. If no new height is specified, this process is exited.

In this embodiment, the image of the actual camera that shot the actual game is used as a sample image, but for example, the image during practice or the portrait taken in the studio may be used, or computer graphics may be used. You may use the image which rendered the person. Further, as long as the resolution can be confirmed, an image using an object different from the actual object may be used as a sample image.

The above is the content of the height adjustment process of the virtual camera according to the present embodiment. In the case of the present embodiment, when editing the height of the virtual camera, the sample image corresponding to the lower limit of the height at the specified height editing point is visualized and displayed, so that the user can use the more appropriate height. It becomes easier to judge.

Embodiment 3

In the first embodiment, the common shooting area for each camera group is visualized on the GUI screen for parameter setting of the virtual camera and projected on the field map so that the camera path and the like can be set appropriately. Next, an embodiment in which the user can confirm the region where the resolution of the object fluctuates abruptly will be described as the third embodiment.

Since the basic system configuration and the general flow of the virtual viewpoint image generation process are the same as those in the first embodiment, the description thereof will be omitted, and the differences will be mainly described below.

FIG. 12 is a flowchart showing the details of the gazing point path setting process (corresponding to step 702 described above) according to the present embodiment.

In step 1201, after accepting the designation of the gazing point, the gazing point path is searched. This is a process of investigating which camera group the coordinate value of the gazing point moves in. Next, in step 1202, it is determined whether or not the gazing point path found by the search intersects the shooting areas of the plurality of camera groups. FIG. 13 is a diagram illustrating this intersection determination. In the example of FIG. 13, the gazing point path 812 starting from the wide-angle camera group shooting area 602 passes through the zoom camera group shooting area 601 and returns to the wide-angle camera group shooting area 602 again. Therefore, in this case, it is determined that they intersect. If it is determined that they intersect in this way, the process proceeds to step 1203. On the other hand, if it is determined that they do not intersect, this process ends. In step 1203, as shown in FIG. 13, the step mark 1301 of the resolution is displayed at the coordinate position where the photographing area and the gazing point path intersect. This is because the specified gazing point moves so as to straddle the shooting area of a plurality of camera groups, and the resolution may change sharply in the vicinity of the step mark 1301 where the focal length of the camera changes. , To warn the user of it. In such a case, the user needs to take measures such as moving the virtual camera away from the gazing point in the vicinity of the step mark 1301.

The above is the content of the warning process of the change in resolution according to the present embodiment. In the case of this embodiment, since the fluctuation of the resolution of the virtual viewpoint image is visualized when editing the virtual camera or the gazing point path, it is easy for the user to determine a more appropriate virtual camera or gazing point path setting. Become.

In the above embodiments 1 to 3, the image processing device 100 generates a GUI for setting a virtual camera (virtual viewpoint), displays the GUI, accepts user operations for the GUI, and generates a virtual viewpoint image. I explained an example of doing all of the above. However, it is not limited to this example. For example, a device that generates / displays a GUI and accepts a user operation and a device that generates a virtual viewpoint image based on the user operation may be separate devices. Further, a virtual viewpoint video system having a device for generating a GUI (typically one), a plurality of devices for receiving GUI display and user operation, and one or a plurality of devices for generating a virtual viewpoint video. It is also possible to apply the present invention to.

Further, in the above-described embodiments 1 to 3, the common shooting area in which the shooting areas of all the cameras belonging to the zoom camera group 109 overlap and the common shooting area in which the shooting areas of all the cameras belonging to the wide-angle camera group 110 overlap are used. Was mainly explained as an example of displaying the image in an identifiable manner. However, it is not limited to this example. For example, of the 12 cameras belonging to the zoom camera group 109, the area where the shooting areas of 10 or more cameras overlap may be used as the common shooting area.

(Other embodiments)
The present invention supplies a program that realizes one or more functions of the above-described embodiment to a system or device via a network or storage medium, and one or more processors in the computer of the system or device reads and executes the program. It can also be realized by the processing to be performed. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

Claims (17)

A first common area where the shooting areas shot by the plurality of shooting devices included in the first shooting device group overlap, and a plurality of shootings included in the second shooting device group different from the first shooting device group. A specific means for identifying a second common area in which the shooting areas captured by the apparatus overlap and the resolution of the captured image is higher than that of the captured image corresponding to the first common area .
A display control means for causing the display means to display information based on the first common area and the second common area specified by the specific means.
Based on the user input based on the information displayed on the display means by the display control means, the plurality of captured images obtained by the plurality of imaging devices included in the first imaging device group and the second imaging device group can be obtained. A determination means for determining information representing a virtual viewpoint related to a virtual viewpoint image generated based on the
An image processing device characterized by having. The image processing apparatus according to claim 1, wherein the display control means causes the display means to display information that can identify the first common area and the second common area. The image according to claim 1 or 2, wherein the display control means causes the display means to display recognizable information on an area in which it is impossible to generate a virtual viewpoint image having a quality higher than a predetermined quality. Processing equipment. The display control means displays information that can identify the first common area and the second common area in the entire shooting space shot by the first shooting device group and the second shooting device group. The image processing apparatus according to any one of claims 1 to 3, wherein the image processing apparatus is displayed on the screen. The specific means has the first common area and the first common region based on the imaging information including at least the angle of view information of the plurality of imaging devices included in the first imaging apparatus group and the second imaging apparatus group. The image processing apparatus according to any one of claims 1 to 4, wherein a common area of 2 is specified. It has a storage means for storing the shooting information, and has a storage means.
The specifying means identifies the first common area and the second common area based on the photographing information acquired from the storage means.
The image processing apparatus according to claim 5. The plurality of photographing devices included in the first photographing apparatus group and the second photographing apparatus group are connected via a network.
The specifying means identifies the first common area and the second common area based on the shooting information acquired from the plurality of shooting devices via the network.
The image processing apparatus according to claim 5. The image processing apparatus according to any one of claims 1 to 7, wherein the display control means causes the display means to display information indicating a restriction condition regarding the position of the virtual viewpoint. When the position of the virtual viewpoint that does not satisfy the restriction condition regarding the position of the virtual viewpoint is specified by the user input, the display control means causes the display means to display a warning display indicating that the restriction condition is not satisfied. The image processing apparatus according to claim 8. The image processing apparatus according to claim 8 or 9, wherein the limiting condition is a condition relating to the height of the virtual viewpoint. The image processing apparatus according to any one of claims 1 to 10, wherein the display control means causes the display means to display a sample image corresponding to a position of a virtual viewpoint designated by the user input. The image processing apparatus according to any one of claims 1 to 11, wherein the determination means determines information representing a movement path of the virtual viewpoint based on the user input. The image processing apparatus according to any one of claims 1 to 12, wherein the determination means determines a movement path of a point of interest of interest by the virtual viewpoint based on the user input. Claims 1 to 13, wherein the angle of view of the plurality of imaging devices included in the first imaging device group is wider than the angle of view of the plurality of imaging devices included in the second imaging device group. The image processing apparatus according to any one of the following items. The image processing apparatus according to any one of claims 1 to 14 , wherein at least a part of the first common area and the second common area overlap each other. A first common area in which shooting areas taken by a plurality of shooting devices included in the first shooting device group overlap, and a plurality of shootings included in a second shooting device group different from the first shooting device group. A specific step of specifying a second common area in which the shooting areas taken by the apparatus overlap and the resolution of the shot image is higher than that of the shot image corresponding to the first common area .
A display step for displaying information based on the first common area and the second common area specified in the specific step, and a display step.
Based on user input based on the information displayed in the display step, it is generated based on a plurality of captured images obtained by the plurality of imaging devices included in the first imaging device group and the second imaging device group. The decision process to determine the information representing the virtual viewpoint related to the virtual viewpoint image and
An image processing method characterized by having. A program for operating a computer as the image processing device according to any one of claims 1 to 15.

Images