JP7458363B2 - Information processing device, information processing method, and information processing program - Google Patents

Description

The present invention relates to an information processing device, an information processing method, and an information processing program.

2. Description of the Related Art Conventionally, a technique has been provided for generating multi-view images (free-view images), which are a plurality of images of a subject (photographing target) taken from different viewpoints (see, for example, Patent Document 1).

Japanese Patent Application Publication No. 2016-119513

However, the above-described conventional technology may not be able to provide sufficient services using multi-view images. For example, although the above-mentioned conventional technology allows users to utilize multi-view images, there is room for improvement in the quality of service provision using multi-view images.

The present application has been made in view of the above, and aims to further improve the quality of service provision using multi-view images.

The information processing device according to the present application includes a guide display unit that displays a shooting guide on the screen that indicates the pose of the subject and the composition of the belongings at the time of shooting when shooting a multi-view image, and a guide display unit that displays a shooting guide on the screen that shows the pose of the subject and the composition at the time of shooting the object, and every time the viewpoint during shooting is moved. a guide changing unit that automatically changes the form of the photography guide displayed on the screen according to the viewpoint at that time ; and when it is determined that the subject's pose and belongings match the photography guide. , and a shooting determination unit that automatically shoots images .

According to one aspect of the embodiment, the quality of service provision using multi-view images can be further improved.

FIG. 1 is an explanatory diagram showing an overview of an information processing method according to an embodiment. FIG. 2 is an explanatory diagram showing an outline of the display of the photographing guide. FIG. 3 is an explanatory diagram showing an overview of replacement of a face image of a photographed model. FIG. 4 is a diagram illustrating a configuration example of an information processing system according to an embodiment. FIG. 5 is a diagram illustrating a configuration example of a terminal device according to an embodiment. FIG. 6 is a diagram illustrating a configuration example of an information providing device according to an embodiment. FIG. 7 is a diagram showing an example of a user information database. FIG. 8 is a diagram showing an example of a history information database. FIG. 9 is a diagram showing an example of an image information database. FIG. 10 is a flowchart showing a processing procedure according to the embodiment. FIG. 11 is a diagram showing an example of the hardware configuration.

DESCRIPTION OF THE PREFERRED EMBODIMENTS An information processing apparatus, an information processing method, and an information processing program according to the present application (hereinafter referred to as "embodiments") will be described in detail below with reference to the drawings. Note that the information processing apparatus, information processing method, and information processing program according to the present application are not limited to this embodiment. In addition, in the following embodiments, the same parts are given the same reference numerals, and redundant explanations will be omitted.

[1. Overview of information processing method]
First, an overview of an information processing method performed by an information processing device according to an embodiment will be described with reference to Fig. 1. Fig. 1 is an explanatory diagram showing an overview of the information processing method according to an embodiment. Note that Fig. 1 will be described by taking as an example a case where a service is provided using multi-view images.

As shown in FIG. 1, the information processing system 1 includes a terminal device 10 and an information providing device 100. The terminal device 10 and the information providing device 100 are connected to be able to communicate with each other by wire or wirelessly via a network N (see FIG. 4). In this embodiment, the terminal device 10 cooperates with the information providing device 100.

The terminal device 10 is a smart device such as a smartphone or a tablet used by a user U (user), and can communicate with any server device via a wireless communication network such as 4G (Generation) or LTE (Long Term Evolution). This is a mobile terminal device that can perform Further, the terminal device 10 has a screen such as a liquid crystal display, which has a touch panel function, and displays data such as content through the user U's finger, stylus, etc., such as a tap operation, a slide operation, a scroll operation, etc. Accepts various operations on. Note that an operation performed on an area of the screen where content is displayed may be an operation on the content. Further, the terminal device 10 may be not only a smart device but also an information processing device such as a desktop PC (Personal Computer) or a notebook PC.

The information providing device 100 is an information processing device that cooperates with the terminal device 10 of each user U and provides API (Application Programming Interface) services for various applications (hereinafter, apps) and various data to the terminal device 10 of each user U, and is realized by a server device, a cloud system, etc.

Further, the information providing device 100 may be an information processing device that provides some kind of web service online to the terminal device 10 of each user U. For example, the information providing device 100 provides web services such as Internet connection, search service, SNS (Social Networking Service), electronic commerce (EC), and fashion coordination (photos (still images) and videos of people wearing fashion items). Services such as posting sites, electronic payments, online games, online banking, online trading, accommodation/ticket reservations, video/music distribution, news, maps, route searches, route guidance, route information, service information, weather forecasts, etc. may be provided. In reality, the information providing apparatus 100 may cooperate with various servers that provide the above-mentioned Web services, mediate the Web services, or may be in charge of processing the Web services.

Note that the information providing device 100 can acquire user information regarding the user U. For example, the information providing device 100 acquires information regarding attributes of the user U, such as the user's U's gender, age, and area of residence. Then, the information providing device 100 stores and manages information regarding attributes of the user U along with identification information indicating the user U (user ID, etc.).

Further, the information providing device 100 acquires various kinds of history information (log data) indicating the behavior of the user U from the terminal device 10 of the user U or from various servers based on the user ID or the like. For example, the information providing device 100 acquires a location history, which is a history of the user U's location, date and time, from the terminal device 10. Further, the information providing device 100 acquires a search history, which is a history of search queries input by the user U, from a search server (search engine), an electronic commerce server, and a posting server. Further, the information providing device 100 acquires a viewing history, which is a history of contents and products (fashion items) viewed by the user U, from a content server, an electronic commerce server, and a posting server. Further, the information providing device 100 acquires a purchase history (payment history), which is a history of contents and products (fashion items) purchased and paid for by the user U, from an electronic commerce server and a payment processing server. Further, the information providing device 100 may acquire the listing history and the sales history, which are the listing history of the user U on the marketplace, from the electronic commerce server or the payment processing server. In addition, the information providing device 100 posts a posting history that is a history of fashion coordinations (fashion items) posted by the user U and a support history that is a history of fashion coordinations (fashion items) that were liked (liked) by the viewer. Obtain from server or SNS server.

In this embodiment, the information providing device 100 determines a combination of fashion items (clothes, accessories, bags, shoes, hats, etc.) and their additional information based on fashion information related to the user's fashion preferences using image analysis and AI (Artificial Intelligence), and displays them in an appropriate position. Here, the information providing device 100 identifies fashion items contained in an image using image analysis, and assigns annotation tags (hereinafter, tags) corresponding to the fashion items. Note that assigning a tag means displaying the tag in an appropriate position near the tagging target. For example, the information providing device 100 assigns tags corresponding to fashion items to images taken by a user wearing fashion items as photos or videos to be posted on a posting site or SNS, or to images taken and posted in this manner by other users.

For example, the information providing device 100 may assign a tag that displays detailed information about the fashion item. The information providing device 100 may also assign a tag that guides the user to the product page (sales page/purchase page/advertising page, etc.) of the fashion item, or to the product pages of other fashion items related to the fashion item (products that are often purchased together, products of the same brand, etc.).

Further, the information providing device 100 may add a characteristic tag to the fashion item. For example, the information providing device 100 assigns a tag that has a high degree of compatibility with a fashion item on the color wheel. Alternatively, a tag is added that is estimated to have a high support rate (like rate) among viewers on posting sites and SNS. Alternatively, tags that are estimated to increase the purchase rate of fashion items by viewers are added. This can encourage viewers to purchase fashion items.

Fashion information regarding the user's fashion preferences can be obtained from each user U's attribute information, history information, and the like. Furthermore, the fashion information includes information regarding fashion items worn by the user U and fashion items viewed by the user U (fashion item information). In the present embodiment, fashion information is behavioral information (search, selection, , viewing, purchasing, posting). Furthermore, the fashion information includes information (fashion item information) regarding fashion items searched for, viewed, purchased, owned, posted, etc. by the user U.

The user's fashion preferences are based on various historical information (log data) from e-commerce sites such as fashion mail order sites and fashion coordination posting sites that connect accounts with this system (or are part of this system). It is determined by the logic that calculates/estimates based on the For example, the information providing device 100 calculates/estimates a "combination of clothes and tags" from clothes that the user has purchased from a fashion mail order site (defined as clothes worn by the user that match the user information) and clothes that the user has viewed. and attach tags that are ideal for pairing with clothing. Further, the information providing device 100 calculates/estimates the "combination of clothes and tags" from the clothes that the user has viewed on the fashion coordination posting site, and assigns the optimal tag to the combination with the clothes.

Furthermore, in addition to fashion information, the information providing device 100 may add tags based on information such as the user's daily activities, habits, and mood. That is, the information providing device 100 may change the content and form of the tag for each user viewing the same fashion item.

In the present embodiment, the information providing device 100 provides fashion items (clothes, accessories, tags (bags, shoes, hats, etc.). For example, as tagging, a relationship with a product page of an electronic commerce site such as a fashion mail order site (for example, "ZOZOTOWN" (registered trademark)) is set. The tag will be displayed superimposed on the photo. When a tag is clicked, a transition is made to a product page where the product (fashion item) is sold on an electronic commerce site such as a fashion mail order site (for example, "ZOZOTOWN" (registered trademark)).

In addition, the information providing device 100 displays images from multiple viewpoints on a fashion coordination posting site (e.g., "WEAR" (registered trademark)). For example, when a photo (image) is displayed by clicking or scrolling on a fashion coordination posting site (e.g., "WEAR" (registered trademark)), the information providing device 100 automatically/over time/in response to a user operation switches to an image with a different viewpoint.

[1-1. Annotation tags]
In this embodiment, when the information providing device 100 assigns an annotation tag to a subject in a multi-view image, it detects the same gaze point (the subject to be tagged) from each viewpoint image of the multi-view image, and sets and displays a tag near that same gaze point.

As shown in FIG. 1, the information providing device 100 acquires a multi-view image from the terminal device 10 of the user U who is the poster via the network N (see FIG. 4) (step S1). For example, the information providing device 100 receives images taken from various viewpoints used to create a multi-view image from the terminal device 10 of the user U who is the poster, via the network N (see FIG. 4). get. In FIG. 1, the person to be photographed (photographed model) is shown as M, the bag to be photographed is shown as B, and the tag is shown as T in the image.

Next, the information providing apparatus 100 performs image recognition or machine learning on all the acquired images to recognize, identify, and classify the photographic subject (step S2). For example, the information providing apparatus 100 identifies the photographic subject included in each image and classifies it by category. There may be a plurality of objects to be photographed.

Next, the information providing apparatus 100 estimates the position of the object to be photographed within the multi-view image (position within the image) (step S3). In the present embodiment, the information providing apparatus 100 estimates (or specifies) the three-dimensional position of the photographic target included in each image (for each viewpoint image) constituting the multi-view image. The three-dimensional position of the object to be imaged may be an absolute position such as coordinates within an image, or may be a relative position from a reference point or another object to be imaged. Further, the three-dimensional position is only an example.

Next, the information providing device 100 selects the target to be tagged from among the photographic targets (candidates for annotation target) whose three-dimensional positions in the multi-view image have been estimated from the user U who is the poster or the viewer. The selection of (annotation target) is accepted (step S4). For example, the information providing device 100 sends a product (fashion item) to be tagged (annotated) from the terminal device 10 of the user U who is the poster or the viewer via the network N (see FIG. 4). and accepts the specification of the product page of the corresponding product to be associated. At this time, the information providing device 100 uses image recognition or machine learning to search for similar images of the product from each product page of an e-commerce site such as a fashion mail order site (for example, "ZOZOTOWN" (registered trademark)), and adds the search result to the product page. The product page of the product may be automatically identified based on the information.

Note that if the information providing apparatus 100 does not accept the selection of tagging targets (annotation targets) from the user U who is the poster or the viewer, the information providing apparatus 100 selects all photographed targets in the multi-view image as tagging targets (annotation targets). (annotation target). In addition, the information providing device 100 unconditionally tags all photographed objects in the multi-view image, regardless of the selection of tagging objects (annotation objects) by the user U who is the poster or the viewer. It may also be used as a target (annotation target). Further, the information providing apparatus 100 may determine a tagging target (annotation target) among the photographing targets (annotation target candidates) according to a preset setting.

In addition, the information providing device 100 may use a learning model to estimate a tagging target (annotation target) from among the shooting targets whose three-dimensional positions in the multi-view image have been estimated. For example, the information providing device 100 may construct a learning model by learning about combinations of shooting targets previously selected as tagging targets and tags assigned to the tagging targets. Then, when the shooting target is input into the learning model, the information providing device 100 may infer and output an appropriate tag if the shooting target is a tagging target.

Next, the information providing device 100 sends a message to the terminal device 10 of the user U who is the viewer via the network N (see FIG. 4) about the selected target to be tagged from among the photographed targets in the multi-view image. (annotation target) is tagged with an annotation and displayed (step S5).

Next, the information providing device 100 changes the position of the annotation tag displayed on the terminal device 10 of the viewer, user U, in accordance with the change in the viewpoint of the multi-view image via the network N (see FIG. 4) (step S6). For example, when the viewpoint of the multi-view image is changed, the information providing device 100 automatically tracks the tagging target (annotation target) in the multi-view image, assigns an annotation tag to it, and displays it in an appropriate position. At this time, the information providing device 100 may assign an annotation tag to the tagging target (annotation target) for each image with a different viewpoint (each time) and display it in an appropriate position.

At this time, the information providing apparatus 100 arranges (displays) the annotation tag so that it does not overlap with other objects or other tags. Furthermore, the information providing device 100 continues to place (display) the tag at a position where the positional relationship between the annotation tag and the target of tagging (annotation target) is maintained.

As described above, in the present embodiment, the information providing apparatus 100 receives multi-view images (multi-view images posted by the poster, multi-view images specified by the viewer, etc.) from users who are posters or viewers. The annotation tagging target is identified from among the photographic targets included in the multi-view image, and the three-dimensional position of the tag target in the multi-view image is estimated. Then, the information providing device 100 attaches a tag to the tagging target according to the three-dimensional position of the tagging target. That is, the information providing device 100 displays (presents) the tag together with the target of tagging on the screen of the terminal device 10 to notify (present) the user.

The information providing device 100 also identifies and classifies the subject of the image captured for each viewpoint of the multi-view image using image recognition or machine learning. The information providing device 100 also accepts a selection of a tagging target from a user who is a contributor or viewer. Then, in response to the selection of the tagging target, the information providing device 100 identifies a tagging target for the annotation from among the subjects captured in the multi-view image.

For example, the information providing device 100 receives a selection of a tagging target and a designation of a web page to be associated with the tagging target from a user who is a poster or a viewer. Alternatively, the information providing device 100 searches a plurality of web pages on the network for images similar to the image to be tagged using image recognition or machine learning, and maps web pages containing similar images to the tagged target. automatically identified as

When the information providing apparatus 100 attaches a tag to an object to be tagged, the information providing apparatus 100 attaches the tag so that the tag does not overlap with other objects or other tags. Further, when attaching a tag to a target of tagging, the information providing apparatus 100 attaches a tag to the target of tagging if the target of tagging is not hidden by another target. Note that when the information providing device 100 attaches a tag to a tag target, even if the tag target is hidden by another target, the tag of the tag target is hidden from the tag of the other target. If the display priority is higher than the tag, the tag is added to the tag target without tags being added to other targets.

The above processes may be performed by the terminal device 10 using a function such as an app, instead of the information providing device 100. In other words, the processes may be completed on the terminal device 10.

[1-2. Change tag display position]
In the present embodiment, the information providing apparatus 100 changes the display position (display mode) of the content (annotation tag) indicating the object to be imaged, depending on the positional relationship of the object to be imaged in the multi-view image. Note that there may be a plurality of contents (annotation tags) indicating the subject to be photographed.

The information providing device 100 changes the display position of the tag according to the positional relationship between the annotation tag and the tagging target (annotation target).

If the annotation target is hidden behind the photographed model/user or other photographed target, the information providing device 100 hides the tag given to the annotation target.

For example, the information providing device 100 displays the annotation tag when the imaging range/display range of the annotation target is less than a predetermined value, or when the annotation target is hidden behind the user or other targets. do not.

Alternatively, the information providing apparatus 100 may change the context of tags between an annotation target and another target. When a plurality of tags exist, the information providing device 100 normally displays the target tag placed closest to the screen unless otherwise specified (nothing is set). In the present embodiment, the information providing apparatus 100 does not display the tag of the target placed furthest on the screen when a target with a higher priority than the target placed furthest on the screen is hidden in the shadows. , display the target tag with the highest priority.

At this time, the information providing apparatus 100 may set a priority for each target and determine the tag to be displayed based on the magnitude relationship of the set priority. Further, the information providing device 100 sets a target that is estimated to have higher interest of the user than the target placed closest to the screen as a target with higher priority than the target placed closest to the screen. You may decide.

In this manner, in the present embodiment, the information providing apparatus 100 identifies the target of annotation tagging from among the photographed targets included in the multi-view image, and attaches the tag according to the position of the target of tagging. . Thereafter, when the position of the tag target in the screen changes due to a change in the viewpoint of the multi-view image, the information providing device 100 changes the display position of the tag in accordance with the change in the position of the tag target. .

Furthermore, the information providing apparatus 100 arranges the tag at a position where the positional relationship between the tag and the tag is maintained even if the viewpoint of the multi-view image is changed.

Note that each of the above processes may be performed by the terminal device 10 instead of the information providing device 100 using functions such as an application. That is, the processing may be completed on the terminal device 10.

[1-3. Shooting guide display〕
In this embodiment, the information providing device 100 uses a terminal device 10 (imaging device) having a camera (in-camera, etc.) to take an image of itself (in-camera shooting) in order for the user to post a multi-view image. When shooting, a predetermined shooting guide is displayed on the screen (photo shooting screen) of the user's terminal device 10 used for shooting (or a display device installed/connected thereto), and each time an image is shot, ), change the shooting guide. FIG. 2 is an explanatory diagram showing an outline of the display of the photographing guide. In FIG. 2, the person to be photographed (photographed model) is indicated as M, and the bag to be photographed is indicated as B in the image.

As shown in FIG. 2, the information providing device 100 confirms that the terminal device 10 having a camera has started capturing a multi-view image in order for the user to post a multi-view image (step S11). For example, the information providing device 100 receives a signal or data indicating the start of capturing a multi-view image from the user's terminal device 10 via the network N (see FIG. 4).

Next, the information providing device 100 cooperates with the user's terminal device 10 via the network N (see FIG. 4) (step S12). For example, the information providing device 100 may control a shooting application of the terminal device 10 via an API. That is, the subsequent processing may be performed by the information providing device 100 in cooperation with the terminal device 10.

Next, the terminal device 10 displays a shooting guide on the screen of the terminal device 10 when the user takes an image of himself (in-camera shooting) (step S13).

Next, if the user's pose or the bag he or she is carrying deviates from the photographing guide, the terminal device 10 notifies the user of the position where the pose is deviated from the photographing guide (step S14). Note that the notification method may be screen display or audio guidance.

Next, the terminal device 10 automatically photographs the user if the user's pose, bag, etc. match the photographing guide (step S15). Note that even if the user's pose or the bag he or she is carrying does not completely match the shooting guide, if the user's pose or the bag they are carrying do not match the shooting guide, if they match at least a predetermined percentage, the terminal device 10 automatically determines that they match. You may take pictures. Further, the terminal device 10 may automatically take a picture if the pose of the user, the bag he/she is carrying, etc. match the photography guide. At this time, the information providing device 100 instructs the terminal device 10 in advance to automatically take a photo if the user's pose, bag, etc. that the user is carrying match the shooting guide. Good too.

Next, the terminal device 10 identifies and notifies a photographing target (candidate for annotation target) included in the photographed image (step S16). Details will be described later.

Next, the terminal device 10 changes the shooting guide in accordance with the change in viewpoint for generating a multi-viewpoint image (step S17). That is, the terminal device 10 displays a shooting guide that corresponds to the changed viewpoint.

Next, after the terminal device 10 has completed capturing images from all viewpoints required to generate the multi-view image, it posts the multi-view image (step S18). For example, the terminal device 10 generates a multi-view image from the captured images and automatically posts the multi-view image to the information providing device 100 via the network N (see FIG. 4). Alternatively, the terminal device 10 may directly post the captured images to the information providing device 100 via the network N (see FIG. 4). In this case, the information providing device 100 may obtain the posted images and then generate a multi-view image from the posted images.

In this embodiment, the user's terminal device 10 used to capture multi-viewpoint images displays, on the screen of the terminal device 10, a guide to facial expressions, a guide to posture (pose), a guide to how to hold the product, a guide to how to wear the product, or the like, as a shooting guide, and changes the shooting guide each time an image is captured by changing the position or angle.

For example, the terminal device 10 displays, on the screen of the user's terminal device 10 used for photographing, a silhouette or outline of the facial expression, posture (pose) the user should take, how to hold the product, or how to wear the product as a photographic guide. However, the silhouette or contour may be changed each time an image is captured by changing the position or angle. In this case, the user adjusts his/her facial expression, posture (pose), how to hold the product, and how to wear the product to match the silhouette and outline of the facial expression, posture (pose), and how to hold or wear the product displayed on the screen. Or change the way you wear it.

Further, the terminal device 10 may provide audio guidance (guidance) in conjunction with displaying and changing the photographing guide. In addition, the terminal device 10 displays comments regarding the content of changes on the shooting screen in areas where the user needs to change the facial expression, posture (pose), how to hold the product, how to wear the product, etc. A tag may be added.

(For pose guide)
Here, a pose guide will be described as an example of a photography guide. A user's terminal device 10 having a camera displays a pose guide on the screen of the terminal device 10 when capturing a multi-view image. The terminal device 10 changes the photography guide every time it photographs an image (every time it moves the viewpoint). That is, the pose guide changes one after another each time a photograph is taken.

Note that since the camera of the terminal device 10 is fixed, the user changes the position and angle of the camera by moving/rotating the terminal device 10. The photography guide contains information about poses, facial expressions, how to hold the camera, etc. The pose includes the direction of the face, the direction of the body, etc. The pose guide changes gradually (gradually) according to the position and angle of the camera. That is, the terminal device 10 changes the pose guide in stages according to the position and angle of the camera.

When the terminal device 10 determines that the pose guide matches the user's pose on the screen, it automatically takes a picture. At this time, the terminal device 10 determines the appropriateness of the user's pose. If the terminal device 10 determines that the user's pose is not appropriate, it gives specific advice to the user so that the user's pose is appropriate, and guides the user to take the appropriate pose.

For example, the terminal device 10 can be used to pull your chin, open/close your legs, lean your body, arch your back, move XX steps/XX cm in a specific direction (front/back/left/right), move your bag or pouch, etc. It displays pose guides such as "Kaze ni mo Tsuka" and provides audio guidance.

The terminal device 10 displays the user and the pose guide in a superimposed manner on the screen when photographing with the in-camera. The pose guide may be, for example, a silhouette or an outline, or may be a semi-transparent display. Alternatively, you may say "Please do this" in voice. The terminal device 10 identifies the difference between the user's current pose and the pose guide, and identifies parts that are different from the pose, such as "Please raise your right hand more" or "Please raise your foot a little more." It may be displayed/guided. Then, when the terminal device 10 determines that the user's pose matches the pose guide, the terminal device 10 may automatically shoot the image.

The pose guide may also be a guide to facial expressions. For example, it may be "turn around once and smile." In other words, the facial expression does not have to be the same each time a photo is taken. The terminal device 10 may also identify a pose in the first image and display the identified pose guide.

In this manner, in this embodiment, the terminal device 10 displays a predetermined shooting guide on the screen when shooting a multi-view image. For example, the terminal device 10 displays on the screen, as a photography guide, a guide regarding at least one of the pose, facial expression, and holding style of the person to be photographed (photographed model) among the subjects. Further, the terminal device 10 displays a silhouette, outline, or semi-transparent display of the subject on the screen as a photography guide.

Furthermore, when photographing a subject, the terminal device 10 displays a photographing guide superimposed on the subject. Further, the terminal device 10 specifies the pose of the photographed person among the subjects in the first photographed image, and selects and displays a photographic guide corresponding to the specified pose.

Furthermore, each time the terminal device 10 moves the viewpoint at the time of photographing, the terminal device 10 changes the photographing guide displayed on the screen according to the viewpoint. Furthermore, the terminal device 10 may change the photographing guide in stages as the viewpoint at the time of photographing moves.

Further, the terminal device 10 instructs the terminal device 10 to automatically take a photograph when the subject matches the photographing guide. Furthermore, when the subject is out of alignment with the photography guide, the terminal device 10 notifies (presents) the location where the subject is out of alignment with the photography guide to the photographer (user U). Note that the subject may be the photographer himself. That is, the photographer and the photographed person may be the same person.

Note that each of the above processes may be performed by the information providing apparatus 100 via the API instead of the terminal device 10.

[1-4. Recognition of candidates for annotation]
In this embodiment, the terminal device 10 identifies a shooting target (candidate for annotation target) included in the shot image when capturing images constituting a multi-view image, and notifies the user of the identified shooting target. do.

When registering a multi-view image, the terminal device 10 uses image recognition or machine learning to recognize a photographic subject included in the image each time a photograph is taken, and notifies the user of information regarding the photographic subject through a screen display or voice. A plurality of objects may be photographed. By identifying and notifying the subject of the photograph, it becomes easier to tag it later. Also, there is no need to add tags first. Note that the terminal device 10 may notify the user of the tagging target (annotation target) in advance, and notify the user whether or not the tagging target (annotation target) is being photographed each time a photograph is taken. good.

The terminal device 10 may also notify the user by displaying on the screen or by voice, such as "One piece has been taken", "How many shots of the background have been taken", etc. Each time a shot is taken, the terminal device 10 may also notify the user by displaying on the screen or by voice, how many more shots are left to be taken, such as "XX shots remaining". That is, each time a shot is taken, the terminal device 10 may notify the user of the number of shots of the subject, or the number of shots remaining until the required number of images constituting the multi-view image is taken. The terminal device 10 may also notify the user by displaying on the screen or by voice, of the position and angle of the camera where shooting has been completed (or not completed). The terminal device 10 may also notify the user by displaying on the screen or by voice, if there are any viewpoints (images) that have not been captured, regarding the images of each viewpoint required for generating the multi-view image. At this time, the terminal device 10 may notify the user by adding and displaying a tag indicating the notification content.

Additionally, the terminal device 10 may notify you of a subject that has become invisible (hidden behind the user or another subject) during shooting due to a change in the position or angle of the camera. good. At this time, the terminal device 10 adds and displays a tag indicating that the object to be photographed is hidden behind another object, so that the object becomes invisible during the photographing process (hidden behind the user or another object). ) The subject to be photographed may be notified.

Note that each of the above processes may be performed by the information providing apparatus 100 via the API instead of the terminal device 10.

[1-5. Replacing the face image of the photographed model]
In the present embodiment, the information providing apparatus 100 replaces the face of the photographed person (photographed model) in the multi-view image with the face of the user U who is the viewer. That is, the information providing device 100 replaces the face of the photographed person (photographed model) in the multi-view image with the face of another person. FIG. 3 is an explanatory diagram showing an overview of replacement of a face image of a photographed person (photographed model). In FIG. 3, in the image, M1 is the photographed person (photographed model) before face image replacement, B is the bag photographed with the photographed person, and the photographed person whose face image has been replaced with the face image of user U. is shown as M+U.

As shown in FIG. 3, the information providing device 100 acquires a multi-view face image of the user U's face from the user U's terminal device 10 via the network N (see FIG. 4). Step S21). For example, the information providing device 100 acquires, from the terminal device 10 of the user U, a multi-view face image of the face of the user U who views the posted multi-view image from a plurality of viewpoints.

Next, the information providing device 100 identifies the face of the photographed person (photographed model) from the multi-view image to be viewed by the user U (step S22). For example, the information providing device 100 identifies the face of the photographed person (photographed model) for each viewpoint of the multi-view image. In this embodiment, the multi-view image to be viewed is a multi-view image that includes the face of a photographed person (photographed model) who is different from the user U (user). Note that, in reality, it may be a multi-view image that includes the face of the user U (user) himself/herself.

Next, the information providing device 100 identifies the viewpoints at the time of shooting from the multi-view image to be viewed (step S23). For example, the information providing device 100 identifies the position and angle of the face of the person being photographed (model) for each viewpoint of the multi-view image.

Next, the information providing device 100 changes the face of the person to be photographed in the multi-view image to be viewed to the face of the user U (user) who is the viewer, according to the viewpoint at the time of photography (step S24 ). For example, the information providing device 100 displays the face of the photographed person (photographed model) of the user according to the position and angle of the photographed person's (photographed model) face for each viewpoint of the multi-view image. Generate an image with the face replaced. At this time, the information providing apparatus 100 displays the face of the person in the multi-view image to be viewed in the face of the user U (user) who is the viewer, in a form as natural as possible (with as little discomfort as possible). ). In addition, the information providing device 100 collectively displays the face of each photographed person (photographed model) for the user U (user) according to the viewpoint at the time of photographing each of a plurality of multi-view images that are viewed at the same time. Convert to face.

Next, when changing the face of the photographed person (photographed model) in the multi-view image to the face of the user U (user) who is the viewer, the information providing apparatus 100 changes the face of the photographed person (photographed model) of the multi-view image to the face of the user U (user) who is the viewer. The height of the person to be photographed (photographed model) is adjusted (step S25). That is, the information providing apparatus 100 may change not only the face but also the height of the photographed person (photographed model) according to the user. For example, the information providing apparatus 100 may change the height of the photographed person (photographed model) in the multi-view image to the height of the user. Further, the information providing device 100 may adjust the height of the person to be photographed based on the face of the person to be photographed (photographed model) in the multi-view image and the face of the user. Further, the information providing device 100 may adjust the height of the person to be photographed according to the background or the size of the object to be photographed together with the person to be photographed.

In addition, when the face of the person (photographed model) in the multi-view image is changed to the face of the user, or when the person (photographed model) in the multi-view image is the user himself, the information providing device 100 may change the hairstyle, hair color (including shade), facial expression, etc. of the user in the image by image processing and editing, etc. For example, long hair may be changed to short hair, black hair to brown hair, or the corners of the eyes and mouth may be raised and lowered. Alternatively, the information providing device 100 may convert the current facial image (such as the converted facial image of the user) displayed in the multi-view image into a facial image with a different hairstyle, hair color (including shade), facial expression, etc., in response to the user's instructions, etc.

Next, the information providing device 100 displays the converted multi-view image on the terminal device 10 of the user U (step S26) via the network N (see FIG. 4).

Note that in the above explanation, the information providing device 100 converts the face of the person to be photographed in the multi-view image into the face of the user U who is the viewer, but in reality, the information providing device 100 converts the face of the person to be photographed in the multi-view image It's not limited to the face. The information providing device 100 may convert the face of any person other than the face of the user U who is the viewer. For example, the information providing device 100 may use the face of a friend of the user U who will be the recipient of a gift when purchasing a product that corresponds to the tagging target (annotation target) among the photographic targets included in the multi-view image. .

Further, in the above description, the information providing apparatus 100 converts the face of the person to be photographed in the multi-view image into the face of another person, but in reality, the information providing apparatus 100 is not limited to the face of another person. The information providing device 100 may convert not only the face of another person but also another face of the same person. For example, if the person to be photographed in a certain multi-view image is the user U who is the viewer (such as when the person to be photographed in the multi-view image and the viewer are the same person), the information providing device 100 may The face of the user U who is the subject of the viewpoint image may be converted to another face of the user U.

Further, the information providing apparatus 100 may transform the face of the person to be photographed in the multi-view image to be viewed using a technique such as deep fake. Additionally, the information providing device 100 generates a multi-view image of the face to be replaced using a known technique that generates a multi-view image from a single-view face image, and uses this to convert the face in the image. You may.

Note that each of the above processes may be performed by the terminal device 10 instead of the information providing device 100 using functions such as an application. That is, the processing may be completed on the terminal device 10.

[2. Configuration example of information processing system]
Next, the configuration of the information processing system 1 including the information providing apparatus 100 according to the embodiment will be described using FIG. 4. FIG. 4 is a diagram illustrating a configuration example of the information processing system 1 according to the embodiment. As shown in FIG. 4, the information processing system 1 according to the embodiment includes a terminal device 10 and an information providing device 100. These various devices are connected via a network N so that they can communicate by wire or wirelessly. The network N is, for example, a LAN (Local Area Network) or a WAN (Wide Area Network) such as the Internet.

Furthermore, the number of devices included in the information processing system 1 shown in FIG. 4 is not limited to what is illustrated. For example, in FIG. 4, only one terminal device 10 is shown for simplification of illustration, but this is just an example and is not limited, and there may be two or more terminal devices.

The terminal device 10 is an information processing device used by the user U. For example, the terminal device 10 may be a smart device such as a smartphone or a tablet terminal, a feature phone, a PC (Personal Computer), a PDA (Personal Digital Assistant), a game console or AV device with a communication function, a car navigation system, a smart watch, or the like. These include wearable devices such as head-mounted displays, smart glasses, etc.

The terminal device 10 also supports wireless communication networks such as LTE (Long Term Evolution), 4G (4th Generation), and 5G (5th Generation), Bluetooth (registered trademark), and wireless LAN (Local It is possible to connect to a network N via short-distance wireless communication such as a local area network) and communicate with the information providing device 100.

The information providing device 100 is, for example, a PC, a server device, a mainframe, a workstation, etc. The information providing device 100 may be realized by cloud computing.

[3. Configuration example of terminal device]
Next, the configuration of the terminal device 10 will be explained using FIG. 5. FIG. 5 is a diagram showing a configuration example of the terminal device 10. As shown in FIG. As shown in FIG. 5, the terminal device 10 includes a communication section 11, a display section 12, an input section 13, a positioning section 14, an imaging section 15, a sensor section 20, a control section 30 (controller), A storage unit 40 is provided.

(Communication Department 11)
The communication unit 11 is connected to a network N (see FIG. 4) by wire or wirelessly, and transmits and receives information to and from the information providing device 100 via the network N. For example, the communication unit 11 is realized by a NIC (Network Interface Card), an antenna, or the like.

(Display section 12)
The display unit 12 is a display device that displays various information such as position information. For example, the display unit 12 is a liquid crystal display (LCD) or an organic electro-luminescent display (EL display). Further, the display unit 12 is a touch panel type display, but is not limited to this.

(Input section 13)
The input unit 13 is an input device that receives various operations from the user U. For example, the input unit 13 includes buttons for inputting characters, numbers, and the like. Note that the input unit 13 may be an input/output port (I/O port), a USB (Universal Serial Bus) port, or the like. Further, when the display section 12 is a touch panel display, a part of the display section 12 functions as the input section 13. Further, the input unit 13 may be a microphone or the like that receives voice input from the user U. The microphone may be wireless.

(Positioning unit 14)
The positioning unit 14 receives signals (radio waves) transmitted from satellites of a GPS (Global Positioning System), and acquires position information (e.g., latitude and longitude) indicating the current position of the terminal device 10 based on the received signals. That is, the positioning unit 14 measures the position of the terminal device 10. Note that the GPS is merely an example of a GNSS (Global Navigation Satellite System).

Further, the positioning unit 14 can measure the position using various methods other than GPS. For example, the positioning unit 14 may use various communication functions of the terminal device 10 to measure the position as an auxiliary positioning means for position correction and the like, as described below.

(Imaging unit 15)
The imaging unit 15 is an image sensor (camera) that photographs a subject. For example, the imaging unit 15 is a CMOS image sensor, a CCD image sensor, or the like. Note that the imaging unit 15 is not limited to a built-in camera, and may be an external camera such as a wireless camera that can communicate with the terminal device 10 or a web camera.

(Wi-Fi positioning)
For example, the positioning unit 14 positions the terminal device 10 using the Wi-Fi (registered trademark) communication function of the terminal device 10 or the communication network provided by each communication company. Specifically, the positioning unit 14 performs Wi-Fi communication, etc., and determines the position of the terminal device 10 by measuring the distance to nearby base stations and access points.

(Beacon positioning)
Further, the positioning unit 14 may use the Bluetooth (registered trademark) function of the terminal device 10 to measure the position. For example, the positioning unit 14 measures the position of the terminal device 10 by connecting to a beacon transmitter connected by a Bluetooth (registered trademark) function.

(geomagnetic positioning)
Furthermore, the positioning unit 14 positions the terminal device 10 based on the geomagnetic pattern of the structure measured in advance and the geomagnetic sensor included in the terminal device 10 .

(RFID positioning)
Furthermore, for example, if the terminal device 10 has a function of an RFID (Radio Frequency Identification) tag equivalent to a contactless IC card used at station ticket gates, in stores, etc., or has a function of reading an RFID tag, the location of use is recorded together with information on a payment or the like made by the terminal device 10. The positioning unit 14 may obtain such information to measure the location of the terminal device 10. Furthermore, the location may be measured by an optical sensor, an infrared sensor, or the like provided in the terminal device 10.

The positioning unit 14 may position the terminal device 10 using one or a combination of the above-mentioned positioning means, if necessary.

(sensor section 20)
The sensor unit 20 includes various sensors mounted on or connected to the terminal device 10. Note that the connection may be a wired connection or a wireless connection. For example, the sensors may be a detection device other than the terminal device 10, such as a wearable device or a wireless device. In the example shown in FIG. 5, the sensor section 20 includes an acceleration sensor 21, a gyro sensor 22, an atmospheric pressure sensor 23, an air temperature sensor 24, a sound sensor 25, an optical sensor 26, and a magnetic sensor 27.

The sensors 21 to 27 described above are merely examples and are not limiting. That is, the sensor unit 20 may be configured to include some of the sensors 21 to 27, or may include other sensors such as a humidity sensor in addition to or instead of the sensors 21 to 27. The imaging unit 15 is also a type of image sensor.

The acceleration sensor 21 is, for example, a three-axis acceleration sensor, and detects the physical movement of the terminal device 10, such as the moving direction, speed, and acceleration of the terminal device 10. The gyro sensor 22 detects physical movements of the terminal device 10 such as tilt in three axes directions based on the angular velocity of the terminal device 10 and the like. The atmospheric pressure sensor 23 detects the atmospheric pressure around the terminal device 10, for example.

Since the terminal device 10 is equipped with the above-mentioned acceleration sensor 21, gyro sensor 22, atmospheric pressure sensor 23, etc., it is possible to implement technologies such as pedestrian autonomous navigation (PDR) using these sensors 21 to 23, etc. It becomes possible to measure the position of the terminal device 10 using the . This makes it possible to obtain indoor position information that is difficult to obtain using positioning systems such as GPS.

For example, a pedometer using the acceleration sensor 21 can calculate the number of steps, walking speed, and distance walked. Furthermore, by using the gyro sensor 22, it is possible to know the direction of travel of the user U, the direction of the line of sight, and the inclination of the user's body. Further, from the atmospheric pressure detected by the atmospheric pressure sensor 23, it is also possible to know the altitude and the number of floors where the terminal device 10 of the user U is located.

The temperature sensor 24 detects, for example, the temperature around the terminal device 10. The sound sensor 25 detects, for example, sounds around the terminal device 10. The optical sensor 26 detects the illuminance around the terminal device 10 . The magnetic sensor 27 detects, for example, the earth's magnetism around the terminal device 10. The imaging unit 15 captures an image of the surroundings of the terminal device 10.

The above-mentioned atmospheric pressure sensor 23, temperature sensor 24, sound sensor 25, optical sensor 26, and imaging unit 15 each detect atmospheric pressure, temperature, sound, and illuminance, and capture images of the surroundings, thereby providing information to the terminal device 10. It is possible to detect the surrounding environment and situation. Furthermore, it is possible to improve the accuracy of the location information of the terminal device 10 based on the environment and situation around the terminal device 10.

(Control unit 30)
The control unit 30 includes, for example, a microcomputer having a CPU (Central Processing Unit), a ROM (Read Only Memory), a RAM, an input/output port, etc., and various circuits. Further, the control unit 30 may be configured with hardware such as an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array). The control section 30 includes a transmitting section 31 , a receiving section 32 , a processing section 33 , a guide display section 34 , a guide changing section 35 , an imaging determination section 36 , a recognition section 37 , and a notification section 38 . Note that, in reality, the processing section 33 may include a guide display section 34, a guide change section 35, a shooting determination section 36, a recognition section 37, and a notification section 38.

(Transmission unit 31)
The transmitting unit 31 receives, for example, various information input by the user U using the input unit 13, various information detected by the sensors 21 to 27 mounted on or connected to the terminal device 10, and information measured by the positioning unit 14. The location information of the terminal device 10 and the like can be transmitted to the information providing device 100 via the communication unit 11.

(Receiving unit 32)
The receiving unit 32 can receive various information provided from the information providing device 100 and requests for various information from the information providing device 100 via the communication unit 11 .

(Processing unit 33)
The processing unit 33 controls the entire terminal device 10, including the display unit 12 and the like. For example, the processing unit 33 can output various types of information transmitted by the transmitting unit 31 and various types of information received by the receiving unit 32 from the information providing device 100 to the display unit 12 for display.

(Guide display section 34)
The guide display unit 34 displays a predetermined photographing guide on the screen when photographing a multi-view image. For example, the guide display unit 34 displays on the screen, as a photography guide, a guide regarding at least one of the pose, facial expression, and holding style of the person to be photographed (photographed model) among the subjects. Further, the guide display unit 34 displays a silhouette, outline, or semi-transparent display of the subject on the screen as a photography guide.

Further, the guide display unit 34 displays a photographing guide superimposed on the subject when photographing the subject. Further, the guide display unit 34 specifies the pose of the photographed person among the subjects in the first photographed image, and selects and displays a photographic guide corresponding to the specified pose.

(Guide change unit 35)
The guide change unit 35 changes the shooting guide displayed on the screen according to the viewpoint every time the viewpoint during shooting is moved. The guide change unit 35 may also change the shooting guide step by step as the viewpoint during shooting is moved.

(Photographing determination unit 36)
The photographing determination unit 36 automatically photographs the subject using the imaging unit 15 when the subject matches the photographing guide. Further, when the shooting of the multi-view image is completed, the shooting determination unit 36 posts the multi-view image to the information providing device 100 using the transmitting unit 31 . At this time, the photographing determination unit 36 may generate a multi-view image from the photographed images.

(Recognition unit 37)
The recognition unit 37 recognizes a photographic subject included in an image when an image constituting a multi-view image is photographed. For example, the recognition unit 37 recognizes a photographic subject included in an image by image recognition or machine learning. Furthermore, the recognition unit 37 recognizes the photographic subject included in the image every time the image is photographed. Further, the recognition unit 37 recognizes each of the plurality of photographic targets included in the image every time a photograph is taken. Moreover, the recognition unit 37 recognizes a photographic subject that is hidden behind another photographic subject and becomes invisible every time a photograph is taken.

(Notification unit 38)
When the subject is out of alignment with the photography guide, the notification unit 38 notifies the photographer of the portion where the subject is out of alignment with the photography guide. The subject may be the photographer himself. In other words, the photographer and the person being photographed may be the same person.

Furthermore, the notification unit 38 notifies the user of the photographic subject recognized by the recognition unit 37. For example, the notification unit 38 notifies the user of the recognized photographic subject every time a photograph is taken. Furthermore, the notification unit 38 notifies the user of each of the plurality of recognized photographic targets each time a photograph is taken. Furthermore, the notification unit 38 notifies the user of the number of images of the recognized object to be photographed each time a photograph is taken. Furthermore, the notification unit 38 notifies the user of the remaining number of images constituting the multi-view image, up to the required number, each time a photograph is taken. Moreover, the notification unit 38 notifies the user of a photographic subject that is hidden behind another photographic subject and is no longer visible every time a photograph is taken.

At this time, the notification unit 38 may notify the user of the identified shooting target by voice. Further, the notification unit 38 may notify the user of the identified photographic subject through a screen display.

(Storage unit 40)
The storage unit 40 is realized by, for example, a semiconductor memory element such as a RAM (Random Access Memory) or a flash memory, or a storage device such as an HDD (Hard Disk Drive), an SSD (Solid State Drive), or an optical disk. Ru. The storage unit 40 stores various programs, various data, and the like.

[4. Configuration example of information providing device]
Next, the configuration of the information providing apparatus 100 according to the embodiment will be described using FIG. 6. FIG. 6 is a diagram showing a configuration example of the information providing apparatus 100 according to the embodiment. As shown in FIG. 6, the information providing device 100 includes a communication section 110, a storage section 120, and a control section 130.

(Communication Department 110)
The communication unit 110 is realized by, for example, a NIC (Network Interface Card). Further, the communication unit 110 is connected to the network N (see FIG. 4) by wire or wirelessly.

(Memory unit 120)
The storage unit 120 is realized by, for example, a semiconductor memory element such as a random access memory (RAM) or a flash memory, or a storage device such as a HDD, an SSD, an optical disk, etc. As shown in FIG. 6 , the storage unit 120 has a user information database 121, a history information database 122, and an image information database 123.

(User information database 121)
The user information database 121 stores user information regarding the user U. For example, the user information database 121 stores various information such as user U's attributes. FIG. 7 is a diagram showing an example of the user information database 121. In the example shown in FIG. 7, the user information database 121 has items such as "user ID (identifier),""age,""gender,""home,""worklocation," and "interest."

“User ID” indicates identification information for identifying user U. Note that the "user ID" may be user U's contact information (telephone number, email address, etc.), or may be identification information for identifying user U's terminal device 10.

Moreover, "age" indicates the age of the user U identified by the user ID. Note that the "age" may be information indicating the specific age of the user U (for example, 35 years old, etc.), or may be information indicating the age of the user U (for example, 30s, etc.) . Alternatively, the "age" may be information indicating the date of birth of the user U, or may be information indicating the generation of the user U (for example, born in the 1980s). Furthermore, “gender” indicates the gender of the user U identified by the user ID.

Moreover, "home" indicates the location information of the home of the user U identified by the user ID. In the example shown in FIG. 7, "home" is illustrated as an abstract code such as "LC11", but it may also be latitude/longitude information or the like. Furthermore, for example, "home" may be a region name or address.

Moreover, "work place" indicates the location information of the work place (school in the case of a student) of the user U identified by the user ID. In the example shown in FIG. 7, the "work location" is illustrated as an abstract code such as "LC12," but it may also be latitude and longitude information. Further, for example, the "work location" may be a region name or address.

Moreover, "interest" indicates the interest of the user U identified by the user ID. That is, "interest" indicates an object in which the user U identified by the user ID has a high interest. For example, "interest" may be a search query (keyword) that the user U inputs into a search engine. In the example shown in FIG. 7, one "interest" is shown for each user U, but there may be a plurality of "interests".

For example, in the example shown in FIG. 7, the age of the user U identified by the user ID "U1" is "20s", and the gender is "male". Further, for example, the user U identified by the user ID "U1" indicates that his home is "LC11". Further, for example, the user U identified by the user ID "U1" indicates that the work location is "LC12". Further, for example, the user U identified by the user ID "U1" indicates that he is interested in "sports."

Here, in the example shown in FIG. 7, abstract values such as "U1", "LC11", and "LC12" are used for illustration, but "U1", "LC11", and "LC12" have specific values. It is assumed that information such as character strings and numerical values is stored. Below, abstract values may be illustrated in diagrams related to other information as well.

Note that the user information database 121 is not limited to the above, and may store various information depending on the purpose. For example, the user information database 121 may store various information regarding the terminal device 10 of the user U. In addition, the user information database 121 includes information such as demographic (demographic attributes), psychographic (psychological attributes), geographic (geographical attributes), behavioral (behavioral attributes), etc. of user U. Information regarding attributes may also be stored. For example, the user information database 121 includes name, family structure, place of birth (locality), occupation, position, income, qualifications, type of residence (single-family house, condominium, etc.), presence or absence of a car, commuting/commuting time, commuting/commuting. Memorizes information such as routes, commuter pass sections (stations, lines, etc.), frequently used stations (other than the station closest to your home or work), lessons (location, time zone, etc.), hobbies, interests, lifestyle, etc. You can.

(History information database 122)
The history information database 122 stores various information related to history information (log data) indicating the actions of the user U. FIG. 8 is a diagram showing an example of the history information database 122. In the example shown in FIG. 8, the history information database 122 has items such as "user ID", "location history", "search history", "browsing history", "purchase history", and "posting history".

“User ID” indicates identification information for identifying user U. Further, “position history” indicates a position history that is a history of the user U's position and movement. Further, “search history” indicates a search history that is a history of search queries input by the user U. In addition, “browsing history” indicates a browsing history that is a history of contents that the user U has viewed. Moreover, "purchase history" indicates a purchase history that is a history of purchases by user U. Moreover, "posting history" indicates a posting history that is a history of postings by user U. Note that the "posting history" may include questions regarding user U's belongings.

For example, in the example shown in FIG. 8, user U identified by user ID "U1" moves as shown in "location history #1," searches as shown in "search history #1," views content as shown in "browsing history #1," purchases specific products at specific stores as shown in "purchase history #1," and posts as shown in "posting history."

In the example shown in FIG. 8, abstract values such as "U1", "Location History #1", "Search History #1", "Browse History #1", "Purchase History #1", and "Post History #1" are used for illustration, but specific information such as character strings and numerical values is stored in "U1", "Location History #1", "Search History #1", "Browse History #1", "Purchase History #1", and "Post History #1".

Note that the history information database 122 is not limited to the above, and may store various information depending on the purpose. For example, the history information database 122 may store the user U's usage history of a predetermined service. Further, the history information database 122 may store the user U's visit history to a physical store, visit history to a facility, or the like. Further, the history information database 122 may store the payment history of user U's payment using the terminal device 10 (electronic payment).

(Image information database 123)
The image information database 123 stores various information regarding multi-view images. FIG. 9 is a diagram showing an example of the image information database 123. In the example shown in FIG. 9, the image information database 123 includes "multi-view image", "image", "viewpoint", "shooting target", "position", "annotation target", "tag", "face position It has items such as ”.

“Multi-view image” indicates identification information for identifying a multi-view image. Note that, in reality, it may be a storage location or location of data of a multi-view image. Further, "image" indicates identification information for identifying images forming a multi-view image. Note that, in reality, it may be a storage location or location of data of images forming a multi-view image.

Moreover, "viewpoint" indicates the viewpoint when images forming the multi-view image are photographed. In other words, the viewpoint indicates the position and angle of the photographic subject included in the images constituting the multi-view image.

Further, "photographing target" indicates a photographing target included in images forming a multi-view image. In other words, it indicates a photographic subject photographed as a subject. For example, it shows the classification (category) of the photographic object, specific product name, product code, etc. Furthermore, the subject to be photographed may be a person to be photographed (person). Furthermore, there may be a plurality of objects to be photographed. That is, one image may include a plurality of shooting targets. For example, a person to be photographed and two fashion items worn by the person to be photographed may be photographed.

Further, "position" indicates the position of the photographing target within the multi-view image (position within the image). In this embodiment, the three-dimensional position of the photographing target in each image constituting the multi-view image is shown. The three-dimensional position of the object to be imaged may be an absolute position such as coordinates within an image, or may be a relative position from a reference point or another object to be imaged. Further, the three-dimensional position is only an example.

In addition, the "annotation target" refers to the target of tagging (annotation target) among the photographed subjects (candidates for annotation targets). The tagging target may be selected by the user, or may be automatically determined by pre-setting, machine learning, etc.

Further, "tag" indicates a tag attached to a tagging target (annotation target). For example, the information may be identification information for identifying a tag registered in advance, or may be the contents of the tag. For example, the information may be information regarding a product page of a product (fashion item) to be tagged (annotated). At this time, image recognition or machine learning is used to search for similar images of the product from each product page on e-commerce sites such as fashion mail order sites (e.g. "ZOZOTOWN" (registered trademark)), and based on the search results, the products of the product are The page may be automatically identified.

Further, "position of face" indicates the position of the face of the person to be photographed within the multi-view image (position of the face within the image). In this embodiment, the three-dimensional position of the face of the photographed person in each image constituting the multi-view image is shown. The position of the face may be an absolute position such as coordinates within an image, or may be a relative position from a reference point or other imaging target. It may also be the outline of the face or the position of each part of the face (eyebrows, eyes, ears, nose, mouth, chin, etc.).

For example, in the example shown in FIG. 9, the image "A1" constituting the multi-view image "A" is photographed at "viewpoint #A1", and the photographed object "bag" is located at "position #A1" in the image. Yes, it has been selected as an "annotation target" (tagging target), a link to the website "Site #W1" related to the target bag is added as a tag, and the position of the photographed person's face in the image is " face position #A1".

Here, in the example shown in FIG. 9, abstract values such as "A", "A1", "Viewpoint #A1", "Position #A1", "Site #W1" and "Face position #A1" are used. As shown in the figure, "A", "A1", "Viewpoint #A1", "Position #A1", "Site #W1" and "Face position #A1" contain information such as specific character strings and numerical values. shall be remembered.

Note that the image information database 123 is not limited to the above, and may store various information depending on the purpose. For example, the image information database 123 may store identification information for identifying the poster or viewer of the multi-view image. Further, the image information database 123 may store detailed information regarding a subject (a subject to be photographed, a person to be photographed, etc.). The image information database 123 may also store a list of candidate tags. Further, the image information database 123 may store information regarding the shooting location and the shooting date and time. The image information database 123 may also store information regarding the photographing device used for photographing (user's terminal device, etc.) and the photographing environment.

(Control unit 130)
Returning to FIG. 6, the explanation will be continued. The control unit 130 is a controller, and uses, for example, a CPU (Central Processing Unit), an MPU (Micro Processing Unit), an ASIC (Application Specific Integrated Circuit), or an FPGA (Field Programmable Gate Array) to control the information providing apparatus 100. This is achieved by executing various programs (corresponding to an example of an information processing program) stored in an internal storage device using a storage area such as a RAM as a work area. In the example shown in FIG. 6, the control unit 130 includes an acquisition unit 131, an identification unit 132, an estimation unit 133, a tag assignment unit 134, a tag change unit 135, an image conversion unit 136, and a provision unit 137. have

(Acquisition unit 131)
The acquisition unit 131 acquires a search query input by the user U. For example, when the user U inputs a search query into a search engine or the like and performs a keyword search, the acquisition unit 131 acquires the search query via the communication unit 110. That is, the acquisition unit 131 acquires, via the communication unit 110, a keyword input by the user U into a search window of a search engine, site, or application.

The acquisition unit 131 also acquires user information about the user U via the communication unit 110. For example, the acquisition unit 131 acquires identification information (such as a user ID) indicating the user U, location information of the user U, attribute information of the user U, and the like from the terminal device 10 of the user U. The acquisition unit 131 may also acquire identification information indicating the user U, attribute information of the user U, and the like when registering the user U. Then, the acquisition unit 131 registers the user information in the user information database 121 of the storage unit 120.

The acquisition unit 131 also acquires various types of history information (log data) indicating the behavior of the user U via the communication unit 110. For example, the acquisition unit 131 acquires various types of history information indicating the behavior of the user U from the terminal device 10 of the user U, or from various servers, etc. based on the user ID, etc. Then, the acquisition unit 131 registers the various types of history information in the history information database 122 of the storage unit 120.

Further, the acquisition unit 131 acquires a multi-view image from the user U who is the poster or the viewer via the communication unit 110. For example, the acquisition unit 131 acquires a multi-view image taken by the poster from the terminal device 10 of the user U who is the poster. The acquisition unit 131 also acquires multi-view images taken by other posters designated by the user U who is the viewer.

The acquisition unit 131 also acquires a multi-view image of the face of another person different from the photographed person via the communication unit 110. For example, the acquisition unit 131 acquires a multi-view face image in which the face of the other person is photographed from multiple viewpoints. The face of the other person may be the face of the user U who is the viewer. In this embodiment, the acquisition unit 131 acquires a multi-view image of the viewer's face via the communication unit 110. Note that the acquisition unit 131 may acquire a multi-view image of the viewer's face in advance, or may acquire it at the time of viewing. Furthermore, the multi-view image of the viewer's face may be a multi-view image that includes at least the viewer's face. For example, the acquisition unit 131 acquires a multi-view face image in which the face of the user U who is the viewer is photographed from multiple viewpoints.

The acquisition unit 131 also functions as a reception unit that receives a selection of a tagging target from a poster via the communication unit 110. For example, the acquisition unit 131 (reception unit) receives from the poster a selection of a tagging target and a designation of a web page to be associated with the tagging target.

(Specific unit 132)
The specifying unit 132 specifies a shooting target included in the multi-view image. The identifying unit 132 then identifies an annotation tagging target from among the photographic targets included in the multi-view image. At this time, the specifying unit 132 uses image recognition or machine learning to specify and classify the shooting target for each viewpoint image of the multi-view image.

For example, the specifying unit 132 specifies an annotation tagging target from among the shooting targets included in the multi-view image in response to a tagging target selection from a poster. Alternatively, the specifying unit 132 uses image recognition or machine learning to specify a target for annotation tagging from among photographic targets included in the multi-view image.

The identification unit 132 also searches for similar images to the image to be tagged from a plurality of web pages on the network using image recognition or machine learning, and associates the web page containing the similar image with the tagged target as a web page. Automatically identify.

Further, the specifying unit 132 specifies the face of the person to be photographed in the multi-view image that is to be viewed by the user U. For example, the specifying unit 132 specifies the face of the person to be photographed in the multi-view image that is to be viewed by the user U, and the viewpoint at the time the multi-view image was taken. At this time, the specifying unit 132 may specify the face of the person to be photographed in each of the plurality of multi-view images that are viewed at the same time, and the viewpoint at the time of shooting each of the plurality of multi-view images.

(Estimation unit 133)
The estimation unit 133 estimates the position of the shooting target in the multi-view image. That is, the estimation unit 133 estimates the position of the tagging target selected from the shooting targets in the multi-view image. In this embodiment, the estimation unit 133 estimates the three-dimensional position of the tagging target in the multi-view image. The three-dimensional position in the multi-view image may be an absolute position such as coordinates in the image, or may be a relative position from a reference point or another imaging target. Note that, in practice, the identification unit 132 may function as the estimation unit 133. At this time, the identification unit 132 identifies the three-dimensional position of the tagging target in the multi-view image.

(Tagging section 134)
The tagging unit 134 adds a tag according to the position of the tag target. For example, the tagging unit 134 tags the tagging target according to the three-dimensional position of the tagging target. As a result, the assigned tag will be displayed on the screen. Furthermore, when tagging a target to be tagged, the tagging unit 134 tags the tag so that the tag does not overlap with other targets or tags.

In addition, when tagging the target of tagging, the tagging unit 134 tags the target of tagging if the target of tagging is not hidden by another target. Note that when tagging a tag to a target, the tagging unit 134 is configured to tag the target to be tagged, even if the target to be tagged is hidden by another target. If the display priority is higher than that of the tag, the tag may be added to the tagged object without adding the tag to other objects.

(tag change unit 135)
When the position of the tag target in the screen changes due to a change in the viewpoint of the multi-view image, the tag change unit 135 changes the display position of the tag in accordance with the change in the position of the tag target. Furthermore, the tag changing unit 135 places the tag at a position where the positional relationship between the tag and the tag is maintained even if the viewpoint of the multi-view image is changed. Note that, in reality, the tag adding section 134 may function as the tag changing section 135. In this case, the tagging unit 134 tags the target of tagging in accordance with the change in the position of the target of tagging each time the viewpoint of the multi-view image is changed.

(Image conversion unit 136)
The image conversion unit 136 converts the face of the person to be photographed in the multi-view image into the face of another person for each image forming the multi-view image. For example, the image converting unit 136 converts the face of the person to be photographed in the multi-view image into the face of another person, in accordance with the viewpoint at the time of shooting the multi-view image, for each image forming the multi-view image. At this time, the image conversion unit 136 may generate a new multi-view image in which the face of the person to be photographed in the multi-view image is replaced with the face of another person in accordance with the viewpoint at the time the multi-view image was taken.

In this embodiment, the image conversion unit 136 converts the face of the person photographed in the multi-viewpoint image into the face of the user U who is the viewer, for each image constituting the multi-viewpoint image. For example, the image conversion unit 136 converts the face of the person photographed in the multi-viewpoint image into the face of the user U, for each image constituting the multi-viewpoint image, in accordance with the viewpoint at the time the multi-viewpoint image was captured. At this time, the image conversion unit 136 may generate a new multi-viewpoint image in which the face of the person photographed in the multi-viewpoint image is replaced with the face of the user U in accordance with the viewpoint at the time the multi-viewpoint image was captured.

In addition, the image conversion unit 136 converts the height of the person in the multi-view image when converting the face of the person in the multi-view image into the face of another person in accordance with the viewpoint at the time of shooting the multi-view image. Adjust to suit the height of another person. At this time, the image conversion unit 136 may adjust the height of the person to be photographed based on the face of the person to be photographed and the face of another person.

In the present embodiment, the image conversion unit 136 converts the face of the person to be photographed in the multi-view image into the face of the user U, who is the viewer, in accordance with the viewpoint at the time of shooting the multi-view image. The height of the person to be photographed in the viewpoint image is adjusted to match the height of the user U. At this time, the image conversion unit 136 may adjust the height of the person to be photographed based on the face of the person to be photographed and the face of user U.

Further, the image converting unit 136 converts the face of the photographed person in the multi-view image into the face of another person in accordance with the viewpoint at the time of shooting the multi-view image, and also converts the facial expression of the other person in the converted image. change. Further, the image conversion unit 136 converts the face of the photographed person in the multi-view image to the face of another person in accordance with the viewpoint at the time of shooting the multi-view image, and also changes the hairstyle of the other person in the converted image. . In addition, the image conversion unit 136 converts the face of the photographed person in the multi-view image to the face of another person in accordance with the viewpoint at the time of shooting the multi-view image, and also converts the color of the hair of the other person in the converted image ( (including shading).

In this embodiment, the image conversion unit 136 performs conversion when converting the face of the photographed person in the multi-view image into the face of the user U who is the viewer, in accordance with the viewpoint at the time of shooting the multi-view image. The facial expression of the subsequent user U is changed. In addition, when converting the face of the person to be photographed in the multi-view image into the face of the user U in accordance with the viewpoint at the time of shooting the multi-view image, the image conversion unit 136 also converts the hairstyle of the user U after the conversion into the face of the user U. change. In addition, when converting the face of the person to be photographed in the multi-view image to the face of the user U in accordance with the viewpoint at the time of shooting the multi-view image, the image conversion unit 136 also converts the user U's hair after the conversion. Change the color of.

For example, the image conversion unit 136 may change user U's hairstyle, hair color (including shading), facial expression, etc. in the image by image processing and editing. Alternatively, the image conversion unit 136 converts the current face image displayed in the multi-view image (the face image of the user U after conversion, etc.) according to an instruction from the user U to the hairstyle or hair color ( (including shading), facial expressions, etc. may be converted into facial images with different facial expressions.

Further, the image converting unit 136 collectively converts the face of each person to be photographed into the face of another person in accordance with the viewpoint at the time of photographing each of a plurality of multi-view images that are viewed at the same time. In the present embodiment, the image conversion unit 136 collectively converts the faces of the photographed persons to the user U who is the viewer, according to the viewpoint at the time of shooting each of the plurality of multi-view images that are viewed at the same time. Convert to face.

(Providing Department 137)
The providing unit 137 provides the user U with the multi-view image converted into another person's face. For example, the providing unit 137 provides the user U with the generated new multi-view image. Further, the providing unit 137 provides the user U with each of the plurality of multi-view images after being converted into another person's face.

In the present embodiment, the providing unit 137 provides the user U, who is the viewer, with a multi-view image converted into the face of the user U. For example, the providing unit 137 provides the user U with the generated new multi-view image. Further, the providing unit 137 provides the user U with each of the plurality of multi-view images after the conversion into the user's U face.

[5. Processing procedure]
Next, a processing procedure by the terminal device 10 and the information providing device 100 according to the embodiment will be described using FIG. 10. FIG. 10 is a flowchart showing the processing procedure according to the embodiment. Note that the processing procedure shown below is repeatedly executed by the control unit 30 of the terminal device 10 and the control unit 130 of the information providing device 100. Further, the terminal device 10 and the information providing device 100 cooperate.

As shown in FIG. 10, the guide display unit 34 of the terminal device 10 displays a predetermined shooting guide on the screen when shooting a multi-view image (step S101).

Subsequently, when the subject matches the photographing guide, the photographing determining section 36 of the terminal device 10 automatically photographs the subject using the imaging section 15 (step S102). At this time, if the subject is misaligned with the photography guide, the notification unit 38 of the terminal device 10 notifies the photographer of the location where the subject is misaligned with the photography guide. At this time, the notification unit 38 may notify the user of the specified object to be photographed by voice, or may notify the user of the specified object to be photographed by displaying the screen.

Subsequently, the recognition unit 37 of the terminal device 10 recognizes the photographic subject included in the photographed image. The notification unit 38 of the terminal device 10 notifies the user of the photographic subject recognized by the recognition unit 37 (step S103). For example, each time a photograph is taken, the notification unit 38 informs the user of the number of images of the recognized photographic target, the number of remaining images up to the required number of images constituting a multi-view image, and the number of images of the recognized photographic target that are hidden behind other photographic targets. Notify users of the target etc.

Subsequently, the guide changing unit 35 of the terminal device 10 changes the photographing guide displayed on the screen according to the viewpoint each time the viewpoint at the time of photographing is moved (step S104). At this time, the guide changing unit 35 may change the photographing guide in stages as the viewpoint at the time of photographing is moved.

Subsequently, when the shooting of the multi-view image is completed, the shooting determination unit 36 of the terminal device 10 posts the multi-view image to the information providing device 100 using the transmitting unit 31 (step S105). At this time, the photographing determination unit 36 may generate a multi-view image from the photographed images. Furthermore, the photographing determination unit 36 may confirm with the photographer whether it is okay to post before posting.

Subsequently, the estimation unit 133 of the information providing device 100 estimates the position of the photographing target within the multi-view image (step S106). For example, the acquisition unit 131 of the information providing device 100 acquires multi-view images from the terminal device 10 via the communication unit 110. The specifying unit 132 of the information providing device 100 specifies a shooting target in a multi-view image. Then, the estimation unit 133 of the information providing device 100 estimates the three-dimensional position of the photographing target within the multi-view image.

Subsequently, the identifying unit 132 of the information providing apparatus 100 identifies an annotation tagging target from among the photographic targets included in the multi-view image (step S107). For example, the specifying unit 132 specifies an annotation tagging target from among the shooting targets included in the multi-view image in response to a tagging target selection from a poster. Alternatively, the specifying unit 132 uses image recognition or machine learning to specify a target for annotation tagging from among photographic targets included in the multi-view image.

Subsequently, the tagging unit 134 of the information providing device 100 adds a tag to the target of tagging (step S108). At this time, the estimation unit 133 of the information providing device 100 estimates the position of the tag target within the multi-view image. The tagging unit 134 adds a tag according to the position of the tag target. Note that the position where the tag is added may be specified in advance by the poster who posted the multi-view image.

At this time, when tagging the target to be tagged, the tagging unit 134 tags the tag so that the tag does not overlap with other targets or tags. Furthermore, when tagging the tagging target, the tagging unit 134 tags the tagging target if the tagging target is not hidden by another target. Note that when tagging a tag to a target, the tagging unit 134 is configured to tag the target of tagging even if the target of tagging is hidden by another target. If the display priority is higher than that of the tag, the tag may be added to the tag target without tags being added to other targets.

Next, the tag change unit 135 of the information providing device 100 changes the display position of the tag assigned to the tagging target in accordance with each viewpoint image of the multi-viewpoint image (step S109).
For example, when the position of the tagged target on the screen changes with a change in the viewpoint of a multi-view image, the tag change unit 135 changes the display position of the tag in accordance with the change in the position of the tagged target.

Next, the image conversion unit 136 of the information providing device 100 converts the face of the person photographed in the multi-viewpoint image into the face of the viewer, in accordance with the viewpoint at the time of photographing the multi-viewpoint image (step S110). At this time, the acquisition unit 131 of the information providing device 100 acquires the multi-viewpoint image of the viewer's face via the communication unit 110. Note that the acquisition unit 131 may acquire the multi-viewpoint image of the viewer's face in advance, or may acquire it at the time of viewing. Furthermore, the provision unit 137 of the information providing device 100 provides the multi-viewpoint image after conversion to the viewer's face to the viewer via the communication unit 110.

[6. Modified example]
The terminal device 10 and the information providing device 100 described above may be implemented in various different forms other than the above embodiments. Therefore, a modification of the embodiment will be described below.

In the above embodiment, part or all of the processing executed by the information providing device 100 may actually be executed by the terminal device 10. For example, the process may be completed stand-alone (by the terminal device 10 alone). In this case, it is assumed that the terminal device 10 has the functions of the information providing device 100 in the above embodiment. Furthermore, in the embodiment described above, since the terminal device 10 cooperates with the information providing device 100, it appears to the user U that the terminal device 10 also executes the processing of the information providing device 100. That is, from another perspective, it can be said that the terminal device 10 includes the information providing device 100.

Further, in the above embodiment, a fashion item is described as an example of a photographing target (annotation target candidate) and a tagging target (annotation target), but the present invention is not actually limited to fashion items. The object to be photographed and the object to be tagged (annotated object) may be an article that is photographed together with the user when a multi-view image is photographed. For example, it may be a wearable device worn by the user or a terminal device held in the user's hand at the time of shooting, home appliances (home appliances/electrical appliances) placed around the user at the time of shooting, or as a background. It may also be the interior of the room that was photographed together, books on a bookshelf, food and tableware in the kitchen or table, artwork, etc.

In addition, in the above embodiment, when capturing a multi-view image, only a specific item may be captured, rather than a user. For example, when adding an annotation tag to a tagging target (annotation target) in a multi-view image, the user does not need to be included in the multi-view image.

[7. effect〕
As described above, the information processing device (terminal device 10 and information providing device 100) according to the present application includes a guide display unit 34 that displays a predetermined shooting guide on the screen when shooting a multi-view image, and a guide display unit 34 that displays a predetermined shooting guide on the screen when shooting a multi-view image. The camera includes a guide changing unit 35 that changes the photographing guide displayed on the screen according to the viewpoint each time the viewpoint is moved.

The guide display unit 34 displays a guide regarding at least one of the pose, facial expression, and way of holding the photographed person among the subjects on the screen as a photographing guide.

The guide display unit 34 displays a silhouette, outline, or semi-transparent display of the subject on the screen as a photography guide.

The guide display unit 34 superimposes a shooting guide on the subject when the subject is photographed.

The guide display unit 34 specifies the pose of the photographed subject among the subjects in the first photographed image, and selects and displays a photographic guide corresponding to the specified pose.

The guide change unit 35 gradually changes the shooting guide as the viewpoint during shooting moves.

The information processing device according to the present application further includes a photography determination unit 36 that automatically takes a photograph when the subject matches the photography guide.

The information processing device according to the present application further includes a notification unit 37 that notifies the photographer of the location where the subject is misaligned with the photography guide when the subject is misaligned with the photography guide.

By using any one or a combination of the above-described processes, the information processing apparatus according to the present application can further improve the quality of service provision using multi-view images.

[8. Hardware configuration]
Furthermore, the terminal device 10 and the information providing device 100 according to the embodiments described above are realized by, for example, a computer 1000 having a configuration as shown in FIG. 11. The information providing apparatus 100 will be described below as an example. FIG. 11 is a diagram showing an example of the hardware configuration. The computer 1000 is connected to an output device 1010 and an input device 1020, and a calculation device 1030, a primary storage device 1040, a secondary storage device 1050, an output I/F (Interface) 1060, an input I/F 1070, and a network I/F 1080 are connected to a bus. 1090.

The arithmetic unit 1030 operates based on programs stored in the primary storage device 1040 and the secondary storage device 1050, programs read from the input device 1020, and performs various processes. The arithmetic device 1030 is realized by, for example, a CPU (Central Processing Unit), an MPU (Micro Processing Unit), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or the like.

The primary storage device 1040 is a memory device such as a RAM (Random Access Memory) that temporarily stores data used by the calculation device 1030 for various calculations. Further, the secondary storage device 1050 is a storage device in which data used by the calculation device 1030 for various calculations and various databases are registered, and includes a ROM (Read Only Memory), an HDD (Hard Disk Drive), and an SSD (Solid Disk Drive). This is realized using flash memory, etc. The secondary storage device 1050 may be a built-in storage or an external storage. Further, the secondary storage device 1050 may be a removable storage medium such as a USB (Universal Serial Bus) memory or an SD (Secure Digital) memory card. Further, the secondary storage device 1050 may be a cloud storage (online storage), a NAS (Network Attached Storage), a file server, or the like.

The output I/F 1060 is an interface for transmitting information to be output to an output device 1010 that outputs various types of information, such as a display, projector, printer, etc., and is realized by a connector conforming to a standard such as USB (Universal Serial Bus), DVI (Digital Visual Interface), or HDMI (registered trademark) (High Definition Multimedia Interface). The input I/F 1070 is an interface for receiving information from various input devices 1020, such as a mouse, keyboard, keypad, button, scanner, etc., and is realized by a USB, etc.

Further, the output I/F 1060 and the input I/F 1070 may be wirelessly connected to the output device 1010 and the input device 1020, respectively. That is, output device 1010 and input device 1020 may be wireless devices.

Moreover, the output device 1010 and the input device 1020 may be integrated like a touch panel. In this case, the output I/F 1060 and the input I/F 1070 may also be integrated as an input/output I/F.

The input device 1020 may be a device that reads information from, for example, an optical recording medium such as a CD (Compact Disc), a DVD (Digital Versatile Disc), or a PD (Phase change rewritable Disk), a magneto-optical recording medium such as an MO (Magneto-Optical disk), a tape medium, a magnetic recording medium, or a semiconductor memory.

Network I/F 1080 receives data from other devices via network N and sends it to computing device 1030, and also sends data generated by computing device 1030 to other devices via network N.

Arithmetic device 1030 controls output device 1010 and input device 1020 via output I/F 1060 and input I/F 1070. For example, the arithmetic device 1030 loads a program from the input device 1020 or the secondary storage device 1050 onto the primary storage device 1040, and executes the loaded program.

For example, when the computer 1000 functions as the information providing device 100, the arithmetic unit 1030 of the computer 1000 realizes the functions of the control unit 130 by executing a program loaded onto the primary storage device 1040. Further, the arithmetic device 1030 of the computer 1000 may load a program obtained from another device via the network I/F 1080 onto the primary storage device 1040, and execute the loaded program. Furthermore, the arithmetic unit 1030 of the computer 1000 may cooperate with other devices via the network I/F 1080, and may call and use program functions, data, etc. from other programs of other devices.

[9. others〕
Although the embodiments of the present application have been described above, the present invention is not limited to the contents of these embodiments. Furthermore, the above-mentioned components include those that can be easily assumed by those skilled in the art, those that are substantially the same, and those that are in a so-called equivalent range. Furthermore, the aforementioned components can be combined as appropriate. Furthermore, various omissions, substitutions, or modifications of the constituent elements can be made without departing from the gist of the embodiments described above.

Further, among the processes described in the above embodiments, all or part of the processes described as being performed automatically can be performed manually, or the processes described as being performed manually can be performed manually. All or part of this can also be performed automatically using known methods. In addition, information including the processing procedures, specific names, and various data and parameters shown in the above documents and drawings may be changed arbitrarily, unless otherwise specified. For example, the various information shown in each figure is not limited to the illustrated information.

Furthermore, each component of each device shown in the drawings is functionally conceptual, and does not necessarily need to be physically configured as shown in the drawings. In other words, the specific form of distributing and integrating each device is not limited to what is shown in the diagram, and all or part of the devices can be functionally or physically distributed or integrated in arbitrary units depending on various loads and usage conditions. Can be integrated and configured.

For example, the information providing apparatus 100 described above may be realized by a plurality of server computers, and depending on the function, it may be realized by calling an external platform etc. using an API (Application Programming Interface), network computing, etc. can be changed flexibly.

The above-described embodiments and variations can be combined as appropriate to the extent that they do not cause inconsistencies in the processing content.

Further, the above-mentioned "section, module, unit" can be read as "means", "circuit", etc. For example, the acquisition unit can be read as an acquisition means or an acquisition circuit.

1 Information processing system 10 Terminal device 34 Guide display unit 35 Guide change unit 36 Shooting determination unit 37 Recognition unit 38 Notification unit 100 Information providing device 110 Communication unit 120 Storage unit 121 User information database 122 History information database 123 Image information database 130 Control Section 131 Acquisition section 132 Specification section 133 Estimation section 134 Tagging section 135 Tag changing section 136 Image conversion section 137 Providing section

Claims (9)

a guide display unit that displays on a screen a shooting guide indicating a pose of a subject and a composition for shooting belongings when shooting a multi-viewpoint image;
a guide change unit that automatically changes the form of the imaging guide displayed on the screen according to the viewpoint at the time of imaging each time the viewpoint is moved;
a photography determination unit that automatically takes a photograph when it is determined that the pose and belongings of the subject match the photography guide;
An information processing device comprising: The information processing device according to claim 1, wherein the guide display unit displays, as the photographing guide, a guide regarding at least one of a pose, an expression, and a way of holding the photographed person among the subjects on the screen. . The information processing device according to claim 1 or 2, wherein the guide display unit displays a silhouette, an outline, or a semi-transparent display of the subject on the screen as the photographing guide. The information processing apparatus according to any one of claims 1 to 3, wherein the guide display section displays the photographing guide superimposed on the subject when photographing the subject. Claims 1 to 4, wherein the guide display unit specifies the pose of the photographed subject among the subjects in the first photographed image, and selects and displays a photographic guide according to the specified pose. The information processing device according to any one of the above. The information processing device according to any one of claims 1 to 5, wherein the guide changing unit changes the photographing guide in stages as the viewpoint at the time of photographing moves. a notification unit that notifies a photographer of a location where the subject is misaligned with the photography guide when the subject is misaligned with the photography guide;
The information processing device according to any one of claims 1 to 6 , further comprising the following. An information processing method executed by an information processing device, the method comprising:
a guide display step of displaying a shooting guide on the screen that indicates the pose of the subject and the composition of the belongings when shooting the multi-view image;
a guide changing step of automatically changing the form of the photographing guide displayed on the screen each time the viewpoint at the time of photographing is moved according to the viewpoint at that time ;
a photographing determination step of automatically photographing when it is determined that the subject's pose and belongings match the photographing guide;
An information processing method characterized by comprising: A guide display procedure for displaying a shooting guide on a screen that indicates the pose of a subject and the composition of their belongings when shooting a multi-view image;
a guide changing procedure for automatically changing the form of the photographing guide displayed on the screen according to the viewpoint at that time each time the viewpoint at the time of photographing is moved;
a photographing determination procedure for automatically photographing when it is determined that the subject's pose and belongings match the photographing guide;
An information processing program that allows a computer to execute.

Images