WO2020202347A1

WO2020202347A1 - Information provision system and information terminal

Info

Publication number: WO2020202347A1
Application number: PCT/JP2019/014254
Authority: WO
Inventors: 山岡　大祐; 田中　一彦; 祐瀧口; 瞳 ▲濱▼村
Original assignee: 本田技研工業株式会社
Priority date: 2019-03-29
Filing date: 2019-03-29
Publication date: 2020-10-08
Also published as: JPWO2020202347A1; JP7237149B2; CN113383363A

Abstract

This information provision system which uses an information terminal having a camera and a display to provide a user with information pertaining to the compatibility of the designated article with the applicable site to which said designated article is to be mounted, wherein the information terminal has a display control means which displays images captured by the camera on the display, an acquiring means which acquires image data for the designated article, and a generating means which generates an augmented reality image of the designated article on the basis of the image data acquired by the acquiring means. The display control means displays, on the display, an augmented reality image of the designated article generated by the generating means as the information, overlaid on the applicable site within the captured image.

Description

Information provision system and information terminal

The present invention relates to an information providing system and an information terminal that provide information to a user.

An application that provides services by connecting to a network via a mobile terminal such as a smartphone has been devised. For example, in Patent Document 1, a captured image is displayed on a display in a portable wireless communication terminal (smartphone), and guidance (name) of a component included in the captured image is superimposed and displayed on the display. It is disclosed that the operation manual of the component is displayed on the display when the guidance of the superimposed component is pressed.

Japanese Unexamined Patent Publication No. 2014-215845

For example, if the user is considering purchasing an in-vehicle part, can the in-vehicle part be mounted or attached to the vehicle, or if the user is considering purchasing a large item, the user can load the item in the vehicle. It is desirable to easily grasp whether or not the article that the user is considering purchasing is suitable for the vehicle (that is, the suitability of the article for the vehicle), such as whether or not the article can be purchased. is there. In addition, the user may want to easily grasp information on articles that fit the structure of the vehicle, for example.

Therefore, an object of the present invention is to provide the user with information desired by the user easily and intuitively.

The information providing system as one aspect of the present invention is an information providing system that provides a user with information on compatibility between a designated article and a target location where the designated article is to be mounted by using an information terminal having a camera and a display. Therefore, the information terminal has display control means for displaying the captured image obtained by the camera on the display, acquisition means for acquiring image data of the designated article, and image data acquired by the acquisition means. Based on this, the display control means has a generation means for generating an extended reality image of the designated article, and the display control means uses the extended reality image of the designated article generated by the generation means as the information in the captured image. It is characterized in that it is superimposed on the target location and displayed on the display.

According to the present invention, for example, the information desired by the user can be provided to the user through augmented reality, so that the user can easily and intuitively grasp the information.

The accompanying drawings are included in the specification, which form a part thereof, show an embodiment of the present invention, and are used together with the description to explain the principle of the present invention.
Block diagram showing the configuration of the information provision system Flowchart showing the process of accepting the designation of goods A flowchart showing a process of providing a user with information on the suitability of a designated article. Diagram showing a display example of the initial screen Diagram showing a display example of the facility list Diagram showing a display example of a list of article categories Diagram showing a display example of a candidate list of goods The figure which shows the display example of article information A diagram showing the inside of a car being photographed with an information terminal Figure showing display example of augmented reality image of designated article Figure showing display example of augmented reality image of designated article Figure showing display example of augmented reality image of designated article The figure which shows the example which applied the information provision system to a motorcycle Diagram showing an example of applying an information provision system to a cultivator Flowchart showing the process of providing information on conforming goods to the user

Hereinafter, embodiments will be described in detail with reference to the attached drawings. It should be noted that the following embodiments do not limit the invention according to the claims, and not all combinations of features described in the embodiments are essential to the invention. Two or more of the plurality of features described in the embodiments may be arbitrarily combined. In addition, the same or similar configuration will be given the same reference number, and duplicate description will be omitted.

<First Embodiment>
The first embodiment according to the present invention will be described. FIG. 1 is a block diagram showing a configuration of the information providing system 100 of the present embodiment. The information providing system 100 of the present embodiment includes, for example, an information terminal 10 and a server device 20 that are communicably connected to each other via a network NTW, and is a target of an article specified by a user and a vehicle on which the article is to be mounted. It is a system for providing a user with information on compatibility with a location. In the following, the article designated by the user may be referred to as a "designated article", and the target location where the designated article is to be mounted may be referred to as a "scheduled loading location". Further, in the present embodiment, a four-wheeled vehicle will be illustrated as a vehicle.

First, the configuration of the information terminal 10 will be described. The information terminal 10 may include, for example, a processing unit 11, a storage unit 12, a camera 13, a display 14, a position detection sensor 15, a posture detection sensor 16, and a communication unit 17. Each part of the information terminal 10 is connected to each other so as to be able to communicate with each other via the system bus 18. Examples of the information terminal 10 include a smartphone and a tablet terminal. In the present embodiment, an example in which a smartphone is used as the information terminal 10 will be described. Smartphones and tablet terminals are mobile terminals having various functions other than the call function, but the dimensions of the displays are different from each other. In general, tablet terminals have larger display dimensions than smartphones.

The processing unit 11 includes a processor typified by a CPU, a storage device such as a semiconductor memory, an interface with an external device, and the like. The storage unit 12 stores a program executed by the processor, data used by the processor for processing, and the like, and the processing unit 11 reads the program stored in the storage unit 12 into a storage device such as a memory and executes the program. be able to. In the case of the present embodiment, the storage unit 12 stores an application program (information providing program) for providing the user with information on the compatibility between the designated article and the planned mounting location of the vehicle, and the processing unit 11 stores the application program (information providing program). , The information providing program stored in the storage unit 12 can be read into a storage device such as a memory and executed.

The camera 13 has a lens and an image sensor, and captures a subject to acquire a captured image. The camera 13 may be provided, for example, on an outer surface opposite to the outer surface on which the display 14 is provided. In addition, the display 14 notifies the user of information by displaying an image. In the case of the present embodiment, the display 14 can display the captured image acquired by the camera 13 in real time. Here, the display 14 of the present embodiment includes, for example, a touch panel type LCD (Liquid Crystal Display), and has a function of receiving information input from a user in addition to a function of displaying an image. However, the present invention is not limited to this, and the display 14 may have only the function of displaying an image, and an input unit (for example, a keyboard, a mouse, etc.) may be provided independently of the display 14.

The position detection sensor 15 detects the position and orientation of the information terminal 10. As the position detection sensor 15, for example, a GPS sensor that receives a signal from a GPS satellite to acquire the current position of the information terminal 10 or detects the direction in which the camera 13 of the information terminal 10 is directed based on geomagnetism or the like. An orientation sensor or the like can be used. In the present embodiment, when the description "position of the information terminal 10" is described, it is assumed that the orientation of the information terminal 10 is included in addition to the current position of the information terminal 10. In addition, the posture detection sensor 16 detects the posture of the information terminal 10. As the posture detection sensor 16, for example, an acceleration sensor, a gyro sensor, or the like can be used.

The communication unit 17 is communicably connected to the server device 20 via the network NTW. Specifically, the communication unit 17 has a function as a transmission unit that transmits information to the server device 20 via the network NTW and a function as a reception unit that receives information from the server device 20 via the network NTW. Have.

As a specific configuration of the processing unit 11, for example, a first acquisition unit 11a, a second acquisition unit 11b, a specific unit 11c, a generation unit 11d, a reception unit 11e, and a display control unit 11f are provided. sell. The first acquisition unit 11a acquires the data of the captured image acquired by the camera 13. The second acquisition unit 11b acquires the article information and the like stored in the server device 20 in association with the article from the server device 20. The identification unit 11c analyzes the captured image by performing image processing such as a pattern matching method, and identifies a planned mounting location in the captured image displayed on the display 14. The generation unit 11d generates an Augmented Reality (AR) image of the designated article based on the image data of the appearance of the article. The reception unit 11e executes a process of accepting the designation of the article by the user. The display control unit 11d displays the captured image acquired by the first acquisition unit 11a on the display 14. Further, the display control unit 11e is based on the position and orientation of the information terminal 10 detected by the position detection sensor 15 and the attitude detection sensor 16 when the augmented reality image of the designated article is generated by the generation unit 11d. , The augmented reality image of the designated article is superimposed on the photographed image and displayed on the display 14.

Next, the configuration of the server device 20 will be described. The server device 20 may include a processing unit 21, a storage unit 22, and a communication unit 23. The processing unit 21 includes a processor typified by a CPU, a storage device such as a semiconductor memory, an interface with an external device, and the like. The storage unit 22 stores a program executed by the processor, data used by the processor for processing, and the like, and the processing unit 21 reads the program stored in the storage unit 22 into a storage device such as a memory and executes the program. can do. Further, the communication unit 23 is communicably connected to the information terminal 10 via the network NTW. Specifically, the communication unit 23 has a function as a receiving unit that receives information from the information terminal 10 via the network NTW and a function as a transmitting unit that transmits information to the information terminal 10 via the network NTW. Have.

In the case of the present embodiment, the server device 20 (storage unit 22) stores article information for each of a plurality of types of articles. The article may be, for example, an in-vehicle part attached to the vehicle, or a transported object (including a packaging container) loaded in the luggage compartment of the vehicle for transportation by the vehicle. In addition, the article information includes, for example, information indicating the model and dimensions of the article, image data of the appearance of the article, facility where the article is sold and its location information, information posted by a user of the article, and the like. Can include. The posted information is information posted by the user of each article, for example, the model of the vehicle on which the article can be loaded, the orientation of the article when the article is loaded in the luggage compartment of the vehicle, and the article. It may contain information such as how to crush the corrugated cardboard when it is loaded on the vehicle. The posted information may include information that is not described in the operation manual of the article, such as a method of using the article more comfortably and the ease of use of the article.

Next, the processing performed on the information terminal 10 when the information providing program is executed will be described. 2A to 2B are flowcharts showing the processing performed by the processing unit 11 of the information terminal 10. FIG. 2A is a flowchart showing a process of accepting the designation of the article by the user, and FIG. 2B is a flowchart showing the process of providing the user with information on the suitability of the designated article with respect to the vehicle.

First, the process of accepting the designation of an article by the user will be described using the flowchart shown in FIG. 2A. The processing of the flowchart shown in FIG. 2A can be performed by the reception unit 11e of the processing unit 11. 3 to 7 are diagrams showing images displayed on the display 14 of the information terminal 10, and can be used to explain the processing performed by the processing unit 11 of the information terminal 10.

In S11, the processing unit 11 displays the initial screen on the display 14. FIG. 3 shows a state in which the initial screen is displayed on the display 14 of the information terminal 10. On the initial screen, as shown in FIG. 3, for example, an input field 31a in which the category (classification, type) of the article is input by the user, and a search button for receiving the user's instruction to start the search for the article. 31b and can be provided. In the example shown in FIG. 3, "vehicle-mounted parts" is input by the user in the input field 31a as the category of the article.

In S12, the processing unit 11 determines whether or not the search button 31b has been touched on the display 14 by the user. If the search button 31b is touched by the user, the process proceeds to S13, and if the search button 31b is not touched, S12 is repeated. In S13, the processing unit 11 determines whether or not the category of the article has been input to the input field 31a by the user. If the search button 31b is touched while the article category is not entered in the input field 31a, the process proceeds to S14, and selection processes (S14 to S15) for allowing the user to select the article category are executed. On the other hand, when the search button 31b is touched while the article category is entered in the input field 31a, the selection process of S14 to S15 is omitted and the process proceeds to S16.

In S14, the processing unit 11 displays a list of facilities such as shops existing around the information terminal 10 on the display 14. Specifically, the storage unit 22 of the server device 20 stores map information indicating the locations of a plurality of facilities. The processing unit 11 transmits the current position of the information terminal 10 detected by the position detection sensor 15 to the server device 20 via the communication unit 17. The server device 20 that has received the current position of the information terminal 10 searches for facilities existing within a predetermined range from the current position of the information terminal 10 based on the map information stored in the storage unit 22, and uses the search results. The obtained list of facilities is transmitted to the information terminal 10 via the communication unit 23. Then, as shown in FIG. 4, the information terminal 10 displays an image showing a list of received facilities on the display 14. In the example shown in FIG. 4, the area where the name of each facility is displayed is the selection button 32 for the user to select the facility, and the user touches one of the selection buttons 32 on the display 14. By doing so, you can select the facility. If the facility is selected by the user, the process proceeds to S15. Here, the predetermined range can be set in advance and arbitrarily by the user. The predetermined range may be the range of the distance from the information terminal 10 or the range of the time until arrival at the facility.

In S15, the processing unit 11 displays a list of categories of articles provided (sold) at the facility selected in S14 on the display 14. Specifically, in the storage unit 22 of the server device 20, article information that can be provided by each of the plurality of facilities is stored for each category of articles associated with each facility. The processing unit 11 obtains the article information stored in the server device 20 (storage unit 22) linked to the facility selected in S14, thereby displaying a list of article categories as shown in FIG. It can be displayed on the display 14. In the example shown in FIG. 5, the area where the category name of the article is displayed is the selection button 33 for the user to select the category of the article, and the user presses one of the selection buttons 33 on the display 14. By touching with, you can select the category of the article.

In S16, the processing unit 11 displays an article candidate list on the display 14 for the article category specified by the user. The article category specified by the user is an article category input by the user in the input field 31a of the initial screen, or an article category selected by the user in S15. For example, as shown in FIG. 6, the processing unit 11 acquires the article information stored in the storage unit 22 of the server device 20 in association with the category of the article specified by the user from the server device 20. , The candidate list of articles can be displayed on the display 14. In the example shown in FIG. 6, "drive recorder" is specified by the user as the category of the article, and the area where the model of the article is displayed is the selection button 34 for the user to select the article. It has become. The user can specify (select) an article by touching one of the selection buttons 34 on the display 14. Here, when a facility is selected in S14, the processing unit 11 can display a candidate list of articles provided (sold) at the selected facility on the display 14.

In S17, the processing unit 11 determines whether or not the article has been designated by the user. If the article is specified by the user, the process proceeds to S18, and if the article is not specified by the user, S17 is repeated. In S18, the processing unit 11 (second acquisition unit 11b) acquires the article information regarding the article (designated article) designated in S17 from the server device 20, and displays the acquired article information on the display 14. For example, as described above, the article information includes information indicating the model and dimensions of the article, image data of the appearance of the article, information posted by the user of the article, and the like, and the information and data are displayed on the display 14. Is displayed in. FIG. 7 shows an example of a display screen of article information regarding a drive recorder as a designated article. As shown in FIG. 7, the article information display screen is provided with a designated article model display column 35a, an appearance display column 35b of the designated article, an article dimension display column 35c, and a posted information display column 35d. .. Further, the OK button 35e and the cancel button 35f are provided on the article information display screen. When the OK button 35e is touched by the user, the process proceeds to S21 in FIG. 2B, and when the cancel button 35f is touched by the user, the process ends.

Next, the process of providing the user with information on the suitability of the designated article with respect to the vehicle will be described with reference to the flowchart shown in FIG. 2B. 8 to 10B are diagrams showing images displayed on the display 14 of the information terminal 10, and can be used to explain the processing performed by the processing unit 11 of the information terminal 10.

In S21, the processing unit 11 determines the target location (for example, the position of the structure or the vehicle) of the vehicle on which the designated article is to be mounted, based on the article information of the designated article. In the following, the target location where the designated article will be loaded may be referred to as the “planned loading location”. For example, the article information may include information on whether or not the article is an in-vehicle component, and if the article is an in-vehicle component, information indicating a planned mounting location of the article. Therefore, the processing unit 11 can determine whether or not the designated article is an in-vehicle component and determine the planned mounting location of the designated article based on the article information of the designated article. Specifically, when the drive recorder, which is an in-vehicle component, is a designated article, the processing unit 11 can determine the planned mounting location of the drive recorder to be the windshield near the room mirror based on the article information. On the other hand, when the processing unit 11 determines that the designated article is not an in-vehicle part based on the article information of the designated article, the processing unit 11 can determine the planned loading location of the designated article in the luggage compartment (loading section) of the vehicle. ..

In S22, the processing unit 11 requests the user to take a picture of the planned mounting location determined in S21. For example, when the designated article is a drive recorder, the processing unit 11 displays a comment requesting a picture of the windshield near the rearview mirror, which is the planned mounting location, on the display 14. If it is determined that the designated article is not an in-vehicle part, a comment requesting a photograph of the luggage compartment of the vehicle, which is the planned mounting location, is displayed on the display 14.

In S23, the processing unit 11 (first acquisition unit 11a) causes the camera 13 to start shooting and acquires a shot image from the camera 13. In S24, the processing unit 11 (display control unit 11f) sequentially displays the captured images acquired from the camera 13 on the display 14. FIG. 8 shows a state in which the inside of the vehicle is photographed by the information terminal 10 (camera 13) so as to include the rearview mirror and the center console.

In S25, the processing unit 11 (specific unit 11c) identifies the planned mounting location determined in S21 in the captured image displayed on the display 14. For example, the processing unit 11 can determine which part of the vehicle is captured by the camera 13 and the captured image is an image obtained by performing known image processing. As an example of known image processing, a portion (feature point) having a feature amount such as a corner, a curvature, a change in brightness, or a change in color is detected in a captured image, and the feature of the detected feature point is detected. A method of recognizing a vehicle portion (photographed portion) photographed by the camera 13 from feature information indicating an amount, a positional relationship, or the like can be mentioned. By such a method, the processing unit 11 can specify the planned mounting location in the captured image.

In S26, the processing unit 11 determines whether or not the planned mounting location is specified in the captured image displayed on the display 14. If the planned mounting location is specified in the captured image, the process proceeds to S27, and if the planned mounting location is not specified in the captured image, the process returns to S25.

In S27, the processing unit 11 acquires the dimensional information of the planned mounting location specified in S25 to S26. For example, the storage unit 12 of the information terminal 10 stores feature information for each of a plurality of types of vehicles, and the processing unit 11 has a high degree of agreement with the feature information specified in S25 to S26 ( That is, it is possible to specify the model of the vehicle having the characteristic information (the degree of matching exceeds a predetermined value). Further, in the storage unit 22 of the server device 20, dimensional information of each part of the vehicle is stored in association with the model of the vehicle for each model of the vehicle. The processing unit 11 of the information terminal 10 transmits information on the model of the specified vehicle to the server device 20, and transmits the dimensional information of each component associated with the model of the vehicle and stored in the server device 20 to the server device 20. Receive from. As a result, the processing unit 11 of the information terminal 10 can acquire the dimensional information of the planned mounting location specified in S25 to S26.

In S28, the processing unit 11 (generation unit 11d) generates an augmented reality image 40 of the designated article based on the image data of the appearance of the designated article. Then, the processing unit 11 (display control unit 11f) superimposes (superimposes) the augmented reality image 40 of the generated designated article on the planned mounting location in the captured image obtained by the camera 13 and displays it on the display 14. To do. At this time, the processing unit 11 moves the augmented reality image 40 of the designated article according to the movement of the information terminal 10 based on the position and orientation information of the information terminal 10 detected by the position detection sensor 15 and the attitude detection sensor 16. As such, the image captured by the camera 13 and the augmented reality image 40 of the designated article are aligned. That is, the processing unit 11 displays the augmented reality image 40 of the designated article on the display 14 so as to match the position of the planned mounting location in the captured image displayed on the display 14 based on the information on the position and orientation of the information terminal 10. To display. Further, the processing unit 11 reflects the actual dimensional relationship between the designated article and the planned mounting location based on the dimensional information of the designated article and the dimensional information of the planned mounting location acquired in S27 so that the designated article is reflected. The augmented reality image 40 of the above is displayed on the display 14. The image data of the appearance of the designated article and the dimensional information of the designated article are the information included in the article information acquired from the server device 20 in S18.

For example, FIG. 9 shows an example in which the augmented reality image 40a of the drive recorder is superimposed on the planned mounting location (windshield near the rearview mirror) in the captured image and displayed on the display 14 when the designated article is a drive recorder. Is shown. In the example shown in FIG. 9, a photographed image of the inside of the vehicle including the rearview mirror 41 and the center console 42 is displayed on the display 14, and the drive recorder is used with respect to the windshield near the rearview mirror specified as the planned mounting location. The augmented reality image 40a is displayed. As a result, the user can easily and intuitively grasp the suitability of the drive recorder for the vehicle, such as the positional relationship and the dimensional relationship between the vehicle and the drive recorder, before purchasing the drive recorder.

Further, in FIGS. 10A to 10B, when the designated article is a transported object (for example, a packaging container such as a desk or a bed), the augmented reality image 40b of the transported object is mounted on the captured image (vehicle 43). An example is shown in which the display 14 is superimposed on the luggage compartment of the above. In the examples shown in FIGS. 10A to 10B, a photographed image of the rear part (baggage compartment) of the vehicle 43 with the back door open is displayed on the display 14, and the luggage compartment specified as the planned mounting location is displayed. The augmented reality image 40b of the transported object is displayed. FIG. 10A shows the case where the size of the transported object is smaller than the size of the luggage compartment of the vehicle 43, and FIG. 10B shows the case where the size of the transported object is larger than the size of the luggage compartment of the vehicle 43. As a result, the user can easily and intuitively grasp the suitability of the transported object for the vehicle, such as whether or not the transported object as the designated article can be loaded on the loading platform of the vehicle 43, before purchasing the transported object. be able to.

As described above, in the information providing system 100 of the present embodiment, the augmented reality image of the designated article designated by the user is included in the captured image obtained by the camera 13 as information indicating the suitability of the designated article with respect to the planned mounting location. It is superimposed on the planned mounting location in the above and displayed on the display 14. As a result, if the user designates an article by the information terminal 10 and takes a picture of the planned mounting location of the vehicle with the camera 13 of the information terminal 10, the suitability of the designated article with respect to the planned mounting location of the vehicle can be easily and intuitively obtained. Can be grasped.

<Second Embodiment>
A second embodiment according to the present invention will be described. In the first embodiment described above, an example of applying the information providing system 100 to a four-wheeled vehicle has been described, but the present invention is not limited thereto. For example, the above information providing system 100 shall be applied to saddle-mounted vehicles (motorcycles, tricycles), or other machines and devices such as ships, aircraft, cultivators, lawnmowers, generators, and snowplows. Can be done. In this embodiment, an example in which the above information providing system is applied to a motorcycle and a cultivator will be described.

First, an example of applying the information providing system 100 to a two-wheeled vehicle will be described. Here, by executing the flowcharts shown in FIGS. 2A to 2B described above, the phosphorescent cover mounted on the key insertion portion 51 of the motorcycle 50 is designated as a designated article, and the key insertion portion 51 is designated as a planned mounting location. Here is a determined example. FIG. 11 is a diagram showing a state in which the information terminal 10 (camera 13) is photographing the vicinity of the key insertion portion 51 of the motorcycle 50. In the example shown in FIG. 11, the augmented reality image 40c of the cover of the key insertion portion 51 is superimposed on the key insertion portion 51 in the captured image and displayed on the display 14. As a result, the user can easily determine the suitability of the cover of the key insertion portion 51 with respect to the motorcycle, such as the positional relationship and the dimensional relationship between the key insertion portion 51 of the motorcycle 50 owned by the user and the cover, before purchasing the cover. It can be grasped intuitively.

Next, an example of applying the information providing system 100 to the cultivator will be described. Here, by executing the flowcharts shown in FIGS. 2A to 2B described above, the moving wheels mounted on the left and right sides of the claw portion 61 of the cultivator 60 are designated as designated articles as optional equipment, and the claw portion 61 is mounted. An example determined as a planned location is shown. FIG. 12 is a diagram showing a state in which the cultivator 60 is photographed by the information terminal 10 (camera 13). In the example shown in FIG. 12, the augmented reality image 40d of the wheel is superimposed on the claw portion 61 in the captured image and displayed on the display 14. As a result, the user can easily and intuitively grasp the suitability of the wheel for the cultivator 60 owned by the user before purchasing the wheel. Here, in S27 of the flowchart shown in FIG. 2B, the model of the cultivator 60 (vehicle) is specified in order to acquire the dimensional information of the planned mounting location. In this case, if the processing unit 11 determines that the wheel does not fit (that is, cannot be attached) to the specified type of cultivator 60 based on the article information of the wheel, a comment indicating that the wheel does not fit. A mark such as a wheel or a cross mark may be displayed on the display 14.

<Third Embodiment>
A third embodiment according to the present invention will be described. This embodiment basically inherits the first to second embodiments, and the terms, definitions, and the like are as described in the first and second embodiments. In the present embodiment, as a process performed by the information terminal 10 when the information providing program is executed, a structure included in the captured image obtained by the camera 13 is specified, and an article conforming to the specified structure is specified. The process of providing information to the user will be described.

For example, when the information providing program is activated, the information terminal 10 (processing unit 11) displays a screen on the display 14 for allowing the user to select either the first mode or the second mode, and is selected by the user. Execute the mode. The first mode is a mode for acquiring article information about an article designated by the user, and when the first mode is selected, the processes described in the first to second embodiments are executed. On the other hand, the second mode is a mode for identifying a structure in a captured image and acquiring article information about an article conforming to the specified structure, and the process described below is executed.

FIG. 13 is a flowchart showing processing performed by the processing unit 11 of the information terminal 10 when the second mode is selected by the user. In S31, the processing unit 11 (first acquisition unit 11a) causes the camera 13 to start shooting and acquires a shot image from the camera 13. In S32, the processing unit 11 (display control unit 11f) sequentially displays the captured images acquired from the camera 13 on the display 14.

In S33, the processing unit 11 (specific unit 11b) identifies the structure included in the captured image displayed on the display 14. For example, the processing unit 11 first recognizes the structure contained in the captured image by performing known image processing. As an example of known image processing, as described in S25 of the flowchart of FIG. 2B, a feature point is detected in a captured image, and a structure is obtained from feature information indicating the feature amount and positional relationship of the detected feature point. There is a method of recognition. In the storage unit 22 of the server device 20, feature information about each of the plurality of types of structures is stored in association with the model of each structure, and the processing unit 11 of the information terminal 10 recognizes it. Investigate (determine) whether or not there is a structure having a high degree of matching of feature information with respect to the structure (that is, the degree of matching exceeds a predetermined value). Thereby, the processing unit 11 can specify the model of the structure included in the captured image. As an example, as shown in FIG. 8, when the user photographs the inside of the vehicle with the information terminal 10 (camera 13) so as to include in-vehicle parts such as the rearview mirror and the center console, the processing unit 11 sets the rearview mirror as a structure. Can be specified as.

In S34, the processing unit 11 determines whether or not the structure has been specified in the captured image displayed on the display 14. If the structure is specified in the captured image, the process proceeds to S35, and if the structure is not specified in the captured image, the process returns to S33.

In S35, the processing unit 11 acquires a list of goods that can be provided at facilities such as shops existing around the information terminal 10. For example, the processing unit 11 may acquire a list of articles recommended by the facility (a list of recommended articles) as an article list that can be provided by a facility such as a store existing around the information terminal 10.

Specifically, the storage unit 22 of the server device 20 stores map information indicating the locations of a plurality of facilities. The processing unit 11 transmits the current position of the information terminal 10 detected by the position detection sensor 15 to the server device 20 via the communication unit 17. The server device 20 that has received the current position of the information terminal 10 searches for facilities existing within a predetermined range from the current position of the information terminal 10 based on the map information stored in the storage unit 22, and uses the search results. A list of a plurality of articles (article list) that can be provided at the obtained facility is transmitted to the information terminal 10 via the communication unit 23. As a result, the information terminal 10 can acquire a list of articles that can be provided at the surrounding facilities. Here, the predetermined range can be set in advance and arbitrarily by the user. The predetermined range may be the range of the distance from the information terminal 10 or the range of the time until arrival at the facility.

In S36, the processing unit 11 determines whether or not there is an article conforming to the structure specified in S33 (hereinafter, may be referred to as a conforming article) in the article list acquired in S35. If there is a conforming article, the process proceeds to S37, and if there is no conforming article, the process returns to S33. Further, in S37, the processing unit 11 (second acquisition unit 11b) acquires the article information stored in the server device 20 (storage unit 22) associated with the conforming article from the server device 20. As described above, the article information may include information indicating the model and dimensions of the article, image data of the appearance of the article, facilities where the article is sold, and location information thereof.

In S38, the processing unit (generation unit 11d) generates an augmented reality image 40 of the conforming article based on the image data of the appearance of the conforming article. Then, the processing unit 11 (display control unit 11f) superimposes the augmented reality image of the conforming article on the structure specified in S33 in the captured image obtained by the camera 13 and displays it on the display 14. At this time, the processing unit 11 moves the augmented reality image of the conforming article according to the movement of the information terminal 10 based on the position and orientation information of the information terminal 10 detected by the position detection sensor 15 and the attitude detection sensor 16. As described above, the image captured by the camera 13 and the augmented reality image of the conforming article are aligned. That is, the processing unit 11 displays the augmented reality image of the conforming article on the display 14 so as to match the position of the structure in the captured image displayed on the display 14 based on the information on the position and orientation of the information terminal 10. To do. Further, the processing unit 11 determines the conforming article so that the actual dimensional relationship between the conforming article and the structure is reflected based on the dimensional information of the conforming article and the dimensional information of the structure specified in S33. The augmented reality image 40 may be displayed on the display 14.

Here, in S38, the processing unit 11 may display information on the facility where the conforming article is provided on the display 14. The facility information may include, for example, information such as the homepage and telephone number of the facility, and information on the route to the facility. The information of such a facility may be displayed on the display 14 when the user touches the augmented reality image of the conforming article on the display 14.

As an example, as shown in FIG. 8, the inside of the vehicle is photographed by the information terminal 10 (camera 13) so that the user includes in-vehicle parts such as the rearview mirror and the center console, and in S33, the front of the rearview mirror or the vicinity of the rearview mirror is taken. Suppose glass is identified as a structure. In this case, assuming that the processing unit 11 has acquired a list of articles that can be provided at a car accessory store (facility) existing within a predetermined range from the information terminal 10 in S35, in S37, the room mirror or the windshield near the room mirror Goods information for drive recorders compatible with glass can be obtained. Then, as shown in FIG. 9, the augmented reality image 40a of the drive recorder generated based on the acquired article information (image data) is superimposed on the structure in the captured image (for example, the windshield near the rearview mirror). Is displayed on the display 14.

In this way, in the second mode, the structure in the photographed image is specified, and the article information regarding the article (conforming article) conforming to the specified structure is acquired. Then, an augmented reality image of the conforming article is generated based on the article information (image data), and the augmented reality image of the conforming article is superimposed on the structure in the captured image and displayed on the display. As a result, the user can easily and intuitively acquire information on the article that matches the structure in the captured image obtained by the camera 13 of the information terminal 10 without designating the article on the information terminal 10. Can be done.

<Summary of Embodiment>
1. 1. The information providing system of the above embodiment is
An information providing system (for example, 10) that uses an information terminal (for example, 10) having a camera (for example, 13) and a display (for example, 14) to provide a user with information on compatibility between a designated article and a target location where the designated article will be mounted. For example, 100)
The information terminal is
A display control means (for example, 11f) for displaying a captured image obtained by the camera on the display, and
An acquisition means (for example, 11b) for acquiring image data of the designated article, and
A generation means (for example, 11d) that generates an augmented reality image of the designated article based on the image data acquired by the acquisition means, and
Have,
The display control means superimposes the augmented reality image of the designated article generated by the generation means on the target location in the captured image as the information and displays it on the display.
According to this configuration, the user specifies an article on the information terminal, and if the target location where the designated article (designated article) is to be mounted is photographed by the camera of the information terminal, the designated article is adapted to the target location. It is possible to easily and intuitively grasp the sex.

2. In the information providing system of the above embodiment
The display control means displays an augmented reality image of the designated article on the display so that the actual dimensional relationship between the designated article and the target location is reflected.
According to this configuration, the user can more easily grasp the dimensional relationship between the designated article and the target location, so that the suitability of the designated article with respect to the target location can be grasped more easily and intuitively. ..

3. 3. In the information providing system of the above embodiment
The information terminal further includes a specific means (for example, 11c) for identifying the target location in the captured image.
The display control means superimposes and displays an augmented reality image of the designated article on the target location in the captured image specified by the specific means.
According to this configuration, the target location is specified on the information terminal side without the user instructing the position of the target location in the captured image, so that the user can grasp the suitability of the designated article for the target location. It is possible to improve convenience.

4. In the information providing system of the above embodiment
The information terminal further has a receiving means (for example, 11e) for displaying a candidate list of articles on the display and accepting designation of articles by a user.
The designated article is an article that has been designated by the user by the receiving means.
According to this configuration, it is possible to improve the convenience of the user when designating the article.

5. In the information providing system of the above embodiment
The reception means displays on the display as the candidate list the articles that can be provided at the facility existing within a predetermined range from the current position of the information terminal.
According to this configuration, information on articles provided (sold) at a nearby facility can be presented to the user, for example, while moving, so that the user can obtain necessary articles based on the information. For example, it is possible to drop in at the facility and improve the convenience of the user.

6. In the information providing system of the above embodiment
The reception means displays a list of articles belonging to the category input by the user on the display as the candidate list.
According to this configuration, the user can search for the necessary articles by category, so that the convenience of the user can be improved.

7. The information providing system of the above embodiment is
An information providing system (eg 100) that provides information to a user using an information terminal (eg 10) having a camera (eg 13) and a display (eg 14).
The information terminal is
A display control means (for example, 11f) for displaying a captured image obtained by the camera on the display, and
Specific means (for example, 11c) for identifying a structure contained in the captured image displayed on the display, and
Acquisition means (for example, 11b) for acquiring image data of an article conforming to the structure specified by the specific means among a plurality of articles that can be provided at a facility existing within a predetermined range from the current position of the information terminal. When,
A generation means (for example, 11d) that generates an augmented reality image of the article based on the image data acquired by the acquisition means, and
Have,
The display control means superimposes an augmented reality image of the article generated by the generation means on the structure in the captured image and displays it on the display.
According to this configuration, the user can easily and intuitively acquire information on an article that matches the structure in the captured image obtained by the camera of the information terminal without specifying the article on the information terminal. Is possible.

The present invention is not limited to the above embodiments, and various modifications and modifications can be made without departing from the spirit and scope of the present invention. Therefore, in order to make the scope of the present invention public, the following claims are attached.

10: Information terminal, 11: Processing unit, 12: Storage unit, 13: Camera, 14: Display, 15: Position detection sensor, 16: Posture detection sensor, 17: Communication unit, 20: Server device, 21: Processing unit, 22: Storage unit, 23: Communication unit

Claims

An information providing system that provides a user with information on compatibility between a designated article and a target location where the designated article is to be mounted, using an information terminal having a camera and a display.
The information terminal is
A display control means for displaying the captured image obtained by the camera on the display, and
An acquisition means for acquiring image data of the designated article, and
A generation means for generating an augmented reality image of the designated article based on the image data acquired by the acquisition means, and
Have,
The display control means provides information by superimposing an augmented reality image of the designated article generated by the generation means on the target location in the captured image as the information and displaying it on the display. system.
The first aspect of the present invention is characterized in that the display control means displays an augmented reality image of the designated article on the display so that the actual dimensional relationship between the designated article and the target location is reflected. Information provision system.
The information terminal further includes a specific means for identifying the target location in the captured image.
The display control means according to claim 1 or 2, wherein the augmented reality image of the designated article is superimposed and displayed on the target location in the captured image specified by the specific means. The information provision system described.
The information terminal further has a receiving means for displaying an article candidate list on the display and accepting a user's designation of an article.
The information providing system according to any one of claims 1 to 3, wherein the designated article is an article that has been designated by the user by the receiving means.
The information providing system according to claim 4, wherein the receiving means displays on the display as a candidate list items that can be provided at a facility existing within a predetermined range from the current position of the information terminal.
The information providing system according to claim 4, wherein the reception means displays a list of articles belonging to the category input by the user as the candidate list on the display.
An information providing system that provides information to users using an information terminal having a camera and a display.
The information terminal is
A display control means for displaying the captured image obtained by the camera on the display, and
A specific means for identifying a structure contained in the captured image displayed on the display, and
An acquisition means for acquiring image data of an article conforming to the structure specified by the specific means among a plurality of articles that can be provided at a facility existing within a predetermined range from the current position of the information terminal.
A generation means for generating an augmented reality image of the article based on the image data acquired by the acquisition means, and
Have,
The display control means is an information providing system characterized in that an augmented reality image of the article generated by the generation means is superimposed on the structure in the captured image and displayed on the display.
An information terminal that has a camera and a display and provides the user with information on the compatibility between the designated article by the user and the target location where the designated article is to be mounted.
A display control means for displaying the captured image obtained by the camera on the display, and
An acquisition means for acquiring image data of the designated article, and
A generation means for generating an augmented reality image of the designated article based on the image data acquired by the acquisition means, and
Have,
The display control means is an information terminal characterized in that an augmented reality image of the designated article generated by the generation means is superimposed as the information on the target location in the captured image and displayed on the display. ..
An information terminal that has a camera and a display and provides information to the user.
A display control means for displaying the captured image obtained by the camera on the display, and
A specific means for identifying a structure contained in the captured image displayed on the display, and
An acquisition means for acquiring image data of an article conforming to the structure specified by the specific means among a plurality of articles that can be provided at a facility existing within a predetermined range from the current position of the information terminal.
A generation means for generating an augmented reality image of the article based on the image data acquired by the acquisition means, and
Have,
The display control means is an information terminal characterized in that an augmented reality image of the article generated by the generation means is superimposed on the structure in the captured image and displayed on the display.