JP2010087989A - Mobile terminal device - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a mobile terminal device which easily acquires information related to an image captured by photographing from the image. <P>SOLUTION: The invention relates to a mobile terminal device including: a photographing section; a status detection section for detecting statuses of the present device and around the present device; an information extraction section for extracting information related to an object included in image data captured by photographing by the photographing section on the basis of the image data and a result of the detection detected by the status detection section; and a display section for displaying the image data and the information extracted by the information extraction section. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present case relates to a portable terminal device that displays an image obtained by photographing and information related to the image.

In recent years, mobile terminals having a camera function are increasing. An image obtained by photographing with a mobile terminal is often used as it is by displaying it on a display of the mobile terminal or transmitting it to another user's mobile terminal.
JP 2002-238080 A

  However, an image obtained by photographing with a mobile terminal has not been used effectively.

  This subject makes it a subject to provide the portable terminal device which acquires easily the information relevant to an image from the image obtained by imaging | photography.

  In order to solve the above problems, the following means are adopted.

That is, in the first aspect,
A shooting section;
A state detection unit for detecting the own device and a state around the own device;
An information extraction unit that extracts information related to the object included in the image data based on the image data obtained by photographing of the photographing unit and the detection result detected by the state detection unit;
A display unit for displaying the image data and the information extracted by the information extraction unit;
The portable terminal device provided with was used.

In the second aspect,
A shooting section;
A state detection unit for detecting the own device and a state around the own device;
A transmission unit that transmits image data obtained by imaging of the imaging unit and a detection result detected by the state detection unit to another device;
A receiving unit that receives information related to the image data and the detection result detected by the state detection unit from the other device;
A display unit for displaying the image data and the information received by the receiving unit;
The portable terminal device provided with was used.

  According to the embodiment of the disclosure, it is possible to provide a mobile terminal device that easily acquires information related to an image from an image obtained by photographing.

  Hereinafter, embodiments will be described with reference to the drawings. The configuration of the embodiment is an exemplification, and is not limited to the configuration of the disclosed embodiment.

Embodiment 1
(Configuration example)
<System configuration example>
FIG. 1 is a diagram illustrating a system configuration example of the present embodiment.

  As shown in FIG. 1, the system of this embodiment includes a network 100, a server 200, a radio base station 300, and a mobile terminal 400.

  The server 200 can receive information from the portable terminal 400 through the network. The server 200 can process information from the mobile terminal 400 and transmit the processing result to the mobile terminal 400.

  The radio base station 300 relays communication between the mobile terminal 400 and the network 100.

  The mobile terminal 400 communicates with the network 100 via the radio base station 300. The portable terminal 400 can communicate with the server 200 through a network.

<Configuration example of mobile terminal>
FIG. 2 is a diagram illustrating a configuration example of the mobile terminal.

  Specific examples of the mobile terminal include a mobile phone, a personal digital assistant (PDA), and a notebook personal computer.

The portable terminal 400 includes a CPU (Central Processing Unit) 402, a storage unit 404, a transmission / reception unit 406, an antenna 408, an image capturing unit 410, an image analysis unit 412, an information extraction unit 414, a sensor unit 416, and a display unit 418.

  Any number of these processing units may be used as one processing unit. Further, any one of these processing units may perform processing as a plurality of processing units. Each of these processing units can be realized as hardware or software. The same applies to the constituent elements described later.

  The CPU 402 is a central processing unit that executes various programs. The CPU 402 controls each processing unit in the mobile terminal 400 and performs necessary calculations.

The storage unit 404 stores various data such as a program executed by the portable terminal 400 and a table used in the program. The storage unit 404 may be a hard disk drive (HDD, Hard Disk Drive) or a memory such as a ROM (Read Only Memory). The storage unit 404 can store image data, various data, and the like that are used by the image analysis unit 412 during image analysis. The storage unit 404 can store various types of information corresponding to image data.

  The transmission / reception unit 406 transmits / receives a signal to / from the radio base station 300. The antenna 408 transmits a signal to another device and receives a signal from the other device.

  The imaging unit 410 captures images with a camera or the like. In addition, an image obtained through the camera can be displayed on the display unit 418 as it is.

The image analysis unit 412 analyzes the image obtained by the imaging unit 410 by imaging. For example, when the image captured by the image capturing unit 410 is a number written on paper, the image analysis unit 412 extracts numerical information from the image. Hereinafter, an image obtained by photographing by the photographing unit 410 is also referred to as a photographed image.

  The information extraction unit 414 extracts information required by the user from the storage unit 404 or the server 200 on the network 100 based on the information obtained by the image analysis by the image analysis unit 412.

  The sensor unit 416 includes various sensors. Examples of sensors include an inclination sensor, a GPS (Global Positioning System) unit, a temperature sensor, and the like.

  The display unit 418 performs screen display. Examples of the display unit 418 include a liquid crystal display and an organic EL display. The display unit 418 can be a display device without a frame. The display unit 418 can have a touch panel function.

(Operation example 1)
Operation example 1 will be described.

  FIG. 3 is a diagram illustrating an example of a processing flow of the mobile terminal according to the first operation example.

  The user of the mobile terminal 400 captures an object whose information is desired to be captured by the image capturing unit 410 of the mobile terminal 400 (S1002). An image obtained by shooting is stored in the storage unit 404. The sensor unit 416 detects the state when the image is taken with various sensors, and stores the detection result in the storage unit 404.

  The image analysis unit 412 extracts the image stored in the storage unit 404 and performs image analysis (S1004). The image analysis unit 412 compares the captured image with image data for image analysis stored in the storage unit 404 and identifies what the captured image is. When performing image analysis, the image analysis unit 416 can use information detected by the sensor unit 416 as information for specifying an image.

  The information extraction unit 414 extracts information related to the information from the storage unit 404 based on the information on the captured image analyzed by the image analysis unit 412 (S1006). The information extraction unit 412 may extract information related to the information from the server 200 or the like on the network 100 based on the information analyzed by the image analysis unit 412. The information extraction unit 414 can perform necessary calculations based on the information analyzed by the image analysis unit 412. Further, what kind of information the user wants to acquire may be designated in advance in the portable terminal 400 by mode selection or the like.

  Based on a plurality of pieces of information related to a plurality of captured images obtained by shooting, the information extraction unit 414 analyzes the image of the image analysis unit 412, and stores information related to the information in the storage unit 404 or the server 200 on the network 100. You may extract from etc. The information extraction unit 414 can perform necessary calculations based on a plurality of pieces of information related to a plurality of captured images subjected to image analysis by the image analysis unit 412.

  The display unit 418 displays the information extracted by the information extraction unit 414, the calculation result, or the like (S1008). The display unit 418 can be a display device without a frame. The display unit 418 may simultaneously display the information extracted by the information extraction unit 414 and the image obtained by shooting by the shooting unit 410 at the same time.

(Specific Example 1-1)
4 and 5 are diagrams illustrating a specific example 1-1 of the operation example 1. FIG.

  Here, an example in which calculation is performed will be described.

  As shown in FIG. 4, there is a piece of paper 10 on which a plurality of numbers (83, 36, 853, 710, 1, 210, etc.) are described. The user uses the portable terminal 400 to use the calculation related to the numbers written on the paper piece 10. The user can set the mobile terminal 400 in the calculation mode in advance. Here, the calculation mode is a mode in which numbers are extracted from the captured image and calculation is performed. At this time, symbols “+”, “−”, “×”, “÷”, and “=”, which are arithmetic operators, are displayed on the display unit 418 of the portable terminal 400. The display unit 418 has a touch panel function, and by touching these displayed symbols, calculation corresponding to the symbols is executed.

  The user selects and shoots “36” among the numbers written on the piece of paper 10 with the photographing unit 410 of the portable terminal 400. The mobile terminal 400 stores the captured image in the storage unit 404. The image analysis unit 412 performs image analysis on the captured image. Since the calculation mode is designated by the portable terminal 400, the imaging unit 412 performs image analysis assuming that the captured image is a number. The image analysis unit 412 takes out image data related to numbers from the storage unit 404 and compares it with the captured image. The image analysis unit 412 determines that the captured image is the number “36”, and displays the number “36” on the display unit 418.

  Next, the user touches the “+” symbol on the display unit 418. Thereby, the mobile terminal 400 recognizes that another number is added to “36” that has been subjected to the image analysis.

  The user uses the photographing unit 410 of the portable terminal 400 to select “853” from among the numbers written on the piece of paper 10 and photograph it. Similarly to the above case, the mobile terminal 400 performs image analysis on the captured image, determines that the number “853” has been captured, and displays the number “853” on the display unit 418.

  The user touches the “+” symbol on the display unit 418 again. Thereby, the portable terminal 400 recognizes that another number is added to “36” and “853” that have been subjected to the image analysis.

  The user uses the photographing unit 410 of the portable terminal 400 to select and photograph “710” from among the numbers written on the piece of paper 10. Similarly to the above case, the mobile terminal 400 performs image analysis on the captured image, determines that the number “710” has been captured, and displays the number “710” on the display unit 418.

  Next, the user touches the symbol “=” on the display unit 418. As a result, as shown in FIG. 5, the portable terminal 400 calculates “36 + 853 + 710” and displays “1599” as the calculation result on the display unit 418 (display unit 418 of the portable terminal 400 in FIG. 5).

  As described above, using the mobile terminal, it is possible to extract a number written on a piece of paper and perform a calculation related to the number.

(Operation example 2)
Operation example 2 will be described. The operation example 2 has common points with the operation example 1. Here, description of points in common with Operation Example 1 may be omitted.

FIG. 6 is a diagram illustrating an example of a processing sequence of the server and the mobile terminal in the second operation example. The server 200 is not limited to one, and a plurality of servers may share the processing of the server 200 described here.

  The user of the portable terminal 400 photographs the target for which information is desired to be captured by the photographing unit 410 of the portable terminal 400 (SQ1002). The captured image is stored in the storage unit 404. The sensor unit 416 detects the state when the image is taken with various sensors, and stores the detection result in the storage unit 404.

  The transmission / reception unit 406 of the portable terminal 400 transmits the captured image to the server 200 (SQ1004). The transmission / reception unit 406 can simultaneously transmit a plurality of captured images obtained by the imaging unit 410 to the server 200. The transmission / reception unit 406 can transmit the detection result detected by the various sensors by the sensor unit 416 to the server 200 together with the captured image. Further, what kind of information the user desires to acquire may be designated in advance on the portable terminal 400, and the transmission / reception unit 406 may transmit the type of information that the user desires to acquire to the server 200.

  The server 200 performs image analysis on the captured image received from the mobile terminal 400 and acquired by the mobile terminal 400 (SQ1006). The server 200 compares the captured image with image data for image analysis that can be used by the server 200 and identifies what the captured image is. When performing image analysis, the server 200 can use information detected by the sensor unit 416 of the mobile terminal 400 as information for specifying an image.

  The server 200 extracts information related to the information related to the captured image from the server 200 itself or another server based on the information related to the captured image analyzed and the information received from the portable terminal (SQ1008). Further, the server 200 can perform a necessary calculation based on the information obtained by image analysis.

  The server 200 may extract information related to the information from the server 200 itself or another server based on a plurality of pieces of information related to the plurality of photographed images. The server 200 can perform necessary calculations based on a plurality of pieces of information related to a plurality of photographed images that have undergone image analysis.

  The server 200 transmits information extracted by the server, a result calculated by the server, and the like to the mobile terminal 400 (SQ1010). The portable terminal 400 receives information extracted by the server, a result calculated by the server, and the like from the server 200.

  Display unit 418 of portable terminal 400 displays information received from the server (SQ1012). The display unit 418 may simultaneously display the information received from the server and the photographed image obtained by photographing by the photographing unit 410.

(Specific example 2-1)
7 and 8 are diagrams illustrating a specific example 2-1 of the operation example 2. FIG.

  Here, an example (1) of acquiring information related to a captured image will be described.

  It is assumed that the user with the portable terminal 400 is in a position where the building 10A, the building 10B, the mountain 10C, and the building 10D can be seen.

The user photographs the building 10B with the photographing unit 410 of the portable terminal 400 (the portable terminal 400 on the left in FIG. 7). The captured image is stored in the storage unit 404. The sensor unit 416 has G
Using the PS unit, the position information of the photographed position is detected. The position information includes latitude, longitude, altitude and the like. The sensor unit 416 can detect other states when the image is taken with various sensors.

  The transmission / reception unit 406 of the portable terminal 400 transmits the captured image to the server 200. The transmission / reception unit 406 transmits the position information of the captured position to the server together with the captured image. The transmission / reception unit 406 can transmit other detection results detected by the sensor unit 416 with various sensors to the server 200.

  The server 200 performs image analysis on the image captured by the mobile terminal 400 received from the mobile terminal 400. The server 200 compares the captured image with image data for image analysis that can be used by the server 200 and identifies what the captured image is. When performing image analysis, the server 200 can use the position information of the captured position received from the mobile terminal. As a result of the image analysis, the server 200 determines that the captured image is “AA building”.

  The server 200 extracts information related to “AA building” from the server 200 itself or another server. For example, information such as the floor number and height of “AA building” is extracted. The server 200 transmits information regarding the extracted “AA building” to the mobile terminal 400. The portable terminal 400 receives information extracted by the server regarding the image transmitted to the server from the server 200.

  As shown in FIG. 8, the display unit 418 of the mobile terminal 400 displays the information received from the server (building name, floor number, height, etc.) (display unit 418 of the mobile terminal 400 on the left in FIG. 8). The portable terminal 400 can simultaneously display information received from the server and a captured image.

  Similarly, the user takes an image of the mountain 10 </ b> C with the imaging unit 410 of the mobile terminal 400 (the mobile terminal 400 on the right in FIG. 7). The captured image is stored in the storage unit 404. The sensor unit 416 detects position information of a photographed position using a GPS unit. The position information includes latitude, longitude, altitude and the like. The sensor unit 416 can detect other states when the image is taken with various sensors.

  The transmission / reception unit 406 of the portable terminal 400 transmits the captured image to the server 200. The transmission / reception unit 406 transmits the position information of the captured position to the server together with the captured image. The transmission / reception unit 406 can transmit the detection result detected by the sensor unit 416 with various sensors to the server 200.

  The server 200 performs image analysis on the captured image received from the mobile terminal 400 and captured by the mobile terminal 400. The server 200 compares the captured image with image data for image analysis that can be used by the server 200 and identifies what the captured image is. When performing image analysis, the server 200 can use the position information of the captured position received from the mobile terminal. The server 200 determines that the photographed image is “Mt. Fuji” as a result of the image analysis.

  The server 200 extracts information related to “Mt. Fuji” from the server 200 itself or another server. For example, information such as the elevation of “Mt. Fuji” is extracted. The server 200 transmits the extracted information about “Mt. Fuji” to the mobile terminal 400. The portable terminal 400 receives information extracted by the server regarding the image transmitted to the server from the server 200.

As shown in FIG. 8, the display unit 418 of the mobile terminal 400 displays information received from the server (mountain name, altitude, etc.) (display unit 418 of the mobile terminal 400 on the right in FIG. 8). The portable terminal 400 can simultaneously display information received from the server and a captured image.

(Specific example 2-2)
9 and 10 are diagrams illustrating a specific example 2-2 of the operation example 2.

  Here, an example (2) of acquiring information related to a captured image will be described.

  It is assumed that the user with the portable terminal 400 is outdoors or indoors where the sky can be seen.

  The user points the mobile terminal 400 upward and takes a picture of the sky (the mobile terminal 400 in FIG. 9). The captured image is stored in the storage unit 404. The sensor unit 416 detects position information of a photographed position using a GPS unit. The position information includes latitude, longitude, altitude and the like. The sensor unit 416 detects the tilt of the mobile terminal at the time of shooting with the tilt sensor. The sensor unit 416 can detect other states when the image is taken with various sensors.

  The transmission / reception unit 406 of the portable terminal 400 transmits the captured image to the server 200. The transmission / reception unit 406 transmits to the server, along with the captured image, the positional information of the captured position and the tilt of the mobile terminal at the time of capturing. The transmission / reception unit 406 can transmit other detection results detected by the sensor unit 416 with various sensors to the server 200.

  The server 200 performs image analysis on the captured image received from the mobile terminal 400 and captured by the mobile terminal 400. The server 200 compares the captured image with image data for image analysis that can be used by the server 200 and identifies what the captured image is. When performing the image analysis, the server 200 can use the position information of the photographed position received from the mobile terminal and the tilt information of the mobile terminal. As a result of the image analysis, the server 200 determines that the captured image is empty and the user desires weather information. For example, the server 200 may determine that the mobile terminal 400 has captured the sky when the captured image and the image data for image analysis do not match in the image analysis. In such a case, if the inclination of the mobile terminal indicates upward, the server 200 can more reliably determine that the mobile terminal 400 has photographed the sky.

  The server 200 extracts weather information related to the position where the user has photographed the sky from the server 200 itself or another server. For example, information such as today's weather and the highest temperature at the position taken by the user is extracted. The server 200 transmits weather information related to the position taken by the user to the mobile terminal 400. The portable terminal 400 receives information extracted by the server from the server 200.

  As shown in FIG. 10, the display unit 418 of the portable terminal 400 displays information received from the server (today's weather, maximum temperature, etc.). The portable terminal 400 can simultaneously display information received from the server and a captured image.

(Specific Example 2-3)
Here, an example (3) of acquiring information related to a captured image will be described.

  It is assumed that the user who has the portable terminal 400 is in a car or the like. The user can set the portable terminal 400 to the route guidance mode in advance. Here, the road guidance mode is a mode for performing road guidance by extracting a currently traveling road or the like from the captured image. The user can designate a destination in advance on the mobile terminal.

A user photographs a road or the like at the current location with the photographing unit 410 of the mobile terminal 400. The captured image is stored in the storage unit 404. The sensor unit 416 detects position information of a photographed position using a GPS unit. The position information includes latitude, longitude, altitude and the like. The sensor unit 416 can detect other states when the image is taken with various sensors.

  The transmission / reception unit 406 of the portable terminal 400 transmits the captured image to the server 200. The transmission / reception unit 406 transmits the position information of the captured position to the server together with the captured image. The transmission / reception unit 406 can transmit other detection results detected by the sensor unit 416 with various sensors to the server 200. Further, the transmission / reception unit 406 can transmit information such as the fact that the portable terminal 400 is set to the route guidance mode and the destination to the server.

  The server 200 performs image analysis on an image received from the portable terminal 400 and obtained by photographing with the portable terminal 400. The server recognizes from the information received from the mobile terminal 400 that the mobile terminal 400 is set to the “direction guide mode”. The server 200 compares the captured image with image data for image analysis that can be used by the server 200, and identifies where the captured image is, which direction the image is captured, and the like. When performing image analysis, the server 200 can use the position information of the captured position received from the mobile terminal.

  When the server 200 identifies the position and direction of the captured image, the server 200 uses the data stored in the server 200 itself and the data stored in the other server 200 to determine the direction to the destination, such as the user's destination direction. Get information about directions. Server 200 transmits the acquired information to portable terminal 400. The portable terminal 400 receives the information regarding the route guidance acquired by the server 200.

  The display unit 418 of the portable terminal 400 displays information received from the server. The portable terminal 400 can simultaneously display information received from the server and a captured image. For example, the portable terminal can display a captured road image and an arrow indicating the direction to go to the destination in an overlapping manner.

(Specific Example 2-4)
Here, an example of acquiring information related to a plurality of captured images will be described.

  It is assumed that the user with the portable terminal 400 is in front of food such as tomato, lettuce, and paprika.

  The user photographs tomatoes, lettuce, and paprika with the photographing unit 410 of the mobile terminal 400. Each captured image is stored in the storage unit 404. The sensor unit 416 can detect the state when the image is taken with various sensors.

  The transmission / reception unit 406 of the portable terminal 400 transmits each captured image to the server 200. The transmission / reception unit 406 can transmit other detection results detected by the sensor unit 416 with various sensors to the server 200.

  The server 200 performs image analysis on an image received from the portable terminal 400 and obtained by photographing with the portable terminal 400. The server 200 compares the captured image with image data for image analysis that can be used by the server 200 and identifies what the captured image is. As a result of the image analysis, the server 200 determines that the captured images are “tomato”, “lettuce”, and “paprika”, respectively.

The server 200 extracts information related to each of “tomato”, “lettuce”, and “paprika” from the server 200 itself or another server. For example, "Tomato"
Information such as nutritional value and price of “lettuce” and “paprika” is extracted. In addition, the server 200 extracts information related to a plurality or all of the captured images from the server 200 itself or another server. For example, a vegetable salad recipe using “tomato”, “lettuce”, and “paprika” is extracted. The server 200 transmits all the extracted information to the mobile terminal 400. The portable terminal 400 receives information extracted by the server regarding the image transmitted to the server from the server 200.

  The display unit 418 of the portable terminal 400 displays information received from the server. The portable terminal 400 can simultaneously display the captured image and information received from the server. In addition, the mobile terminal 400 can simultaneously display a plurality of captured images and information received from the server.

(Operational effect of this embodiment)
According to the present embodiment, information related to an image obtained by photographing with a mobile terminal can be extracted from the mobile terminal itself or an external server and displayed on the display unit of the mobile terminal together with the captured image. .

  According to the present embodiment, information related to a plurality of images obtained by photographing with a portable terminal is extracted from the portable terminal itself or an external server, and displayed together with the plurality of photographed images on the display unit of the portable terminal. can do.

FIG. 1 is a diagram illustrating a system configuration example of the present embodiment. FIG. 2 is a diagram illustrating a configuration example of the mobile terminal. FIG. 3 is a diagram illustrating an example of a processing flow of the mobile terminal according to the first operation example. FIG. 4 is a diagram (1) illustrating a specific example 1-1 of the first operation example. FIG. 5 is a diagram (2) illustrating a specific example 1-1 of the operation example 1. FIG. 6 is a diagram illustrating an example of a processing sequence of the server and the mobile terminal in the second operation example. FIG. 7 is a diagram (1) illustrating a specific example 2-1 of the operation example 2. FIG. 8 is a diagram (2) illustrating a specific example 2-1 of the operation example 2. FIG. 9 is a diagram (1) illustrating a specific example 2-2 of the operation example 2. FIG. 10 is a diagram (2) illustrating a specific example 2-1 of the operation example 2.

Explanation of symbols

10 pieces of paper
20A building
20B Building
20C mountain
20D building
100 network
200 servers
300 radio base station
400 Mobile terminal
402 CPU
404 storage
406 transceiver
408 antenna
410 Shooting unit
412 Image analysis unit
414 Information extraction unit
416 Sensor unit
418 display

Claims (2)

A shooting section;
A state detection unit for detecting the own device and a state around the own device;
An information extraction unit that extracts information related to the object included in the image data based on the image data obtained by photographing of the photographing unit and the detection result detected by the state detection unit;
A display unit for displaying the image data and the information extracted by the information extraction unit;
A mobile terminal device comprising: A shooting section;
A state detection unit for detecting the own device and a state around the own device;
A transmission unit that transmits image data obtained by imaging of the imaging unit and a detection result detected by the state detection unit to another device;
A receiving unit that receives information related to the image data and the detection result detected by the state detection unit from the other device;
A display unit for displaying the image data and the information received by the receiving unit;
A mobile terminal device comprising:

Images