JP2004504676A - Method and apparatus for identifying and processing commands in a digital image in which a user marks commands, for example in a circle - Google Patents

Abstract

The user captures a digital image in which the user has marked one or more objects or commands on the paper with a pen. Marking is performed according to a predetermined set of marking rules, so that objects and commands can be easily separated later. Rules for marking objects and commands include circles, underlining, and highlighting. Commands embedded in a digital image can be associated with one or more objects in the digital image. This association is performed according to a predetermined set of association rules. Next, the image is sent to the computing device for processing. For transmission, a cable, IR or FR technology such as Bluetooth, or a mobile phone is used. The computing device is called a "command server". It identifies and separates objects and commands by analyzing a digital image using a predetermined set of marking rules. The command server also identifies, interprets, and associates commands with the object by analyzing the digital image according to a predetermined set of marking rules and association rules. Finally, the command server starts the service associated with the command on the object.

Description

[0001]
(Field of the Invention)
The present invention relates to a method for processing information in a digital image. More precisely, the present invention relates to processing digital images that contain embedded commands for services that a user wants to perform on the digital image.
[0002]
(background)
Digital cameras capable of capturing digital images are widely used around the world. Such images are transferred to a computer and stored for future retrieval or sent as an e-mail attachment. The recipient of the e-mail can open the attached file and view the image using a dedicated application program. Such e-mail may also be sent via a mobile telephone that includes or connects to the modem.
[0003]
The problem is that the recipient of the digital image must manually perform any necessary processing of the digital image. If the digital image represents a picture, the recipient must open the picture in a software application that can print or store the picture. If the digital image represents handwritten text, the recipient opens the image with a handwriting recognition software application that converts the image to digital characters, and then converts the characters to a file compatible with a software word processing application. You will want to convert it to a format.
[0004]
Another problem is that a digital image may include multiple objects that the user wants to perform various operations. Some objects contain printed or handwritten text that needs to be interpreted by software that converts to digital characters. Or, some objects represent digital pictures that need to be compressed or converted to another picture file format. Known software applications are not accurate enough to separate digital images into objects such as text, pictures and sketches, and are not suitable for automatically performing various services on various objects contained in the digital images. .
[0005]
Yet another problem is that the sender of the digital image must perform a number of steps that are fundamentally different in nature. For example, capture an image with a digital camera, send it to a mobile phone or computer, launch an email application program, insert the digital image into an email, enter or search for the recipient's email address. And sending an email.
[0006]
(Summary of the Invention)
In view of the foregoing, it is an object of the present invention to provide a completely new type of method for intelligently interpreting and processing digital images, overcoming the disadvantages of the conventional methods.
This object can be achieved by a method and apparatus for analyzing a digital image and separating it into objects and commands associated with the objects. Further, various processes related to the related command are performed on the object.
[0007]
In one form of the invention, a user uses a digital camera to capture digital images of commands and objects written by the user on paper. One or more commands or objects are marked with a pen. The marking is performed according to a predetermined set of marking rules, so that individual objects and commands can later be easily separated. Rules for marking objects and commands include, for example, encircling, underlining, highlighting, and the like. Commands embedded in a digital image can be associated with one or more objects in the digital image. This association is performed according to a predetermined set of association rules. The association rules allow various commands in the image to be associated with various objects. Next, the image is sent to the computing device for processing. For transmission, a cable, IR or RF technology such as Bluetooth, or a mobile phone is used. The computing device is called a "command server". It identifies and separates objects and commands by analyzing a digital image using a predetermined set of marking rules. The command server also identifies, interprets, and associates commands with objects by analyzing the digital image according to a predetermined set of marking rules and association rules. The command server is capable of interpreting digital images, for example, consisting of printed text and graphics of advertisements and brochures and handwritten text written by the user. Finally, the command server starts services on the command to the object.
If the digital image contains only one object, the marking of the object may be omitted because the command server does not need to separate it from another object.
[0008]
An advantage of the present invention is that it provides a way to process information contained in pictures quickly, simply, intuitively and automatically. Users can perform advanced electronic messaging and other services without the need for a full-fledged man-machine interface as with a typical computer. Examples of such services are shown below.
Recognize at least one handwritten object Convert at least one object to a selected file format Print and send characters Store at least one object for future retrieval At least one Enter or synchronize objects with a calendar, address book, or "to-do" list database. Make financial transactions. Dial phone numbers. Remember phone numbers. Services Before starting, send a request to the user to confirm the interpretation of the command or object. Retrieve one or more digital images from the digital image mass storage for further processing.
Users can create their own services, association rules, and marking rules by editing their own user profile.
[0009]
As will be realized, the invention disclosed herein achieves the objects set forth above and provides advantages thereof, but the features of the invention described herein are merely examples of the preferred embodiments. Therefore, the scope of the present invention is not limited to the details of the embodiments described herein. Equivalent, compatible, and modified forms of the present invention deduced from this description are intended to be included in the scope of the present invention as defined in the appended claims.
[0010]
(Detailed description of embodiments)
FIG. 1 shows a mobile telephone (101) with an embedded digital camera (103). The mobile telephone has an antenna (102) for communicating with a cellular network, a speaker (104), a display (105), a keypad (106), and a microphone (107). It is also possible to use a separate digital camera that transfers the image to the mobile phone via a system connector, IR port or short-range RF link. Yet another option is to capture the image with a digital camera or scanner and send it to a local computer with a command server, or send the image over the network to another computer with a command server. The command server has a function of processing the digital image and starting a service related to the command.
[0011]
FIG. 2 illustrates an example of a digital image representing information written on a piece of paper with a pen. The image includes one text object (204) and one picture object (207) circled with a pen (201, 205) according to a predetermined set of marking rules. The user has written a circle and a command (206, 203) according to a predetermined set of association rules. The command (203) is represented by "T, E (Max)". This instructs the command server to treat the circled object as text ("T" is text) and send it to Max by email ("E" is email). The circle command (206) is represented by "P". This informs the command server that this circle contains a picture. This information overrides the information in the outer circle that indicated that it contained text. The line (202) between the circle and its associated command specifies a relationship according to a predetermined set of association rules. The command indicates the type of service that the user wants to perform on the objects contained within the circle. In this example, the user wants the server to create an e-mail and send it to Max. “Max” is predefined by the user in the database of the command server and represents the nickname of Max's complete email address. The body of the e-mail includes one text portion that is the result of the command server converting the handwritten text (204) to digital characters, and one picture portion that contains a bit map representation of the object (207).
[0012]
FIG. 3 shows another example of a digital image representing a combination of preprinted information, a handwritten circle, and a command. The image includes a candle advertisement (304) circled (301) with a pen according to a predetermined set of marking rules. The user has written a circle (301) and a command (303) having associated parameters (C, $ 5). The line (302) between the circle (301) and its associated command (303) specifies a relationship according to a predetermined set of association rules. The command (303) describes what service the user wants the object (304) included in the circle to perform. In this case, the user wants the command server to parse and order the text ("T" is text) in the associated object (304) ("B" is purchase). The order is paid by credit card ("C" is a credit card). The total value of the order is $ 5. In this example, the command server interprets the text in the advertisement and looks for an address to send the order to. Next, an email is created that includes a copy of the advertisement and the user's address, credit card number, and amount the user accepts to withdraw from the credit card. The user's personal data is input in advance in the database of the command server.
[0013]
The user can easily identify and separate objects and commands in the digital image using predetermined marking rules. The following is an example of how to mark an object or command.
・ Enclose in a circle ・ Draw a square around ・ Underline ・ Highlight with a special “marker pen”.
[0014]
If a circle is used as a predetermined marking rule, the circle may not be a perfect circle or an ellipse. Any suitable graphic curve surrounding the information may be used. It may be up to the image processing to determine whether the surrounding curve needs to be closed to be interpreted as a circle.
[0015]
The user associates the command with the object using a predetermined association rule. The following is an example of how to associate commands with objects.
• Circle the command and the object together. • Draw a line between the command and the object. • Draw a line between the command and the circle or square that contains the object.
[0016]
If the digital image contains only one object, the association may be omitted.
Both the predetermined set of marking rules and the predetermined set of association rules may be specified in various ways. Each user may personalize the rules according to his preferences. Such personalized rules may be stored in each user's profile.
[0017]
Any kind of printed material, such as paper, flyers, photos, etc., can be written with a pen. The user may mark objects containing handwritten text, pre-printed text or pictures, and combinations thereof, and write commands associated with the objects using a predetermined set of association rules.
The general format of the command is defined as follows:
<Command 1 (parameter 1, parameter 2,...), Command 2 (parameter 1, parameter 2,...),. . . >
[0018]
Below are some examples of commands and their associated parameters.
1. P, E (mag, mar)
This is because the digital image contains a picture (P) and the e-mail (E) gives the recipients "mag" and "mar" (the nickname that the user has associated with the actual Internet addresses of his friends Magnus and Mria). Means to send.
2. T, E (mag, mar)
This means that the digital image contains text (T), which is interpreted by handwriting recognition software and converted to digital characters, and then sent to the "mag" and "mar" e-mail addresses by e-mail (E). Means to send.
3. D
This means that the related object is discarded (D). This is used when the user regrets writing something.
4. M
This means that the associated object contains an appointment (M is "Meeting") and that the command server enters it into the user's electronic calendar for later retrieval or synchronization with any user's computing device. Means
5. TD
This means that the associated object will include an "action item" (TD) and that the command server will place it in the user's electronic "action list" for later retrieval or synchronization with any user's computing device. Means to enter.
[0019]
6. A
This means that the associated object contains the address entry (A) and that the circle server fills in the user's electronic address book for later retrieval or synchronization with any user's computing device. .
7. P
This means that the associated object contains information about the financial transaction (P is "payment") and that the command server performs this for the user.
8. E (Pic3, 6, 12, mar)
This means that the picture (Pic) numbers 3, 6, and 12 are retrieved from a digital camera or mobile telephone containing digital image storage means and sent to the "mar" address as part of the e-mail. This command may relate to an object containing a handwritten or printed text message similar to a normal postcard message. Here, it is assumed that the device storing the digital image can receive a command from the command server to retrieve the picture.
9. C, S
This means that the mobile phone dials the number found in the associated object. This also means that the numbers and names found in the associated objects are stored in the mobile phone's phonebook. Here, it is assumed that the mobile phone can receive commands from the command server for dialing and storage, whether the command server is in the mobile phone or in an external computing device with which it communicates. I do.
[0020]
The set of available commands may be personalized as needed. This is also one of the basis of the subscription fees paid by the user for using the command server.
After requesting the user to authenticate itself, the command server receives any data or digital images from the user or performs any activities required by the user. The authentication can be performed in the same way as on a GSM type mobile phone, ie with a SIM card (subscriber identity module). To further enhance security, the server may require a pin code and may require that all transmitted data be encrypted. Another method of authentication may be performed using a digital signature or a user's handwritten signature included in the picture.
[0021]
FIG. 4 illustrates an example embodiment. This includes a mobile telephone (401) containing an embedded digital camera, a cellular base station (402), a cellular network (403), the Internet (405), a command server (404), and a standard computer (406). ). The digital image is sent from the mobile phone through the base station to the cellular network, which forwards it to the selected command server. The Internet address of the command server is predefined by the user and stored in the mobile phone. This eliminates the need for the user to enter the command server address each time the user uses it. The command server contains the user's personally stored information, including a personalized user profile. This information can be accessed and modified from a general computer (406) connected to the Internet. The user profile may also consist of a personal set of marking rules and association rules and a set of available services.
[0022]
FIG. 5 shows an exemplary embodiment of the command server (501). It comprises a communication link (509) to the Internet, a CPU (508), a database (504) holding user profiles, and a storage area (507) for user files. Upon receiving a file containing a digital image from a user, the command server begins analyzing the image. First, an object and a command are detected and separated by the image analysis SW (502). SW uses a predetermined set of marking rules. The command and the object are associated using a predetermined set of association rules. Next, the identified command is interpreted by the command interpreter SW (503), and finally, various services (505, 506) are started for the object. The service is selected according to the command.
[0023]
The database that holds the user's user profile (504) contains information about the user. For example,
・ Credit card number ・ Electronic business card Name title address E-mail address Fax number Home telephone number Office telephone number Mobile telephone number ・ Complete nickname of individual or group and e-mail address including: Address book containing data. The address book in the circle server can be updated and maintained by the user from any computer via the Internet.
-A calendar containing appointments, etc.-a list of "actions"-a personal set of marking rules-a personal set of association rules-a personal set of accessible commands and services
The user storage area is used to store a copy of all processed images, or keeps a record of all services initiated by the found command and any messages sent.
The command server can send various notifications and status reports to the user (which describe the activities initiated by processing the received digital image) upon the user's request. Also, the command server notifies the user of the result of interpreting the command in the digital image, and waits for confirmation from the user to start a desired service.
[0025]
The command server service is not limited to a single user. Separate user profiles and storage areas can be maintained for each user.
Users can update their profile and interact with the data store at any time from any computer that has Internet access. Access can be protected by a password.
[0026]
Although the present invention has been disclosed as being implemented using a variety of devices for recording digital images, transmitting digital images, and processing the images, almost all necessary hardware and software have been implemented in a single unit. It can be realized in a unit. Such a unit may be a mobile user terminal (such as a mobile telephone) having a powerful digital processor and sufficient storage means. Another option is to have a command server on the digital camera or on the combination of the digital camera and the mobile phone. Generally speaking, a digital camera, a mobile phone (if available) and a command server may be combined or any number of physical devices capable of communicating directly or indirectly with each other. May be distributed to various devices.
[0027]
FIG. 6 is an example of a digital image representing a sheet of paper (601) on which a handwritten message (603) is written. The user has also written the underlined command (602). The physical boundary of the paper (601) may be the surrounding of the single object (603). Therefore, the handwritten circle around the object is omitted.
[Brief description of the drawings]
FIG. 1 is a schematic diagram of a mobile phone with an embedded digital camera.
FIG. 2 is an example of a paper digital image including a circled object with associated commands.
FIG. 3 is another example of a paper digital image that includes a circled object with associated commands.
FIG. 4 is a schematic diagram of an exemplary embodiment of the entire system.
FIG. 5 is a schematic block diagram of a command server capable of analyzing and processing digital images according to any included commands.
FIG. 6 is an example of a digital image representing a picture of a piece of paper containing a handwritten message and associated commands.

Claims (19)

A method for processing information present in a digital image of a source picture marked with at least one processing command, comprising:
Identifying at least one processing command in the digital image;
Performing processing on the digital image according to the at least one processing command;
How to process information containing. The method of claim 1, further comprising identifying at least one graphical object in the image, wherein identifying the at least one object and identifying the at least one processing command is performed using a predetermined set of marking rules. How information is processed. The predetermined set of marking rules is at least one of the following marking rules:
3. A circle around the object or command. 3. A square around the object or command. 3. An underlined object or command. 3. The highlighting of the object or command with a special "marker pen". How information is processed. 3. The method of processing information according to claim 2, further comprising the step of identifying a relationship between the at least one processing command and the at least one graphical object, wherein the identification is performed using a predetermined set of association rules. The predetermined set of association rules is at least one of the following association rules:
The method according to claim 4, further comprising: encircling both the command and the object with a circle; drawing a line between the command and the object or their marking figure; How information is processed. The process comprises at least one of the following services:
• Recognize at least one handwritten object • Convert at least one object to the selected file format • Send e-mail • Send fax • Print text and send • At least one for future search Enter or synchronize at least one object with a calendar database, address book database, or "actions" list database. Make a financial transaction. Dial a phone number. Sends a request to the user to confirm the interpretation of a command or object before starting the service.Starts retrieving one or more digital images from the digital image mass storage device for further processing. The throat according to claim 1, including: Method of processing information as described in any. An apparatus for processing information present in a digital image of a source picture marked with at least one processing command, the apparatus comprising:
Means for identifying at least one processing command in the digital image;
Means for performing processing on the digital image according to the at least one processing command;
Processing device for information containing. Means for identifying at least one graphical object in the image, and wherein the means for identifying the at least one object or the means for identifying the at least one processing command uses means using a predetermined set of marking rules. Item 7. An information processing device according to Item 7. The predetermined set of marking rules is at least one of the following marking rules:
9. A circle around the object or command. Draw a square around the object or command. Underline the object or command. 9. Highlight the object or command with a special "marker pen". Information processing device. 9. The information processing apparatus according to claim 8, further comprising means for identifying a relationship between at least one processing command and at least one graphic object, wherein said identification is performed using a predetermined set of association rules. The predetermined set of association rules is at least one of the following association rules:
11. The method according to claim 10, further comprising: drawing a line between the command and the object or their marking figure; and drawing a line between the command and the circle or the square containing the object. Information processing device. The means for processing the object comprises at least one of the following services:
• Recognize at least one handwritten object • Convert at least one object to the selected file format • Send e-mail • Send fax • Print text and send • At least one for future search Enter or synchronize at least one object with a calendar database, address book database, or "actions" list database. Make a financial transaction. Dial a phone number. Sends a request to the user to confirm the interpretation of a command or object before starting the service.Starts retrieving one or more digital images from the digital image mass storage device for further processing. 12. The method according to claim 7, further comprising: Information processing apparatus according to any Re. A mobile communication terminal capable of processing information present in a digital image of a source picture in which at least one processing command is marked,
Means for obtaining a digital image of said picture;
Means for identifying at least one processing command in the digital image;
Means for performing processing on the digital image according to the at least one processing command;
A mobile communication terminal including: Means for identifying at least one graphical object in the image, and wherein the means for identifying the at least one object or the means for identifying the at least one processing command uses means using a predetermined set of marking rules. Item 14. The mobile communication terminal according to Item 13. The predetermined set of marking rules is at least one of the following marking rules:
15. A circle around the object or command. * A square around the object or command. * Underlining of the object or command. * Highlighting the object or command with a special "marker pen". Mobile communication terminal. 15. The mobile communication terminal according to claim 14, further comprising means for identifying a relationship between at least one processing command and at least one graphic object, wherein said identification is performed using a predetermined set of association rules. The predetermined set of association rules is at least one of the following association rules:
17. The method according to claim 16, comprising: encircling both the command and the object with a circle; drawing a line between the command and the object or their marking figure; and drawing a line between the command and a circle or a square containing the object. Mobile communication terminal. The means for processing the object comprises at least one of the following services:
• Recognize at least one handwritten object • Convert at least one object to the selected file format • Send e-mail • Send fax • Print text and send • At least one for future search Enter or synchronize at least one object with a calendar database, address book database, or "actions" list database. Make a financial transaction. Dial a phone number. Sends a request to the user to confirm the interpretation of a command or object before starting the service.Starts retrieving one or more digital images from the digital image mass storage device for further processing. 18. A method according to claim 13, further comprising: Any mobile communication terminal according to any. A computer program comprising software instructions for performing the method of any of claims 1-6.

Images