JP2018129097A - Information processing apparatus and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To guide a cooperative function.SOLUTION: In the case where a user is not notified of a cooperative function, when a first image associated with a first apparatus necessary for executing the cooperative function is designated, the user is notified of a guide to a second apparatus that can execute the cooperative function together with the first apparatus.SELECTED DRAWING: Figure 15

Description

  The present invention relates to an information processing apparatus and a program.

  Patent Documents 1 and 2 disclose techniques for linking a plurality of devices.

JP2015-177504A Japanese Patent Laying-Open No. 2015-223006

  An object of the present invention is to guide the cooperation function.

  According to the first aspect of the present invention, in the case where the cooperation function is not notified, when the first image associated with the first device necessary for executing the cooperation function is designated, the first device and the first device The information processing apparatus includes a control unit that controls notification of guidance to a second device that can execute the cooperation function.

  According to a second aspect of the present invention, in the information processing apparatus according to the first aspect, the first device and the second device are devices identified by being photographed by photographing means. is there.

  According to a third aspect of the present invention, in the case where the cooperation function is not notified, when a first image associated with the first function necessary for executing the cooperation function is designated, the first function and the first function are used together with the first function. The information processing apparatus includes a control unit that controls notification of guidance to the second function capable of executing the cooperation function.

  The invention according to claim 4 can execute the cooperation function together with the first device from the first image when the first image associated with the first device necessary for executing the cooperation function is designated. It is an information processing apparatus which has a control means which controls the display of the guidance instruction | indication to the 2nd image tied with 2nd apparatus.

  The invention according to claim 5 is characterized in that the control means controls display of an image indicating the second image from the first image as a starting point as control of display of the guidance instruction. It is an information processing apparatus as described in.

  The invention according to claim 6 is characterized in that the control means controls display of an image connecting the first image and the second image as control of display of a guidance instruction. The information processing apparatus described.

  According to a seventh aspect of the present invention, the control device further includes a third device capable of executing the linkage function together with the first device and the second device when the first image and the second image are designated. The information processing apparatus according to claim 4, wherein display of a guidance instruction to the user is controlled.

  According to an eighth aspect of the present invention, the control means controls the display of the guidance instruction by displaying an image indicating the second image starting from the first image and the third image starting from the second image. The information processing apparatus according to claim 7, wherein display of the third image pointing to the device is controlled.

  According to a ninth aspect of the present invention, the control means performs display of an image connecting the first image and the second image, control of the display of the guidance instruction, and the second image and the third image. The information processing apparatus according to claim 7, wherein display of images to be connected is controlled.

  The invention according to claim 10 is characterized in that the control means controls the display of the guidance instruction by changing the third device in accordance with the designation order of the first image and the second image. An information processing apparatus according to any one of claims 7 to 9.

  According to the eleventh aspect of the present invention, when the first image associated with the first device necessary for executing the cooperation function is designated, the first device is designated together with the first device at the first time point when the designation is performed. Control notification of information to a device capable of executing a cooperation function, and when the first image is specified at a second time point different from the first time point, the change in the surrounding environment of the first device, The information processing apparatus includes a control unit that changes a notification of guidance to a device that can execute the cooperation function.

  According to a twelfth aspect of the present invention, when the surrounding environment changes to an environment that requires the device for the first device at the second time point, the control means notifies the device of the guidance. The information processing apparatus according to claim 11, wherein the information processing apparatus is controlled.

  According to a thirteenth aspect of the present invention, in the information processing apparatus according to the eleventh or twelfth aspect, the surrounding environment includes a state of the device.

  In the invention described in claim 14, when the first image associated with the first device necessary for executing the cooperation function is designated, the one or more second devices that can execute the cooperation function together with the first device. Control means for controlling notification of guidance to the device, wherein the control means controls notification of guidance to each second device according to a priority according to a combination of the first device and each second device. This is an information processing apparatus.

  The invention according to claim 15 is the information according to claim 14, wherein the priority is a priority of the cooperation function that can be executed using the first device and the second device. It is a processing device.

  In the invention described in claim 16, when the first image associated with the first function necessary for executing the cooperation function is designated, the cooperation function can be executed together with the first function. Control means for controlling notification of guidance to functions, wherein the control means controls notification of guidance to each second function according to a priority according to a combination of the first function and each second function. This is an information processing apparatus.

  The invention according to claim 17 is the information according to claim 16, wherein the priority is a priority of the cooperation function that can be executed by using the first function and the second function. It is a processing device.

  The invention according to claim 18 is a first image associated with the first device necessary for executing the cooperation function, and a second image associated with the second device capable of executing the cooperation function together with the first device, Is specified, control means for notifying a guide to a third device capable of executing the linkage function together with the first device and the second device, wherein the control means includes the first image and the second device. The information processing apparatus controls the notification of the guidance by changing the third device according to an image designation order.

  In the invention according to claim 19, when the first image associated with the first device necessary for executing the cooperation function is designated when the cooperation function is not notified to the computer, the first A program that functions as a control unit that controls notification of guidance to a second device that can execute the linkage function together with a device.

  In the invention according to claim 20, when the first function image associated with the first function necessary for executing the cooperation function is designated when the cooperation function is not notified, the first A program that functions as a control unit that controls notification of guidance to a second function that can execute the linkage function together with the function.

  In the invention according to claim 21, when the first image associated with the first device necessary for executing the cooperation function of the computer is designated, the cooperation function is combined with the first device from the first image. Is a program that functions as a control unit that controls display of a guidance instruction on the second image associated with the second device that can execute the above.

  According to a twenty-second aspect of the present invention, when the first image associated with the first device necessary for executing the cooperation function is designated by the computer, the first image is designated at the first time point when the designation is performed. When notification of guidance to a device capable of executing the linkage function is controlled together with the device, and the first image is specified at a second time point different from the first time point, according to changes in the surrounding environment of the first device And a program that functions as control means for changing notification of guidance to a device capable of executing the linkage function.

  In the invention described in claim 23, when the first image associated with the first device necessary for executing the cooperation function of the computer is designated, one or more that can execute the cooperation function together with the first device. Functioning as control means for controlling notification of guidance to the second device, wherein the control means guides each second device according to the priority according to the combination of the first device and each second device. A program that controls notifications.

  In the invention described in claim 24, when the first image associated with the first function necessary for executing the cooperation function of the computer is designated, one or a plurality of the cooperation functions can be executed together with the first function. Functioning as control means for controlling the notification of guidance to the second function of the second function, the control means for guiding the second function according to the priority according to the combination of the first function and each second function. A program that controls notifications.

  According to a twenty-fifth aspect of the present invention, a computer is associated with a first image associated with a first device necessary for executing a cooperation function, and a second device associated with the second device capable of executing the cooperation function together with the first device. When two images are designated, the first and second devices together with the first device function as control means for notifying a third device capable of executing the linkage function, and the control means includes the first image and the second device. A program for controlling notification of the guidance by changing the third device in accordance with the order of designation of the second image.

  According to invention of Claim 1-25, a cooperation function can be guided.

It is a block diagram which shows the apparatus system which concerns on embodiment of this invention. 1 is a block diagram illustrating an image forming apparatus according to an exemplary embodiment. It is a block diagram which shows the server which concerns on this embodiment. It is a block diagram which shows the terminal device which concerns on this embodiment. 1 is a schematic diagram illustrating an appearance of an image forming apparatus. It is a figure which shows an example of an apparatus function management table. It is a figure which shows an example of a cooperation function management table. It is a figure which shows the apparatus utilized independently. It is a figure which shows an example of a function display screen. It is a figure which shows an example of a function display screen. It is a figure which shows the apparatus of cooperation object. It is a figure which shows an example of a function display screen. It is a sequence diagram which shows a connection process. It is a figure which shows an example of an apparatus display screen. It is a figure which shows an example of the apparatus display screen which concerns on Example 1. FIG. FIG. 10 is a diagram illustrating an example of a device display screen according to the second embodiment. It is a figure which shows an example of the apparatus display screen which concerns on Example 3. FIG. It is a figure which shows an example of the apparatus display screen which concerns on Example 3. FIG. It is a figure which shows an example of the apparatus display screen which concerns on Example 3. FIG. It is a figure which shows an example of the apparatus display screen which concerns on Example 3. FIG. It is a figure which shows an example of the apparatus display screen which concerns on Example 4. FIG. It is a figure which shows an example of the apparatus display screen which concerns on Example 4. FIG. It is a figure which shows an example of the apparatus display screen which concerns on Example 5. FIG. It is a figure which shows an example of the apparatus display screen which concerns on Example 5. FIG. FIG. 10 is a diagram illustrating an example of a screen according to the fifth embodiment. It is a figure which shows an example of the apparatus selection screen which concerns on Example 6. FIG. FIG. 10 is a diagram illustrating an example of a device selection screen according to a seventh embodiment. FIG. 10 is a diagram illustrating an example of a device selection screen according to a seventh embodiment. FIG. 10 is a diagram illustrating an example of a function selection screen according to a seventh embodiment. FIG. 20 is a diagram illustrating an example of a screen according to the seventh embodiment. FIG. 10 is a diagram illustrating an example of a message according to the seventh embodiment. FIG. 10 is a diagram illustrating an example of a message according to the seventh embodiment. FIG. 20 is a diagram illustrating an example of a device selection screen according to an eighth embodiment. FIG. 20 is a diagram illustrating an example of a device selection screen according to an eighth embodiment. FIG. 10 is a diagram illustrating an example of a screen according to an eighth embodiment. It is a figure which shows an example of a cooperation function management table. It is a figure which shows an example of an apparatus display screen and a function display screen. It is a figure which shows an example of an apparatus display screen and a function display screen. It is a figure which shows an example of an apparatus function management table. It is a figure which shows an example of an apparatus display screen and a function display screen. It is a figure which shows an example of an apparatus function management table. It is a figure which shows an example of a cooperation function management table. It is a figure which shows an example of the screen displayed on a terminal device. It is a figure which shows an example of the screen displayed on a terminal device. It is a figure which shows an example of the screen displayed on a terminal device.

  With reference to FIG. 1, a device system as an information processing system according to an embodiment of the present invention will be described. FIG. 1 shows an example of a device system according to the present embodiment.

  The device system according to the present embodiment includes a plurality of devices (for example, devices 10 and 12), a server 14 as an example of an external device, and a terminal device 16 as an example of an information processing device. The devices 10 and 12, the server 14, and the terminal device 16 have a function of communicating with each other via a communication path N such as a network. Of course, the devices 10, 12, the server 14, and the terminal device 16 may communicate with other devices via different communication paths. In the example shown in FIG. 1, two devices (devices 10 and 12) are included in the image forming system, but three or more devices may be included in the device system. A plurality of servers 14 and a plurality of terminal devices 16 may be included in the device system.

  The devices 10 and 12 are devices having specific functions. For example, an image forming device having an image forming function, a personal computer (PC), a projector, a display device such as a liquid crystal display or a projector, a telephone, a clock, and a surveillance camera. , Etc. The devices 10 and 12 have a function of transmitting and receiving data to and from other devices. In the present embodiment, the device 10 is an image forming apparatus as an example. The image forming apparatus (device 10) is an apparatus having at least one of a scan function, a print function, a copy function, and a facsimile function, for example.

  The server 14 is a device that manages the functions of each device. For example, the server 14 manages a function of each device, a cooperation function that uses a plurality of functions, and the like. The server 14 has a function of transmitting and receiving data to and from other devices.

  For each user, the server 14 may manage functions that can be used by the user. The functions that can be used by the user are, for example, functions that are provided to the user free of charge, functions that are provided to the user for a fee and purchased by the user, and the like. The server 14 may manage usable function information (for example, function purchase history information) indicating functions available to the user for each user. Of course, since there are functions that can be used free of charge, additional update functions, and functions that are specially managed by the administrator, the server 14 does not need to manage the functions depending on whether or not there is a purchase. The function purchase process is performed by the server 14, for example. Of course, the purchase process may be performed by another apparatus.

  The terminal device 16 is a device such as a personal computer (PC), a tablet PC, a smartphone, or a mobile phone, and has a function of transmitting / receiving data to / from other devices. For example, the terminal device 16 functions as a user interface unit (UI unit) when the device is used.

  In the device system according to the present embodiment, when the first image associated with the first device is designated, the notification of the guidance of the second device that can execute the cooperation function together with the first device is controlled. As another example, when the first image associated with the first function is designated, the notification of the guidance of the second function that can execute the linkage function together with the first function may be controlled.

  Hereinafter, each device included in the device system according to the present embodiment will be described in detail.

  With reference to FIG. 2, the configuration of the device 10 as the image forming apparatus will be described in detail. Hereinafter, the device 10 may be referred to as an image forming apparatus 10. FIG. 2 shows the configuration of the image forming apparatus 10.

  The communication unit 18 is a communication interface and has a function of transmitting data to other devices and a function of receiving data from other devices. The communication unit 18 may be a communication interface having a wireless communication function or a communication interface having a wired communication function.

  The image forming unit 20 has an image forming function. Specifically, the image forming unit 20 has at least one of a scan function, a print function, a copy function, and a facsimile function. By executing the scan function, the document is read and scan data (image data) is generated. By executing the print function, an image is printed on a recording medium such as paper. By executing the copy function, the document is read and printed on the recording medium. By executing the facsimile function, the image data is transmitted or received by facsimile. In addition, a cooperation function that uses a plurality of functions may be executed. For example, a scan transfer function that combines a scan function and a transmission function (transfer function) may be executed. By executing the scan transfer function, the original is read to generate scan data (image data), and the scan data is transmitted to a transmission destination (for example, an external device such as the terminal device 16). Of course, this cooperation function is only an example, and another cooperation function may be executed.

  The storage unit 22 is a storage device such as a hard disk or a memory (for example, SSD). The storage unit 22 indicates, for example, information indicating an image formation command (for example, job information), image data to be printed, scan data generated by executing a scan function, and addresses of other devices. Device address information, server address information indicating the address of the server 14, various control data, various programs, and the like are stored. Of course, they may be stored in separate storage devices or may be stored in one storage device.

  The UI unit 24 is a user interface unit and includes a display unit and an operation unit. The display unit is a display device such as a liquid crystal display. The operation unit is an input device such as a touch panel or a keyboard. Of course, it may be a user interface that serves both as a display unit and an operation unit (for example, including a touch-type display or a device that electronically displays a keyboard or the like on the display). The image forming apparatus 10 may not include the UI unit 24, or may include a hardware user interface unit (hardware UI unit) as hardware without including a display unit. The hardware UI unit is, for example, a hardware key (for example, a numeric keypad) specialized for numerical input, a hardware key (for example, a direction indication key) specialized for direction indication, and the like.

  The control unit 26 controls the operation of each unit of the image forming apparatus 10.

  Hereinafter, the configuration of the server 14 will be described in detail with reference to FIG. FIG. 3 shows the configuration of the server 14.

  The communication unit 28 is a communication interface and has a function of transmitting data to another device and a function of receiving data from another device. The communication unit 28 may be a communication interface having a wireless communication function or a communication interface having a wired communication function.

  The storage unit 30 is a storage device such as a hard disk or a memory (for example, SSD). The storage unit 30 includes, for example, device function management information 32, linkage function management information 34, various data, various programs, device address information indicating the address of each device, server address information indicating the address of the server 14, and the like. It is remembered. Of course, they may be stored in separate storage devices or may be stored in one storage device. The device function management information 32 and the link function management information 34 stored in the storage unit 30 are provided to the terminal device 16 regularly or at a designated timing, whereby the information stored in the terminal device 16 is stored. It may be updated. Hereinafter, the device function management information 32 and the linkage function management information 34 will be described.

  The device function management information 32 is information for managing the functions of each device, for example, device identification information (device identification information) for identifying the device, function information indicating the function of the device, It is the information which shows matching of. The device identification information includes, for example, a device ID, a device name, information indicating the device type, a device model number, information indicating a position where the device is installed (device position information), an appearance image indicating the appearance of the device, Etc. The function information is, for example, a function ID or a function name. For example, when the image forming apparatus 10 has a scan function, a print function, a copy function, and a scan transfer function, the device identification information of the image forming apparatus 10 includes function information indicating the scan function, function information indicating the print function, copy The function information indicating the function and the function information indicating the scan transfer function are associated with each other. By referring to the device function management information 32, the function of each device is specified (identified).

  The devices managed by the device function management information 32 are devices (for example, devices 10 and 12) included in the device system, for example. Of course, devices not included in the device system may be managed by the device function management information 32. For example, the server 14 may acquire information related to a new device not included in the device system (information including device identification information and function information) and newly register it in the device function management information 32. Information about the device may be acquired by using the Internet, for example, or may be input by an administrator or the like. In addition, the server 14 may update the device function management information 32 at an arbitrary timing, periodically, or at a timing designated by an administrator or the like. As a result, function information indicating functions that the device did not have before the update but that the device had after the update may be registered in the device function management information 32. Similarly, when function information indicating a function that the device has before the update but does not have the device after the update is deleted from the device function management information 32 or registered as unusable information There is. The update information may be acquired by using the Internet or the like, or may be input by an administrator or the like.

  The linkage function management information 34 is information for managing a linkage function executed by linking a plurality of functions. By coordinating a plurality of functions, one or a plurality of cooperation functions are executed. The cooperation function may be executed by, for example, linking a plurality of functions of one device (for example, the devices 10 and 12), or linking a plurality of functions of a plurality of devices (for example, the devices 10 and 12). May be executed. In addition, a terminal device that issues an operation instruction (in the present embodiment, the terminal device 16) may be included in the device to be identified, and the functions of the terminal device may be used as part of the linkage function.

  The linkage function may be a function that is executed without using a device as hardware. For example, the cooperation function may be a function executed by linking a plurality of software. Of course, the linkage function may be a function executed by linking a function of a device as hardware and a function realized by software.

  The cooperation function management information 34 is information indicating a correspondence between a combination of function information indicating each function used for the cooperation function and cooperation function information indicating the cooperation function, for example. The cooperation function information is, for example, a cooperation function ID or a cooperation function name. When a single function is updated, the linkage function management information 34 is also updated along with the update. As a result, a collaborative function with multiple functions that could not be linked to each other before the update becomes available after the update, and conversely, an available link function before the update cannot be used after the update. It may be possible. The link function information indicating the link function that can be used after the update is registered in the link function management information 34, and the link function information indicating the link function that has become unavailable after the update is deleted from the link function management information 34. Or, it is registered as unusable information.

  When a plurality of devices are linked, the linkage function management information 34 is information for managing a linkage function that uses a plurality of functions of the plurality of devices, and for identifying each device used for the linkage function. This is information indicating a correspondence between a combination of device identification information and linkage function information. As described above, when the device function management information 32 is updated, the linkage function management information 34 is also updated along with the update. As a result, the cooperation function by multiple devices that could not cooperate with each other before the update becomes available after the update, and conversely, the available cooperation function before the update becomes unavailable after the update. It may be possible.

  The cooperation function may be a function executed by linking a plurality of different functions to each other, or may be a function executed by linking the same function. The cooperation function may be a function that could not be used before the cooperation. The functions that could not be used before cooperation may be functions that can be used by using the same functions among the functions of the devices to be linked, or functions that can be used by combining different functions. May be. For example, by coordinating a device having a print function (printer) and a device having a scan function (scanner), a copy function as a cooperation function is realized. That is, the copy function is realized by linking the print function and the scan function. In this case, a copy function as a linkage function is associated with a combination of a print function and a scan function. In the cooperation function management information 34, for example, cooperation function information indicating a copy function as a cooperation function, device identification information for identifying a device having a print function, and device identification information for identifying a device having a scan function. Are associated with each other.

  The storage unit 30 may store usable function management information. The usable function management information is information for managing a function that can be used by each user. For example, user identification information for identifying a user and function information (link function) indicating a function that can be used by the user. Information (which may include information). As described above, the functions that can be used by the user are, for example, functions provided free of charge to the user, functions purchased by the user, and the like. Also good. The user identification information is user account information such as a user ID and a name, for example. By referring to the available function management information, a function that can be used by each user is specified (identified). The available function management information is updated, for example, every time a function is provided to the user (for example, every time a function is provided to the user free of charge or for a fee).

  The control unit 36 controls the operation of each unit of the server 14. The control unit 36 includes a specifying unit 38.

  The specifying unit 38 receives device identification information for identifying a device, and specifies function information indicating a function associated with the device identification information in the device function management information 32 stored in the storage unit 30. As a result, the function of the device is specified (identified). For example, the device identification information is transmitted from the terminal device 16 to the server 14, and the function information indicating the function associated with the device identification information is specified by the specifying unit 38. Information relating to the function (for example, function information and function description information) is transmitted from the server 14 to the terminal device 16 and displayed on the terminal device 16, for example. As a result, information related to the function of the device specified by the device identification information is displayed on the terminal device 16.

  In addition, the specifying unit 38 receives device identification information for identifying each device to be linked, and the linkage function management information 34 stored in the storage unit 30 is associated with a combination of each device identification information. Specify linked function information that indicates the function. Thereby, the cooperation function performed by cooperating the function which each apparatus of cooperation object has is identified (identified). For example, a plurality of device identification information is transmitted from the terminal device 16 to the server 14, and the specifying unit 38 specifies cooperation function information indicating a cooperation function associated with the plurality of device identification information. Information related to the cooperation function (for example, cooperation function information and description information of the cooperation function) is transmitted from the server 14 to the terminal device 16 and displayed on the terminal device 16, for example. Thereby, information related to the cooperation function executed by the plurality of devices specified by the plurality of device identification information is displayed on the terminal device 16.

  For example, when one device is identified (for example, when one device is photographed), the specifying unit 38 receives device identification information for identifying the device, and the device function management information 32 includes the device identification information. The function information indicating the function associated with may be specified. Thereby, when one device is identified (for example, when one device is photographed), the function of the device is specified (identified). When a plurality of devices are identified (for example, when a plurality of devices are photographed), the specifying unit 38 receives device identification information for identifying each device included in the plurality of devices, and cooperates function management information 34. The link function information indicating the link function associated with each combination of device identification information may be specified. Thereby, when a plurality of devices are identified (for example, when a plurality of devices are photographed), a cooperation function that uses a function of the plurality of devices is specified (identified).

  Further, the specifying unit 38 receives function information indicating each function used for the link function, and displays the link function associated with the combination of each function information in the link function management information 34 stored in the storage unit 30. You may identify the cooperation function information to show. Thereby, the cooperation function performed by making each function of cooperation object cooperate is specified (identification). For example, a plurality of pieces of function information are transmitted from the terminal device 16 to the server 14, and the specifying unit 38 specifies the link function information indicating the link functions associated with the plurality of function information. In the same manner as described above, information related to a cooperative function executed by a plurality of devices specified by a plurality of function information is displayed on the terminal device 16.

  When a function that can be used by a user is managed, the specifying unit 38 receives user identification information for identifying the user, and the user identification information is stored in the available function management information stored in the storage unit 30. You may specify the function information which shows each function matched with. Thereby, the function group which can be used by the user is specified (identified). For example, user identification information is transmitted from the terminal device 16 to the server 14, and the function information indicating each function associated with the user identification information is specified by the specifying unit 38. Information regarding each function available to the user (for example, information indicating the name of each function) is transmitted from the server 14 to the terminal device 16 and displayed on the terminal device 16, for example. As a result, information related to each function available to the user specified by the user identification information is displayed on the terminal device 16. For example, the specifying unit 38 receives the device identification information and the user identification information, specifies the function information indicating the function associated with the device identification information in the device function management information 32, and in the usable function management information. The function information indicating the function associated with the user identification information is specified. As a result, a function that the device specified by the device identification information has and that can be used by the user specified by the user identification information is specified.

  The control unit 36 may execute a function purchase process and manage the purchase history. For example, when a paid function is purchased by a user, the control unit 36 may apply a charging process to the user.

  Further, the control unit 36 may execute functions related to image processing such as a character recognition function, a translation function, an image processing function, and an image forming function. Of course, the control unit 36 may execute a function related to processing other than image processing. By executing the character recognition function, a character in the image is recognized and character data indicating the character is generated. By executing the translation function, the characters in the image are translated into characters expressed in a specific language, and character data indicating the translated characters is generated. The image is processed by executing the image processing function. The control unit 36 receives, for example, scan data generated by executing a scan function from the image forming apparatus 10, and performs image processing such as a character recognition function, a translation function, an image processing function, and the like for the scan data. You may perform the function about. The control unit 36 may receive image data from the terminal device 16 and execute each function for the image data. The character data and image data generated by the control unit 36 are transmitted from the server 14 to the terminal device 16, for example. The server 14 may be used as an external device, and the cooperation function may be a function that uses functions of a plurality of devices including the server 14.

  Hereinafter, the configuration of the terminal device 16 will be described in detail with reference to FIG. FIG. 4 shows the configuration of the terminal device 16.

  The communication unit 40 is a communication interface, and has a function of transmitting data to other devices and a function of receiving data from other devices. The communication unit 40 may be a communication interface having a wireless communication function or a communication interface having a wired communication function.

  The camera 42 as an imaging unit generates image data (for example, still image data or moving image data) by imaging an imaging target. In addition to using the camera of the terminal device 16, the communication unit 40 receives image data captured by an external camera connected to a communication path such as a network, and the UI unit 46 displays the image data. The image data may be manipulated.

  The storage unit 44 is a storage device such as a hard disk or a memory (for example, SSD), and various programs, various data, address information of the server 14, address information of each device (for example, address information of the devices 10 and 12), identification Information about the identified device, information about the identified device to be linked, information about the function of the identified device, information about the linkage function, etc. are stored.

  The UI unit 46 is a user interface unit and includes a display unit and an operation unit. The display unit is a display device such as a liquid crystal display. The operation unit is, for example, an input device such as a touch panel, a keyboard, and a mouse. Of course, a user interface that serves as both a display unit and an operation unit (for example, a touch-type display or a device that electronically displays a keyboard or the like on the display) may be used.

  The control unit 48 controls the operation of each unit of the terminal device 16. The control unit 48 functions as, for example, a display control unit (control unit), and displays various types of information on the display unit of the UI unit 46.

  On the display unit of the UI unit 46, for example, an image captured by the camera 42, an image associated with a device identified as a device to be used (for example, a device used alone or a device to be linked), and a function An image associated with is displayed. The image associated with the device may be an image (still image or moving image) representing the device taken by the camera 42, or may be an image (for example, icon) schematically representing the device. The image data schematically represented may be stored in the server 14 and provided from the server 14 to the terminal device 16, or may be stored in advance in the terminal device 16, or stored in another device. The terminal device 16 may be provided from the other device. The image associated with the function is, for example, an image such as an icon representing the function.

  The device function management information 32 described above may be stored in the storage unit 44 of the terminal device 16. In this case, the device function management information 32 may not be stored in the storage unit 30 of the server 14. Similarly, the cooperation function management information 34 described above may be stored in the storage unit 44 of the terminal device 16. In this case, the cooperation function management information 34 may not be stored in the storage unit 30 of the server 14. The control unit 48 of the terminal device 16 includes the specifying unit 38 described above, identifies the device based on the device identification information, may specify the function of the device, and has a cooperation function that uses a plurality of functions. You may specify. In this case, the server 14 may not have the specifying unit 38.

  When the usable function management information is created, the usable function management information may be stored in the storage unit 44 of the terminal device 16. In this case, the available function management information may not be stored in the storage unit 30 of the server 14. The control unit 48 of the terminal device 16 may manage the purchase history of the function by the user. In this case, the control unit 36 of the server 14 may not have the management function. The control unit 48 of the terminal device 16 may specify a function that can be used by the user based on the user identification information.

  As yet another example, the device function management information 32 and the link function management information 34 may be stored in a device such as the device 10 or 12, and the device such as the device 10 or 12 has the specifying unit 38. May be. In other words, the processing (for example, device identification processing, function identification processing, cooperation function identification processing, etc.) by the specifying unit 38 of the server 14 may be performed by the server 14 or the terminal device 16. Alternatively, it may be performed in devices such as the devices 10 and 12.

  In the present embodiment, as an example, by applying AR (Augmented Reality) technology (augmented reality technology), device identification information is acquired and a device is identified. For example, by applying AR technology, device identification information of a device used alone is acquired and the device is identified, and device identification information of a device to be linked is acquired to identify a device to be linked. Is done. A known AR technique is used as the AR technique. For example, a marker type AR technique using a marker such as a two-dimensional barcode, a markerless type AR technique using an image recognition technique, a position information AR technique using position information, and the like are used. Of course, the device identification information may be acquired and the device may be identified without applying the AR technology. For example, if the device is connected to the network, the device may be identified based on the IP address, or the device ID may be read to identify the device. Furthermore, if the device or terminal device has various wireless communication functions such as infrared communication, visible light communication, Wi-Fi (Wireless Fidelity) (registered trademark), Bluetooth (Bluetooth (registered trademark)), these wireless communication functions are provided. The device may be identified by acquiring the device ID of the device to be used and linked to execute the linkage function.

  The device identification information acquisition process will be described in detail below with reference to FIG. As an example, a case where device identification information of the image forming apparatus 10 is acquired will be described. FIG. 5 schematically shows the appearance of the image forming apparatus 10. Here, a process for acquiring the device identification information by applying the marker AR technology will be described. The casing of the image forming apparatus 10 is provided with a marker 50 such as a two-dimensional barcode. The marker 50 is information in which device identification information of the image forming apparatus 10 is encoded. The user activates the camera 42 of the terminal device 16 and photographs the marker 50 provided in the image forming apparatus 10 to be used by the camera 42. Thereby, image data representing the marker 50 is generated. The image data is transmitted from the terminal device 16 to the server 14, for example. In the server 14, the control unit 36 extracts device identification information by applying a decoding process to the marker image represented in the image data. Thereby, the image forming apparatus 10 to be used (the image forming apparatus 10 having the photographed marker 50) is identified. The specifying unit 38 of the server 14 specifies function information indicating a function associated with the extracted device identification information in the device function management information 32. Thereby, the function of the image forming apparatus 10 to be used is specified (identified).

  Note that the control unit 48 of the terminal device 16 may extract the device identification information by applying a decoding process to the image data representing the marker 50. In this case, the extracted device identification information is transmitted from the terminal device 16 to the server 14. The specifying unit 38 of the server 14 specifies function information indicating the function associated with the device identification information transmitted from the terminal device 16 in the device function management information 32. When the device function management information 32 is stored in the storage unit 44 of the terminal device 16, the control unit 48 of the terminal device 16 has a function associated with the extracted device identification information in the device function management information 32. The function information indicating may be specified.

  The marker 50 may include encoded function information indicating the function of the image forming apparatus 10. In this case, by applying the decoding process to the image data representing the marker 50, the device identification information of the image forming apparatus 10 is extracted, and the function information indicating the function of the image forming apparatus 10 is also extracted. Thereby, the image forming apparatus 10 is specified (identified), and the function of the image forming apparatus 10 is specified (identified). This decoding process may be performed by the server 14 or the terminal device 16.

  When executing a cooperation function that uses a function of a plurality of devices, device identification information of each device is acquired by photographing a marker of each device to be linked, thereby identifying (identifying) the cooperation function. .

  When acquiring the device identification information by applying the markerless AR technology, for example, the user photographs all or a part of the appearance of the device to be used (for example, the image forming apparatus 10) with the camera 42 of the terminal device 16. Of course, it is useful to obtain information for identifying the device such as the name of the device to be used (for example, product name) and the model number from the appearance. Appearance image data representing all or a part of the appearance of the device to be used is generated by photographing. The appearance image data is transmitted from the terminal device 16 to the server 14, for example. In the server 14, the control unit 36 identifies a device to be used based on the appearance image data. For example, the storage unit 30 of the server 14 stores, for each device, appearance image association information indicating association between appearance image data representing all or part of the appearance of the device and device identification information of the device. Has been. For example, the control unit 36 compares the appearance image data transmitted from the terminal device 16 with each appearance image data included in the appearance image association information, and based on the comparison result, device identification information of the device to be used Is identified. For example, the control unit 36 extracts the appearance feature of the device to be used from the appearance image data transmitted from the terminal device 16, and is the same as the appearance feature in the appearance image data group included in the appearance image association information. Alternatively, appearance image data representing similar features is specified, and device identification information associated with the appearance image data is specified. Thereby, a device to be used (device photographed by the camera 42) is identified. As another example, when a device name (for example, a product name) or model number is photographed and appearance image data representing the name or model number is generated, it is based on the name or model number represented in the appearance image data. Thus, the device to be used may be identified. The specifying unit 38 of the server 14 specifies function information indicating each function associated with the specified device identification information in the device function management information 32. Thereby, the function which the apparatus (for example, image forming apparatus 10) of utilization object has is specified.

  Note that the control unit 48 of the terminal device 16 compares the appearance image data representing all or part of the appearance of the device to be used (for example, the image forming apparatus 10) with each appearance image data included in the appearance image association information. Then, based on the comparison result, the device identification information of the device to be used may be specified. The appearance image association information may be stored in the storage unit 44 of the terminal device 16. In this case, the control unit 48 of the terminal device 16 specifies the device identification information of the device to be used by referring to the appearance image association information stored in the storage unit 44 of the terminal device 16. As another example, the control unit 48 of the terminal device 16 acquires the appearance image association information from the server 14 and refers to the appearance image association information to identify the device identification information of the device to be used. Also good.

  When executing a linkage function that uses a plurality of functions of a plurality of devices, the device identification information of each device is acquired by photographing all or part of the appearance of each device to be linked. Is identified (identified).

  When the device identification information is acquired by applying the position information AR technology, for example, by using a GPS (Global Positioning System) function, position information indicating a position where the device (for example, the image forming apparatus 10) is installed is obtained. To be acquired. For example, each device has a GPS function and acquires device position information indicating the position of the device itself. The terminal device 16 outputs information indicating an acquisition request for device position information to a device to be used, and receives device position information of the device from the device as a response to the acquisition request. The device position information is transmitted from the terminal device 16 to the server 14, for example. In the server 14, the control unit 36 identifies a device to be used based on the device position information. For example, the storage unit 30 of the server 14 stores, for each device, position association information indicating association between device position information indicating a position where the device is installed and device identification information of the device. Yes. The control unit 36 specifies device identification information associated with the device position information transmitted from the terminal device 16 in the position association information. Thereby, the device to be used is specified (identified). The specifying unit 38 of the server 14 specifies function information indicating each function associated with the specified device identification information in the device function management information 32. Thereby, the function of the device to be used (for example, the image forming apparatus 10) is specified (identified).

  Note that the control unit 48 of the terminal device 16 may specify the device identification information associated with the position information of the device to be used in the position association information. The position association information may be stored in the storage unit 44 of the terminal device 16. In this case, the control unit 48 of the terminal device 16 specifies the device identification information of the device to be used by referring to the position association information stored in the storage unit 44 of the terminal device 16. As another example, the control unit 48 of the terminal device 16 may acquire the position association information from the server 14 and specify the device identification information of the device to be used by referring to the position association information. .

  When executing a linkage function using a plurality of devices, device position information of each device to be linked is acquired, and device identification information of each device is specified based on each device location information. Thereby, a cooperation function is specified (identified).

  Hereinafter, the image forming system according to the present embodiment will be described in more detail.

  The device function management information 32 will be described in detail with reference to FIG. FIG. 6 shows an example of a device function management table as the device function management information 32. In the device function management table, for example, a device ID, information indicating a device name (for example, device type), information indicating a function of the device (function information), and an image ID are associated with each other. Yes. The device ID and the device name correspond to an example of device identification information. The image ID is an example of image identification information for identifying an image representing a device (for example, an image representing an external appearance of the device, an image schematically representing the device (for example, an icon), or the like). The device function management table may not include the image ID. For example, a device whose device ID is “B” is a multifunction peripheral (an image forming apparatus having a plurality of image forming functions), and has functions such as a print function and a scan function. The device is associated with an image ID for identifying an image representing the device. Image data representing the device is stored in, for example, the storage unit 30 of the server 14 or another device.

  For example, by applying the AR technology, a device ID for identifying a device to be used is acquired. The specifying unit 38 of the server 14 specifies the device name, function, and image ID associated with the device ID by referring to the device function management table. Thereby, the device to be used is identified. For example, information indicating the device name and image data representing the device are transmitted from the server 14 to the terminal device 16 and displayed on the UI unit 46 of the terminal device 16. An image representing a device is displayed as an image associated with the device. Of course, the image itself captured by the camera 42 may be displayed on the UI unit 46 of the terminal device 16. Further, in the UI unit 46 of the terminal device 16, when an image associated with a device (for example, an image photographed by the camera 42 or an image schematically representing the device) is designated by the user, the function of the device is provided. Information (for example, function information and function explanation information) may be transmitted from the server 14 to the terminal device 16 and displayed on the UI unit 46 of the terminal device 16.

  Hereinafter, the cooperative function management information 34 will be described in detail with reference to FIG. FIG. 7 shows an example of a cooperation function management table as the cooperation function management information 34. In the linkage function management table, as an example, a combination of device IDs, information indicating a device name to be linked (for example, the type of each device), and information indicating a linkage function (linkage function information) are associated with each other. ing. For example, a device with a device ID “A” is a PC (personal computer), and a device with a device ID “B” is a multifunction device. By linking the PC (A) and the multifunction peripheral (B), for example, a “scan transfer function” or a “print function” is realized as a link function. The “scan transfer function” is a function for transferring image data generated by scanning by the multifunction peripheral (B) to the PC (A). The “print function” is a function for transmitting data (for example, image data or document data) stored in the PC (A) to the multifunction device (B) and printing the data on the multifunction device (B).

  Hereinafter, with reference to FIG. 8, processing when the device is used alone will be described. FIG. 8 shows an example of a device used alone. As an example, it is assumed that the image forming apparatus 10 is a device used alone. The image forming apparatus 10 is, for example, a multifunction machine. The image forming apparatus 10 is a device that exists in a real space. The terminal device 16 illustrated in FIG. 8 is a device that exists in the real space, and is a mobile terminal device such as a smartphone or a mobile phone.

  For example, a marker 50 such as a two-dimensional barcode is provided on the housing of the image forming apparatus 10. When the marker type AR technology or the markerless type AR technology is used, the user photographs the image forming apparatus 10 to be used by the camera 42 of the terminal device 16 (for example, a smartphone). As a result, image data representing the marker 50 or appearance image data representing the appearance of the image forming apparatus 10 is generated. A device display screen 52 is displayed on the display unit of the UI unit 46 of the terminal device 16, and a device image 54 associated with the image forming apparatus 10 is displayed on the device display screen 52. The device image 54 is, for example, an image generated by photographing with the camera 42 (an image having a size at the time of photographing may be used, or an enlarged or reduced image may be used).

  Image data generated by photographing by the camera 42 is transmitted from the terminal device 16 to the server 14. In the server 14, the control unit 36 applies the decoding process to the image data to extract the device identification information of the image forming apparatus 10, thereby identifying the image forming apparatus 10. As another example, appearance image data representing the appearance of the image forming apparatus 10 may be generated, and the appearance image data may be transmitted from the terminal device 16 to the server 14. In this case, in the server 14, the control unit 36 specifies the device identification information of the image forming apparatus 10 by referring to the appearance image association information. Thereby, the image forming apparatus 10 is identified.

  Further, the specifying unit 38 of the server 14 specifies (identifies) the function of the image forming apparatus 10 by referring to the device function management information 32 (for example, the device function management table shown in FIG. 6). This point will be described in detail with reference to FIG. The image forming apparatus 10 is assumed to be a “multifunction machine (B)” as an example. The specifying unit 38 specifies the function associated with the multifunction peripheral (B) in the device function management table shown in FIG. Thereby, the function which a multifunctional machine (B) has is specified. Information regarding the identified function is transmitted from the server 14 to the terminal device 16. Of course, the process for identifying a device or a function may be performed by the terminal device 16.

  It should be noted that the device display screen 52 is a schematic image (not an image itself obtained by photographing) prepared in advance associated with the identified image forming apparatus 10 instead of an image generated by photographing by the camera 42. An image (for example, an icon) or an image generated by photographing with an external camera may be displayed as the device image 54.

  For example, when using image data obtained by shooting a device, the appearance of the current device itself (for example, an appearance that reflects scratches, notes, stickers attached to the device, etc.) is reflected in the image. As a result, the user can visually understand the difference from other similar devices.

  When using a schematic image, the data of the schematic image is transmitted from the server 14 to the terminal device 16, for example. For example, when the image forming apparatus 10 is identified, the specifying unit 38 of the server 14 associates with the image forming apparatus 10 by referring to the device function management table (device function management information 32) illustrated in FIG. Identify a schematic image. The schematic image data is transmitted from the server 14 to the terminal device 16, and the schematic image is displayed on the device display screen 52 as the device image 54. Schematic image data may be stored in the terminal device 16 in advance. In this case, when the image forming apparatus 10 is identified, the device image 54 stored in the terminal device 16 is displayed on the device display screen 52. Schematic image data may be stored in a device other than the server 14 or the terminal device 16.

  When a device is identified, information indicating the name of the device may be transmitted from the server 14 to the terminal device 16, and the device name may be displayed on the device display screen 52 in the terminal device 16. In the example illustrated in FIG. 8, the image forming apparatus 10 is a multifunction peripheral, and the name “multifunction peripheral (B)” is displayed.

  When the function of the image forming apparatus 10 is specified, for example, as illustrated in FIG. 9, the control unit 48 of the terminal device 16 displays a function display screen 56 on the UI unit 46 of the terminal device 16, and the function display screen is displayed. 56 displays information about the function. As information about a function, for example, a button image for instructing execution of the function is displayed. Since the multifunction peripheral (B) as the image forming apparatus 10 has, for example, a print function, a scan function, a copy function, and a facsimile function, button images for instructing execution of these functions are displayed as functions. It is displayed on the screen 56. For example, when a user designates a button image representing a print function using the terminal device 16 and instructs execution of the print function, execution instruction information indicating an instruction to execute the print function is transmitted from the terminal device 16 to the image forming apparatus 10. Sent. The execution instruction information includes control data for executing the print function, data such as image data to which the print function is applied, and the like. Upon receiving the execution instruction information, the image forming apparatus 10 executes printing according to the execution instruction information.

  FIG. 10 shows another example of the function display screen. As shown in FIG. 8, the function display screen 58 is a screen displayed on the UI unit 46 of the terminal device 16 when using a single device. As described above, the device to be used (for example, the image forming apparatus 10) is specified, and the function of the device to be used is specified. Moreover, the function information which shows the function with which the user using the apparatus of utilization object is matched with user identification information, ie, the function which the said user can utilize, may be specified. Furthermore, since the function of the device to be used is specified, the function that the device to be used does not have may be specified in the function group to be provided. Such information may be displayed on the function display screen 58.

  On the function display screen 58 shown in FIG. 10, as an example of function information, a button image 60 representing the function A, a button image 62 representing the function B, and a button image 64 representing the function C are displayed. Yes. The function A is a function that a device to be used (for example, the identified image forming apparatus 10) has, and is a function that can be used by the user. The function B is a function possessed by the device to be used and cannot be used by the user. When the user receives the provision of the function B, the user can use the function B. If the function B is a paid function, the user can use the function B by purchasing the function B. If the function B is a free function, the user receives the function B free of charge. The function B can be used. Function C is a function that the target device does not have, that is, a function that the target device does not support. The control unit 48 of the terminal device 16 may change the display mode of the button image according to whether or not the function represented by the button image is a function of the device to be used. The control unit 48 may change the display mode of the button image depending on whether or not the function represented by the button image is a function that can be used by the user. For example, the control unit 48 may change the color and shape of the button image. In the example shown in FIG. 10, the button images 60, 62, and 64 are displayed in different colors. As an example, a button image (for example, a button image 60 representing the function A) that represents a function that the target device has and that can be used by the user is displayed in blue. A button image (a button image 62 representing the function B) that represents a function that the target device has and cannot be used by the user is displayed in yellow. A button image representing a function that the device to be used does not have (for example, a button image 64 representing the function C) is displayed in gray. As another example, the control unit 48 may change the shape of the button images 60, 62, and 64, or may change the font of the function display name. Of course, the display mode may be changed by another method. This informs the user whether or not each function can be used with high visibility.

  For example, when the user designates the button image 60 representing the function A using the terminal device 16 and gives an instruction to execute the function A, the execution instruction information indicating the execution instruction of the function A is used from the terminal device 16 as the usage target. Sent to other devices. The execution instruction information includes control data for executing function A, data such as image data to which processing by function A is applied, and the like. When the target device receives the execution instruction information, it executes the function A according to the execution instruction information. For example, when the device to be used is the image forming apparatus 10 and the function A is the scan transfer function, the scan function is executed by the image forming unit 20 of the image forming apparatus 10, and thereby the scan data (image data) is obtained. Generated. The scan data is transmitted from the image forming apparatus 10 to a transmission destination (for example, the terminal device 16) set.

  When the user designates the button image 62 representing the function B using the terminal device 16 and instructs the provision of the function B, the providing process is executed. When the providing process is performed by the server 14, the terminal device 16 accesses the server 14. As a result, a screen (for example, a website) for receiving the provision of the function B is displayed on the UI unit 46 of the terminal device 16 as information for enabling the user to use the function B. By performing the provision procedure on this screen, the user can use the function B. For example, a Web browser program is stored in the terminal device 16, and access from the terminal device 16 to the server 14 is realized by using the Web browser. When the user accesses the server 14 using a Web browser, a function providing screen (for example, a Web site) is displayed on the UI unit 46 of the terminal device 16 and functions are provided via the Web site. Of course, the providing process may be performed by a server other than the server 14. As another example, the UI unit 46 of the terminal device 16 has a use permission request screen (for example, Web) for requesting the administrator or the like to use the function B as information for enabling the user to use the function B. Site) may be displayed. If the administrator or the like is requested to permit the use of the function B via the use permission request screen and the permission is obtained, the user can use the function B.

  Hereinafter, with reference to FIG. 11, processing when the cooperation function is used will be described. FIG. 11 shows an example of a cooperation target device. As an example, it is assumed that the image forming apparatus 10 and a projector as the device 12 (hereinafter sometimes referred to as a projector 12) are used as devices to be linked. The image forming apparatus 10, the projector 12, and the terminal device 16 are devices that exist in the real space.

  For example, a marker 50 such as a two-dimensional barcode is provided on the casing of the image forming apparatus 10, and a marker 66 such as a two-dimensional barcode is provided on the casing of the projector 12. The marker 66 is information obtained by coding the device identification information of the projector 12. When the marker type AR technology or the markerless type AR technology is used, the user captures the image forming apparatus 10 and the projector 12 to be linked by the camera 42 of the terminal device 16 (for example, a smartphone). In the example illustrated in FIG. 11, the image forming apparatus 10 and the projector 12 are photographed together in a state where both the image forming apparatus 10 and the projector 12 are included in the photographing region of the camera 42. Thereby, image data representing each of the markers 50 and 66 is generated. A device display screen 68 is displayed on the display unit of the UI unit 46 of the terminal device 16, and a device image 70 associated with the image forming apparatus 10 and a device image 72 associated with the projector 12 are displayed on the device display screen 68. . The device images 70 and 72 are, for example, images generated by photographing with the camera 42 (an image having a size at the time of photographing may be used, or an enlarged or reduced image may be used).

  Image data generated by photographing by the camera 42 is transmitted from the terminal device 16 to the server 14. In the server 14, the control unit 36 applies the decoding process to the image data to extract the device identification information of the image forming apparatus 10 and the device identification information of the projector 12, whereby the image forming apparatus 10 and the projector 12 are connected. Identify. As another example, appearance image data representing the appearances of the image forming apparatus 10 and the projector 12 may be generated and transmitted from the terminal device 16 to the server 14. In this case, in the server 14, the control unit 36 specifies the device identification information of the image forming apparatus 10 and the device identification information of the projector 12 by referring to the appearance image association information. Thereby, the image forming apparatus 10 and the projector 12 are identified.

  Further, the specifying unit 38 of the server 14 refers to the cooperation function management information 34 (for example, the cooperation function management table illustrated in FIG. 7), thereby determining the functions of the image forming apparatus 10 and the functions of the projector 12. Identify (identify) the linkage function to use. This point will be described in detail with reference to FIG. The image forming apparatus 10 is assumed to be a “multifunction machine (B)” as an example, and the projector 12 is assumed to be a “projector (C)” as an example. The specifying unit 38 specifies the link function associated with the combination of the multifunction peripheral (B) and the projector (C) in the link function management table shown in FIG. Thereby, the cooperation function performed by cooperating the multifunction peripheral (B) and the projector (C) is specified. Information regarding the identified linkage function is transmitted from the server 14 to the terminal device 16. Of course, the processing for identifying the device and the linkage function may be performed by the terminal device 16.

  The device display screen 68 includes an image prepared in advance (for example, a schematic image (for example, an icon)) associated with the identified image forming apparatus 10 instead of an image generated by photographing with the camera 42, An image generated by photographing with an external camera may be displayed as the device image 70. Similarly, an image prepared in advance associated with the identified projector 12 or an image generated by photographing with an external camera may be displayed as the device image 72. As described above, the schematic image data may be transmitted from the server 14 to the terminal device 16, stored in the terminal device 16 in advance, or stored in other devices. Also good.

  When a device is identified, information indicating the name of the device may be transmitted from the server 14 to the terminal device 16, and the device name may be displayed on the device display screen 68 in the terminal device 16. In the example shown in FIG. 11, the name “MFP (B)” of the image forming apparatus 10 and the name “Projector (C)” of the projector 12 are displayed.

  Even when a plurality of devices are photographed, the specifying unit 38 of the server 14 may specify the functions of each device by referring to the device function management information 32. In the example illustrated in FIG. 11, the specifying unit 38 may specify the function of the image forming apparatus 10 and the function of the projector 12. Information regarding the identified function may be transmitted from the server 14 to the terminal device 16.

  When the cooperation function is specified, for example, as illustrated in FIG. 12, the control unit 48 of the terminal device 16 displays the function display screen 74 on the UI unit 46 of the terminal device 16, and the function display screen 56 relates to the cooperation function. Display information. For example, a button image for instructing execution of the cooperation function is displayed as information on the cooperation function. By linking the multifunction machine (B) and the projector (C), for example, a cooperation function for projecting an image generated by scanning by the multifunction machine (B) by the projector (C), or being projected by the projector (C). The cooperative function of printing the existing image by the multifunction peripheral (B) can be executed. A button image for instructing execution of these linkage functions is displayed on the function display screen 74. For example, when the user designates a button image using the terminal device 16 and instructs execution of the cooperation function, execution instruction information indicating the execution instruction of the cooperation function is transmitted from the terminal device 16 to the image forming apparatus 10 and the projector 12. Sent. Upon receiving the execution instruction information, the image forming apparatus 10 and the projector 12 execute the cooperation function designated by the user.

  A device to be linked may be designated by a user operation. For example, when the image forming apparatus 10 and the projector 12 are photographed by the camera 42, as shown in FIG. 11, the UI unit 46 of the terminal device 16 has a device image 70 associated with the image forming apparatus 10 and the projector 12. An attached device image 72 is displayed. The image associated with the device may be an image obtained by photographing with the camera 42, or may be an image prepared in advance (for example, a schematic image (eg, icon)) associated with the identified device. May be. When the user designates device images 70 and 72 on the device display screen 68, the image forming apparatus 10 and the projector 12 are designated as devices to be linked. For example, when the user designates the device image 70, the marker type AR technology or the markerless type AR technology is applied to the device image 70, and thereby the image forming apparatus 10 is specified (identified). Similarly, when the user designates the device image 72, the marker type AR technology or the markerless type AR technology is applied to the device image 72, and thereby the projector 12 is specified (identified). Thereby, the cooperation function executed by the image forming apparatus 10 and the projector 12 is specified, and information related to the cooperation function is displayed on the UI unit 46 of the terminal device 16.

  As another example, when the user touches the device image 70 on the device display screen 68 and operates the device image 72 with a finger or the like (for example, tracing with the finger or the like), the device images 70 and 72 are designated. The image forming apparatus 10 and the projector 12 may be designated as devices to be linked. The order of touching the device images 70 and 72 and the direction of tracing may be the reverse of the above example. Of course, screen contact means other than a finger such as a pen for tracing the device display screen 68 may be used. When the user connects the device image 70 and the device image 72, the device images 70 and 72 may be specified, and the image forming apparatus 10 and the projector 12 may be specified as devices to be linked. When the user overlaps the device image 70 and the device image 72, the device images 70 and 72 may be specified, and the image forming apparatus 10 and the projector 12 may be specified as devices to be linked. The device to be linked may be specified by a drawing operation such as adding a circle, or the device to be linked can be specified by specifying the device image associated with the device to be linked within a preset time. Good. When canceling the cooperation, the user may specify a device to be canceled on the device display screen 68 or may press a cooperation cancellation button. When a device that is not a collaboration target is reflected in the image, the device may be excluded from the collaboration target device group by designating the device on the device display screen 68. The device to be released may be designated by a preset operation such as adding a cross mark.

  Each device to be linked may be shot separately. For example, each device to be linked is identified by dividing the shooting by the camera 42 into a plurality of times. When the shooting by the camera 42 is divided into a plurality of times, the device identification information of the device identified at each shooting time is stored in the storage unit of the server 14 or the terminal device 16. For example, the image forming apparatus 10 is photographed with the image forming apparatus 10 included in the photographing area of the camera 42, and then the projector 12 is photographed with the projector 12 included in the photographing area of the camera 42. Thereby, image data for the image forming apparatus 10 and image data for the projector 12 are generated, and the marker-type AR technique or the markerless-type AR technique is applied to each image data. Identification (identification) is performed, and a cooperation function that uses the functions of the image forming apparatus 10 and the projector 12 is identified (identification). For example, the devices to be linked are arranged at positions close to each other, and both the image forming apparatus 10 and the projector 12 are not necessarily included together in the shooting area of the camera 42. In some cases, it is possible to respond by changing the angle of the shooting area of the camera 42, or by expanding or reducing the shooting area. It is also possible to identify other devices.

  As another example, a device to be linked may be set in advance as a basic linked device. For example, it is assumed that the image forming apparatus 10 is preset as a basic cooperation device. The device identification information of the basic linkage device may be stored in advance in the storage unit of the server 14 or the terminal device 16. The user may specify the basic cooperation device using the terminal device 16. When the basic cooperation device is set, the user uses the camera 42 of the terminal device 16 to photograph a device to be linked other than the basic cooperation device. As a result, the device to be linked is specified (identified), and the linked function that uses the functions of the basic linked device and the captured device is specified (identified).

  In the example shown in FIGS. 11 and 12, the cooperation function is a function that uses a device as hardware, but the cooperation function is a function that uses a function realized by software (application). Also good. For example, instead of a device image, a function image (for example, an image such as an icon) associated with a function realized by software is displayed on the UI unit 46 of the terminal device 16, and a plurality of function images are designated by the user. A linkage function that uses a plurality of functions associated with the plurality of function images may be specified (identified). For example, the cooperation function may be specified by designating a device image associated with a function displayed on the home screen of a smartphone or the desktop screen of a personal computer. Of course, the device image associated with the device as hardware and the function image associated with the function realized by software are displayed on the UI unit 46 of the terminal device 16, and the device image and the function image are designated by the user. In this case, a cooperation function that uses a device associated with the device image and a function associated with the function image may be identified.

  In the above example, the marker type AR technique or the markerless type AR technique is used, but the position information AR technique may be used. For example, the terminal device 16 has a GPS function, acquires terminal position information indicating the position of the terminal device 16 itself, and transmits the terminal position information to the server 14. The control unit 36 of the server 14 refers to the position association information indicating the association between the device position information indicating the installation position of the device and the device identification information, and is set in advance with reference to the position of the terminal device 16. Each device arranged in is identified as a cooperation candidate device. For example, it is assumed that the multifunction device, the PC, the printer, and the scanner are arranged in a range set in advance with reference to the position of the terminal device 16. In this case, a multifunction device, a PC, a printer, and a scanner are specified as cooperation candidate devices. The device identification information of each cooperation candidate device is transmitted from the server 14 to the terminal device 16 and displayed on the UI unit 46 of the terminal device 16. As device identification information, an image of a cooperation candidate device may be displayed, or a character string such as a device ID may be displayed. The user designates a cooperation target device from the cooperation candidate device group displayed on the UI unit 46. The device identification information of the cooperation target device designated by the user is transmitted from the terminal device 16 to the server 14, and the server 14 identifies the cooperation function based on the device identification information of the cooperation target device. The UI unit 46 of the terminal device 16 displays information related to the cooperation function. In addition, the identification process of a cooperation candidate apparatus and the identification process of a cooperation function may be performed by the terminal device 16.

  Note that if the device to be imaged is not identified even when the AR technology or the like is applied, the device image representing the device to be imaged may not be displayed on the device display screen. This improves the visibility of the identified device. For example, when an identified device and an unidentified device are mixed and each is photographed by the camera 42, the device image representing the unidentified device is hidden. Thereby, since the device image representing the identified device is displayed without being buried in the device image representing the unidentified device, the visibility of the identified device is improved. In addition, a device image representing the identified device may be highlighted and displayed. For example, the device image representing the identified device may be displayed in a specific color, the edge of the device image may be displayed highlighted, the device image may be displayed enlarged, The device image may be displayed in three dimensions, or the device image may be displayed with vibration. This improves the visibility of the identified device.

  Hereinafter, processing when the function of the device is executed will be described. As an example, a process when executing the cooperation function will be described. In this case, a connection request is transmitted from the terminal device 16 to the cooperation target device, and the terminal device 16 and the cooperation target device are connected. The connection process will be described below with reference to FIG. FIG. 13 is a sequence diagram showing the processing.

  First, the user uses the terminal device 16 to instruct activation of an application (program) for executing a device function. The control unit 48 of the terminal device 16 activates the application according to the instruction (S01). The application may be stored in advance in the storage unit 44 of the terminal device 16 or may be downloaded from the server 14 or the like.

  Next, by applying the marker type AR technology, the markerless type AR technology, or the position information AR technology, a device to be linked is identified (S02). Of course, a device to be linked may be identified by a technology other than the AR technology. When applying the marker type AR technology or the markerless type AR technology, for example, the user photographs the device to be linked by the camera 42 of the terminal device 16. As an example, when the image forming apparatus 10 (multifunction machine (B)) and the projector 12 (projector (C)) are used as linked devices, the user photographs the image forming apparatus 10 and the projector 12 with the camera 42. Device identification information of each of the image forming apparatus 10 and the projector 12 is acquired, and the image forming apparatus 10 and the projector 12 are identified as devices to be linked. When the position information AR technology is applied, the position information of each of the image forming apparatus 10 and the projector 12 is acquired, and the device identification information of each of the image forming apparatus 10 and the projector 12 is specified based on the position information. The forming apparatus 10 and the projector 12 are identified.

  For example, when an instruction to display a cooperation function is given by the user, a cooperation function that uses functions of a plurality of identified devices is identified. Information on the identified cooperation function is displayed on the UI unit 46 of the terminal device 16 (S03). The identification process of the cooperation function may be performed by the server 14 or the terminal device 16.

  Next, in the terminal device 16, when the cooperation function to be executed is designated by the user, the terminal device 16 sends a connection request to the cooperation target devices (for example, the image forming apparatus 10 and the projector 12) that execute the cooperation function. The information shown is transmitted (S04). For example, when the address information indicating the address of the cooperation target device is stored in the server 14, the terminal device 16 acquires the address information of the cooperation target device from the server 14. When the address information is included in the device identification information, the terminal device 16 may acquire each address information from the identification information of the cooperation target device. As another example, the terminal device 16 may store address information of devices to be linked. Of course, the terminal device 16 may acquire the address information of the cooperation target device by another method. The terminal device 16 transmits information indicating a connection request to the cooperation target devices (for example, the image forming apparatus 10 and the projector 12) using the address information of the cooperation target devices (for example, the image forming apparatus 10 and the projector 12).

  Upon receiving the information indicating the connection request, the image forming apparatus 10 and the projector 12 permit or do not permit connection with the terminal device 16 (S05). For example, when the image forming apparatus 10 and the projector 12 correspond to devices that are not permitted to be connected, or when the number of devices that request connection exceeds the upper limit, the connection is not permitted. When the connection from the terminal device 16 is permitted, the change operation may be prohibited so that the setting information unique to the image forming apparatus 10 and the projector 12 is not changed from the terminal device 16. For example, it may be prohibited to change the color parameters of the image forming apparatus 10 or the set time when shifting to power saving. As a result, the security for the cooperation target device is improved. As another example, when the devices are linked, the change of the setting information may be limited as compared to the case where the devices are used alone without being linked. For example, fewer setting item changes may be permitted as compared to the case where the device is used alone. In addition, browsing of personal information of other users such as operation history may be prohibited. Thereby, the security with respect to a user's personal information improves.

  Result information indicating permission or non-permission of the connection is transmitted from the image forming apparatus 10 and the projector 12 to the terminal device 16 (S06). When the connection to the image forming apparatus 10 and the projector 12 is permitted, communication is established between the terminal device 16 and the image forming apparatus 10 and the projector 12.

  Next, the user uses the terminal device 16 to instruct execution of the cooperation function (S07). In response to this instruction, execution instruction information indicating an instruction to execute the cooperation function is transmitted from the terminal device 16 to the image forming apparatus 10 and the projector 12 (S08). The execution instruction information transmitted to the image forming apparatus 10 includes information (for example, job information) indicating processing executed in the image forming apparatus 10, and the execution instruction information transmitted to the projector 12 includes the projector 12. Includes information (for example, job information) indicating the process executed in step.

  Receiving the execution instruction information, the image forming apparatus 10 and the projector 12 execute the function according to the execution instruction information (S09). For example, the image forming apparatus 10 can be linked to the image forming apparatus 10 such as a function of transferring scan data from the image forming apparatus 10 (multifunction machine (B)) to the projector 12 (PC (A)) and projecting data by the projector 12. When processing for transmitting and receiving data to and from the projector 12 is included, communication is established between the image forming apparatus 10 and the projector 12. In this case, for example, the execution instruction information transmitted to the image forming apparatus 10 includes the address information of the projector 12, and the execution instruction information transmitted to the projector 12 includes the address information of the image forming apparatus 10. Communication is established between the image forming apparatus 10 and the projector 12 using the address information.

  When the execution of the cooperation function is completed, information indicating completion of execution of the cooperation function is transmitted from the image forming apparatus 10 and the projector 12 to the terminal device 16 (S10). Information indicating that the execution of the cooperation function is completed is displayed on the UI unit 46 of the terminal device 16 (S11). When information indicating execution completion is not displayed even after a preset time has elapsed since the execution instruction is given, the control unit 48 of the terminal device 16 causes the UI unit 46 to display information indicating an error, The execution instruction information or information indicating the connection request may be transmitted to the image forming apparatus 10 and the projector 12 again.

  Next, the user confirms whether or not to cancel the cooperative state between the image forming apparatus 10 and the projector 12 (S12), and performs processing according to the presence or absence of the cancellation (S13). When canceling the linked state, the user uses the terminal device 16 to give a cancellation instruction. Thereby, the communication between the terminal device 16 and the image forming apparatus 10 and the projector 12 is released. Similarly, communication between the image forming apparatus 10 and the projector 12 is also released. If the cooperation state is not canceled, an execution instruction may be given continuously.

  Moreover, you may increase the apparatus of cooperation object. For example, device identification information of a third device may be acquired, and a cooperation function executed by linking three devices including the image forming apparatus 10 and the projector 12 may be specified. Information indicating that the image forming apparatus 10 and the projector 12 have already been identified as the cooperation function is stored in the server 14 or the terminal device 16.

  The device identification information indicating the cooperation target device and the cooperation function information indicating the executed cooperation function may be stored in the terminal device 16, the server 14, or the like. For example, user account information (user identification information) of a user who uses the terminal device 16 is acquired, the user account information, device identification information indicating a device to be linked, linkage function information indicating an executed linkage function, and May be created and stored in the terminal device 16, the server 14, or the like. The history information may be created by the terminal device 16 or the server 14. By referring to the history information, it is specified which cooperation function is executed using which device group.

  Note that the devices to be linked (for example, the image forming apparatus 10 and the projector 12) may store the user account information of the user who has requested the connection and the terminal identification information indicating the connection request source terminal device 16 as history information. Good. By referring to the history information, the user who uses the device is specified. For example, the history information may be used to specify a user when specifying a user who has used the device when the device is broken, or when charging a consumable item. The history information may be stored in the server 14 or the terminal device 16 or may be stored in another device.

  For example, the user account information is stored in advance in the storage unit 44 of the terminal device 16, and the control unit 48 of the terminal device 16 functions as an example of a user identification unit, and reads the user account information of the user from the storage unit 44. Thus, the user who uses the terminal device 16 is identified. When user account information of a plurality of users is stored in the storage unit 44, the user uses the terminal device 16 to specify his / her user account information. Thereby, the user account information of the user is read and the user is identified. As another example, the control unit 48 of the terminal device 16 may identify the user by reading the user account information of the user who has logged into the terminal device 16. As yet another example, when only one user account information is stored in the same terminal device 16, the control unit 48 of the terminal device 16 may identify the user by reading the user account information. . If no user account is set and user account information is not created, initial setting is performed, thereby creating user account information.

  The use history of the cooperation function is managed for each user, and information indicating the cooperation function used in the past by the user indicated by the read user account information may be displayed on the UI unit 46 of the terminal device 16. Information indicating the usage history may be stored in the terminal device 16 or may be stored in the server 14. Moreover, the information which shows the cooperation function currently used with the frequency more than the preset usage frequency may be displayed. By providing such a shortcut function, it is possible to reduce the time and effort of the user's operation regarding the linkage function.

  When a single function is executed, information indicating an instruction to execute the single function is transmitted from the terminal device 16 to a device that executes the single function. The device executes a single function according to the execution instruction.

  In the above example, the cooperation function can be executed by a plurality of devices. However, depending on the combination of the devices, the cooperation function is not necessarily executed. Similarly, depending on a combination of a plurality of functions (for example, a combination of software or a combination of software and hardware), it is not always possible to execute the linkage function. Hereinafter, this point will be described in detail.

  FIG. 14 shows an example of a combination of devices that cannot execute the linkage function. For example, it is assumed that the multifunction device (B) and the dryer (D) are identified as devices. As shown in FIG. 14A, a device display screen 68 is displayed on the UI unit 46 of the terminal device 16, and the identified devices (multifunction device (B) and dryer (D) are displayed on the device display screen 68. Device images 70 and 76 associated with ()) are displayed. When there is no linkage function that can be executed by the multifunction device (B) and the dryer (D), even if the multifunction device (B) and the dryer (D) are designated or identified as devices to be linked, Information is not displayed, and a message screen 78 is displayed on the UI unit 46 of the terminal device 16 as shown in FIG. The message screen 78 displays a message indicating that the cooperation function cannot be executed by the multifunction peripheral (B) and the dryer (D).

  The above processing will be described in more detail. When the multi-function device (B) and the dryer (D) are identified and designated as devices to be linked, as described above, the specifying unit 38 of the server 14 displays the linked function management information 34 (for example, shown in FIG. 7). By referring to the linked function management table, the linked function that uses the multifunction peripheral (B) and the dryer (D) is specified (identified). When a cooperation function that uses the multifunction peripheral (B) and the dryer (D) is registered in the cooperation function management table, the specifying unit 38 specifies the cooperation function as described above. On the other hand, when the cooperation function that uses the multifunction device (B) and the dryer (D) is not registered in the cooperation function management table, the specifying unit 38 uses the multifunction device (B) and the dryer (D). Judge that there is no linkage function. In this case, the control unit 36 of the server 14 outputs a message indicating that the cooperation function cannot be executed by the multifunction peripheral (B) and the dryer (D). This message is displayed on the UI unit 46 of the terminal device 16 as shown in FIG.

  As described above, even if there is no available linkage function, the linkage function can be changed depending on the operating status of the device, the environment in which the device is installed (peripheral environment), the change (update) of the device function, etc. May be available. In the above example, when condensation occurs in the environment where the multifunction peripheral (B) is installed, the condensation can be removed or prevented by the dryer (D). In this case, since the cooperation function using the multifunction peripheral (B) and the dryer (D) can be used, information indicating the cooperation function is displayed on the UI unit 46 of the terminal device 16. For example, the control unit 36 of the server 14 monitors the operating status of each device, the environment (peripheral environment) in which each device is set, the update status of the function of each device, and the like based on the monitoring result , Judge whether the cooperation function is available or not. The combination of the multifunction machine (B) and the dryer (D) will be described as an example. The control unit 36 determines that the surrounding environment of the multifunction machine (B) satisfies a specific condition (for example, the multifunction machine (B ) When the condensation occurs in the surrounding environment), it is determined that the cooperation function can be used, and the cooperation function using the dryer (D) is specified (identified). The same applies to the operation status of the device. When the identified or designated device group corresponds to the specific operation status, the control unit 36 determines that the cooperation function using the device group can be used. The same applies to the case where the function of the device is updated and the cooperation function can be used by the updated function.

  Similarly to the above, there are cases where the linkage function cannot be executed by a plurality of functions realized by a plurality of software, or the linkage function cannot be executed by a function realized by software and a device as hardware.

(Guidance processing)
In the present embodiment, for example, when an image associated with a device is designated, a function realized by the device or software as hardware capable of executing the cooperation function with the device is guided. The same applies to the linkage function using software. For example, when an image associated with a function realized by software is specified, the device or software as hardware capable of executing the linkage function together with the function is specified. The functions to be realized are guided. Hereinafter, examples of guidance processing according to the present embodiment will be described in detail.

Example 1
With reference to FIG. 15, the guidance process according to the first embodiment will be described. FIG. 15 shows an example of a device display screen according to the first embodiment. For example, it is assumed that the multifunction peripheral (B), the projector (C), and the dryer (D) are identified as devices. A device display screen 68 is displayed on the UI unit 46 of the terminal device 16, and the device display screen 68 is associated with the identified devices (multifunction device (B), projector (C), and dryer (D)). Device images 70, 72, and 76 are displayed.

  In this case, for example, it is assumed that the multifunction device (B) is selected by the user and the device image 70 associated with the multifunction device (B) is designated by the user. The multifunction device (B) corresponds to the first device, and the device image 70 associated with the multifunction device (B) corresponds to the first image associated with the first device. When the multifunction device (B) as the first device is designated by the user, the specifying unit 38 of the server 14 refers to the cooperation function management information 34 (for example, the cooperation function management table shown in FIG. 7). The second device capable of executing the cooperation function together with the multifunction peripheral (B) as the first device is specified. For example, it is assumed that the cooperation function can be executed by the multifunction peripheral (B) and the projector (C), and the cooperation function cannot be executed by the multifunction peripheral (B) and the dryer (D). That is, the cooperation function using the multifunction machine (B) and the projector (C) is registered in the cooperation function management table, while the cooperation function using the multifunction machine (B) and the dryer (D). Is not registered. In this case, the projector (C) is specified as the second device, and the control unit 36 of the server 14 controls notification of guidance of the projector (C) as the second device. Specifically, device identification information indicating the projector (C) is transmitted from the server 14 to the terminal device 16 under the control of the control unit 36. The control unit 48 of the terminal device 16 notifies the guidance of the projector (C) as the second device. For example, as illustrated in FIG. 15, the control unit 48 of the terminal device 16 causes the device display screen 68 to display an arrow 80 for guiding the projector (C) as a cooperation destination device. The arrow 80 indicates, for example, a device image 70 as a first image associated with the multifunction peripheral (B) as the first device, and a device image 72 as a second image associated with the projector (C) as the second device. It is an image that connects Of course, the control unit 48 may guide the projector (C) as the second device by a method other than the arrow. For example, the control unit 48 may guide the second device by outputting sound, or display a mark superimposed on the second image (for example, the device image 72) associated with the second device on the device display screen 68. The second image associated with the second device may be displayed on the device display screen 68 in a state where it can be distinguished from other images, or a character string indicating the cooperation destination may be displayed on the device display screen 68. It may be displayed.

  As described above, when the multifunction device (B) as the first device is designated by the user, the projector (C) as the second device capable of executing the cooperation function together with the multifunction device (B) Recommended as a candidate for equipment. Thereby, compared with the case where such a candidate is not recommended, the convenience of the user when designating a device necessary for the cooperation function is improved.

  The guidance process similar to the above is also performed when the user designates a functional image associated with the first function realized by software (an image corresponding to the first image). That is, when the functional image as the first image is designated by the user, the guidance to the second function capable of executing the cooperative function is notified together with the first function associated with the functional image. For example, guidance to the second image associated with the second function may be made, or the second function may be guided by voice or a character string. The second function may be a function realized by software, or may be a function of a device as hardware. Of course, when the device image associated with the first device is designated by the user, a guidance to the second function that is a function capable of executing the linkage function with the first device and realized by software is notified. May be.

  Since the dryer (D) is not a device that can execute the cooperation function together with the multifunction peripheral (B), an instruction for guiding the dryer (D) as a cooperation destination is not made.

  Environment (peripheral environment) where the multifunction device (B) as the first device is installed, operating status of the multifunction device (B) (for example, toner amount, paper amount, in use, processing end time, etc.), multifunction The dryer (D) may be guided as the second device by a change (update) in the function of the machine (B), a change (update) in the function of the dryer (D), or the like. For example, as described above, when condensation occurs in the surrounding environment of the multifunction peripheral (B) as the first device, the specifying unit 38 of the server 14 requires the second device necessary to remove the condensation. The dryer (D) is specified as In this case, as in the case of the projector (C), an instruction for guiding the dryer (D) as the cooperation destination is issued. For example, an arrow connecting a device image 70 as a first image associated with a multifunction device (B) as a first device and a device image 76 as a second image associated with a dryer (D) as a second device. Etc. are displayed or the dryer (D) is guided by voice.

  Note that even an image displayed on the device display screen 68 does not necessarily identify a device or an object associated with the image. For example, the dryer (D) may not be identified by the specifying unit 38 of the server 14. Devices and objects that are not identified are excluded from cooperation destination candidates (second devices). An image associated with an unidentified device or object may be displayed on the device display screen 68 or may not be displayed on the device display screen 68.

(Example 2)
With reference to FIG. 16, the guidance processing according to the second embodiment will be described. FIG. 16 shows an example of a device display screen according to the second embodiment. In the second embodiment, a plurality of devices correspond to the second device (cooperation destination device). For example, it is assumed that a multifunction peripheral (B), a projector (C), a dryer (D), and a camera (E) are identified as devices, and a foliage plant (F) is also identified as a foliage plant. Of course, an object that does not correspond to the device may not be identified.

  A device display screen 68 is displayed on the UI unit 46 of the terminal device 16, and the identified devices (multifunction machine (B), projector (C), dryer (D), and camera E) are displayed on the device display screen 68. The device images 70, 72, 76, and 82 associated with each other are displayed, and the image 84 associated with the houseplant (F) is displayed.

  In this case, for example, it is assumed that the multifunction device (B) is selected by the user as the first device, and the device image 70 associated with the multifunction device (B) is designated by the user as the first image. Assume that, for example, a projector (C) and a camera (E) are specified as second devices that can execute the cooperation function together with the multifunction peripheral (B) as the first device. In this case, the guidance of the projector (C) and the camera (E) as the second device is notified. For example, the guidance is notified at the same time. In the example illustrated in FIG. 16, as in the first embodiment, an arrow 80 that connects the device image 70 associated with the multifunction peripheral (B) and the device image 72 associated with the projector (C) is displayed as a guidance notification. In addition, an arrow 86 connecting the device image 70 associated with the multifunction peripheral (B) and the device image 82 associated with the camera (E) is displayed as a guidance notification.

  Note that a priority order may be associated with the cooperation function. Information indicating the priority order is associated with each cooperation function in the cooperation function management information 34 (for example, the cooperation function management table shown in FIG. 7). When performing guidance notification of a plurality of second devices, information indicating the priority order of each cooperation function is transmitted from the server 14 to the terminal device 16, and the priority order is displayed on the device display screen 68. For example, the priority order of the cooperation function 1 that uses the multifunction peripheral (B) and the projector (C) is higher than the priority order of the cooperation function 2 that uses the multifunction peripheral (B) and the camera (E). In this case, the control unit 48 of the terminal device 16 indicates that the priority order of the projector (C) used for the cooperation function 1 is higher than the priority order of the camera (E) used for the cooperation function 2. Is displayed on the device display screen 68. The control unit 48 may display a character string indicating the order on the device display screen 68, may display the arrows 80 and 86 on the device display screen 68 in different colors, or may display the device images 72 and 82. May be displayed on the device display screen 68 in different display forms, or an arrow 80 for the device image 72 with a higher priority is displayed on the device display screen 68, and an arrow 86 for the device image 82 with a lower priority is displayed on the device. The display screen 68 may not be displayed.

  The control unit 48 of the terminal device 16 may display a character string indicating the second device in a specific area in the device display screen 68 without displaying an arrow. For example, the character string is displayed in an area where no device image is displayed. This prevents an arrow from being displayed and makes it difficult to see the information displayed on the screen.

  As described above, when the multifunction device (B) as the first device is designated by the user, the projector (C) and the camera (E) as the second device capable of executing the cooperation function together with the multifunction device (B). ) Is recommended as a candidate for the partner device.

  In addition, when the cooperation destination device candidate (second device) is not displayed on the device display screen 68, the location information indicating the location where the second device is installed or the location where the second device is set Information indicating guidance to the user may be displayed on the UI unit 46 of the terminal device 16. For example, the control unit 36 of the server 14 acquires the position information of the second device by the GPS function or the like, and installs the second device from the location of the terminal device 16 based on the position information and the position information of the terminal device 16. Create information that shows directions to the place. Information indicating the guidance may be transmitted from the server 14 to the terminal device 16 and displayed on the UI unit 46 of the terminal device 16.

  Similar to the first embodiment, the processing according to the second embodiment may be applied when a function image associated with a function is used. For example, when a functional image as the first image is designated by the user, guidance to a plurality of second functions capable of executing the cooperative function together with the first function associated with the functional image may be notified. . Of course, when a device image associated with the first device is designated by the user, notification of guidance to a plurality of functions capable of executing the cooperation function together with the first device may be made.

(Example 3)
With reference to FIGS. 17 to 20, the guidance process according to the third embodiment will be described. 17 to 20 show an example of a device display screen according to the third embodiment. In the third embodiment, after the first device is designated by the user, the second device is designated by the user, and the notification of the guidance of the third device that can execute the cooperation function together with the first device and the second device Is controlled. Further, the device recommended as the third device may be changed according to the designation order of the first device and the second device.

  For example, a PC (A), a multifunction peripheral (B), a projector (C), and a camera (E) are identified as devices. As shown in FIG. 17, the identified device (PC ( It is assumed that device images 70, 72, 82, 88 and an image 84 associated with A), the multifunction device (B), the projector (C), and the camera (E) are displayed.

  In this case, for example, when the device image 70 associated with the multifunction device (B) is designated by the user, the multifunction device (B) is identified as the first device by the specifying unit 38 of the server 14 and the multifunction device (B ), The PC (A), the projector (C), and the camera (E) that can execute the cooperation function are identified as second devices (cooperation destination device candidates). As shown in FIG. 17, for example, arrows 80, 86, and 90 are displayed as the second device guidance notification. An arrow 90 is an image that connects the device image 70 associated with the multifunction peripheral (B) and the device image 88 associated with the PC (A).

  Next, it is assumed that the projector (C) is selected from the second device group by the user as the cooperation destination device, and the device image 72 associated with the projector (C) is designated by the user. In this case, the specifying unit 38 of the server 14 can execute the link function together with the multifunction peripheral (B) as the first device and the projector (C) as the second device by referring to the link function management information 34. Identify possible third devices. In the cooperation function management table shown in FIG. 7, cooperation functions that can be executed by linking two devices are registered, but of course, they can be executed by linking three or more devices. Various linkage functions may be registered. For example, it is assumed that PC (A) is specified as the third device. In this case, as shown in FIG. 18, the guidance to the PC (A) as the third device is notified. For example, an arrow 92 connecting the device image 72 associated with the projector (C) as the second device and the device image 88 associated with the PC (A) as the third device is displayed as the guidance notification.

  For example, the user designates the device image 72 associated with the projector (C), performs an operation of connecting the device image 70 associated with the multifunction device (B) and the device image 72 associated with the projector (C), By superimposing the device image 70 on the device image 72 or tracing the space between the device image 70 and the device image 72 from the device image 70 toward the device image 72, the projector (C) used as the second device can be used. It is specified.

  The order of designation of each device (each device image) corresponds to the order of functions used in each device and the movement order of data that moves between each linked device. The operation of connecting images and the operation of overlapping images) also serve as operations for specifying the order of functions and the order of data movement. In the example shown in FIG. 18, the multifunction device (B) as the first device is used first, and the projector (C) as the second device is used second. The cooperation function can be executed together with the first device and the second device, and the third device used in the cooperation function is guided as the third device. That is, in the cooperation function in which the multifunction device (B) is used first and the projector (C) is used second, the device used third is guided as the third device. In the example shown in FIG. 18, the third device is PC (A). In the third embodiment, the cooperation function management information 34 includes a cooperation function that can be executed by cooperation of a plurality of devices, including the order of use of the devices. The third device is specified by referring to the function management information 34.

  FIG. 19 shows another example. As in FIG. 17, device images 70, 72, 82, 88 and an image 84 are displayed on the device display screen 68.

  In this case, for example, when the device image 72 associated with the projector (C) is designated by the user, the specifying unit 38 of the server 14 identifies the projector (C) as the first device and cooperates with the projector (C). A PC (A), a multifunction peripheral (B), and a camera (E) capable of executing the function are identified as second devices (cooperation destination device candidates). As shown in FIG. 19, for example, arrows 94, 96, and 98 are displayed as the second device guidance notification. An arrow 94 is an image that connects the device image 72 associated with the projector (C) as the first device and the device image 70 associated with the multifunction device (B) as the second device. An arrow 96 is an image connecting the device image 72 and the device image 82 associated with the camera (E) as the second device. An arrow 98 is an image that connects the device image 72 and the device image 88 associated with the dryer (A) as the second device.

  Next, it is assumed that the multifunction device (B) is selected by the user from the second device group as the cooperation destination device, and the device image 70 associated with the multifunction device (B) is designated by the user. In this case, the specifying unit 38 of the server 14 can execute the cooperation function together with the projector (C) as the first device and the multifunction peripheral (B) as the second device by referring to the cooperation function management information 34. Identify possible third devices. For example, it is assumed that the camera (E) is specified as the third device. In this case, as shown in FIG. 20, the guidance of the camera (E) as the third device is notified. For example, an arrow 100 connecting the device image 70 associated with the multifunction device (B) as the second device and the device image 82 associated with the camera (E) as the third device is displayed as a guidance notification. In this case, in the cooperation function in which the projector (C) is used first and the multifunction peripheral (B) is used second, the device (for example, camera (E)) used third is guided as the third device. The

  For example, the user designates the device image 70 associated with the multifunction device (B), or performs an operation of connecting the device image 72 associated with the projector (C) and the device image 70 associated with the multifunction device (B). The device image 72 is superimposed on the device image 70, or the device image 72 and the device image 70 are traced from the device image 72 toward the device image 70, thereby being used as a second device (B ) Is specified.

  As mentioned above, the guidance of the 3rd apparatus which can perform a cooperation function with the 1st apparatus and the 2nd apparatus is notified. In addition, the device that is guided (recommended) as the third device changes according to the designation order of the first device and the second device. The order in which devices are specified also serves as the order of functions used by each device and the movement order of data that moves between each linked device. The operation to specify a device depends on the order of functions or the order of data movement. It also serves as an operation to specify. In the linkage function, the next device to be used and the device to which the data is moved change depending on the designation order of the devices. In the third embodiment, in accordance with the change, the guidance of the third device used in the linkage function is provided. Will be notified.

  Similar to the first embodiment, the process according to the third embodiment may be applied when a function image associated with a function is used. For example, when a function image associated with the first function is designated by the user, guidance to the function image associated with the second function capable of executing the cooperation function together with the first function is notified. In addition, when a function image associated with the second function is designated by the user, a guidance to the function image associated with the third function capable of executing the linkage function together with the first function and the second function is notified. Also good. In this case, the function guided as the third function changes according to the designation order of the first function and the second function. In the third embodiment, the linkage function may be a function that uses a function of a device as hardware and a function realized by software.

Example 4
With reference to FIGS. 21 and 22, the guidance process according to the fourth embodiment will be described. 21 and 22 show an example of a device display screen according to the fourth embodiment. In the fourth embodiment, when a device that cannot execute the cooperation function together with the first device is designated, the guidance of the second device (cooperation destination device) that can execute the cooperation function together with the first device is notified. For example, it is assumed that the multifunction peripheral (B), the projector (C), and the dryer (D) are identified as devices.

  As shown in FIG. 21, a device display screen 68 is displayed on the UI unit 46 of the terminal device 16. Device images 70, 72, and 76 associated with (D)) are displayed.

  In this case, for example, when the multifunction device (B) is selected by the user as the first device and the device image 70 associated with the multifunction device (B) is designated by the user as the first image, the specifying unit 38 of the server 14 is selected. Thus, the multifunction device (B) is identified as the first device. For example, the projector (C) corresponds to the second device capable of executing the cooperation function together with the multifunction device (B) as the first device, and the dryer (D) has the cooperation function together with the multifunction device (B). It shall correspond to equipment that cannot be executed. In this case, for example, the user designates the device image 76 associated with the dryer (D) or connects the device image 70 associated with the multifunction device (B) and the device image 76 associated with the dryer (D). Function, the device image 70 is superimposed on the device image 76, and the device image 70 and the device image 76 are traced from the device image 70 toward the device image 76, thereby being linked together with the multifunction peripheral (B). It is assumed that a dryer (D) that cannot execute the above is designated as a cooperation destination device. In the example illustrated in FIG. 21, as indicated by an arrow 102, an operation for connecting the device image 70 and the device image 76 is performed by the user.

  If a dryer (D) that cannot execute the linkage function together with the multifunction peripheral (B) as the first device is designated by the user as a linkage destination device, the control unit 36 of the server 14 accepts the designation, Control of notification of guidance of the projector (C) as the second device capable of executing the cooperation function together with the machine (B). Thereby, notification for guiding the projector (C) as the second device is made. For example, as illustrated in FIG. 22, the control unit 48 of the terminal device 16 displays an arrow 104 for guiding the projector (C) as a cooperation destination device on the device display screen 68. The arrow 104 is, for example, an image that connects a device image 70 associated with the multifunction device (B) as the first device and a device image 72 associated with the projector (C) as the second device. Of course, the notification of the guidance may be performed by displaying a voice or a character string.

  As described above, in the fourth embodiment, when a device that cannot execute the cooperation function together with the first device is designated, the notification of the guidance of the second device that can execute the cooperation function together with the first device. Is controlled. When the first device is designated and the arrow for guidance is always displayed, the arrow on the screen may be complicated. According to the fourth embodiment, it is possible to avoid the complexity.

  Similar to the first embodiment, the processing according to the fourth embodiment may be applied when a device image associated with a function is used. For example, when a function that cannot execute the cooperation function together with the first function is designated, the guidance of the second function that can execute the cooperation function together with the first function is notified.

(Example 5)
Example 5 will be described with reference to FIGS. 23 to 25 show examples of screens according to the fifth embodiment. In the fifth embodiment, when the device designated by the user as the cooperation destination device cannot execute the cooperation function together with the first device due to a failure or being used, the cooperation function is executed together with the first device. You will be notified of guidance for another device that can. In this case, guidance for a device of the same type (for example, a device having the same type of function) as the device specified by the user as the cooperation destination device may be preferentially notified. The control unit 36 of the server 14 acquires information indicating the operation status of each device (for example, during execution of a process, during failure, during maintenance, etc.) from each device, and manages the operation status of each device.

  For example, it is assumed that the multifunction peripheral (B) and the projectors (C) and (F) are identified as devices. As shown in FIG. 23, a device display screen 68 is displayed on the UI unit 46 of the terminal device 16. Device images 70, 72, 106 associated with F)) are displayed.

  In this case, for example, when the multifunction device (B) is selected by the user as the first device and the device image 70 associated with the multifunction device (B) is designated by the user as the first image, the specifying unit 38 of the server 14 is selected. Thus, the multifunction device (B) is identified as the first device. For example, it is assumed that the projectors (C) and (F) correspond to the second device that can execute the cooperation function together with the multifunction peripheral (B) as the first device. For example, the user designates the device image 106 associated with the projector (F), or performs an operation of connecting the device image 70 associated with the multifunction device (B) and the device image 106 associated with the projector (F), By superimposing the device image 70 on the device image 106 or tracing between the device image 70 and the device image 106 from the device image 70 toward the device image 106, the projector (F) is designated as the cooperation destination device. And In the example shown in FIG. 23, as indicated by an arrow 108, an operation for connecting the device image 70 and the device image 106 is performed by the user.

  When the projector (F) is designated as a cooperation destination device by the user, the control unit 36 of the server 14 accepts the designation and confirms the operating status of the projector (F). For example, when the projector (F) is out of order or in use, the control unit 36 of the server 14 is a device other than the projector (F) that is capable of executing the linkage function together with the first device. Control notifications for guidance. The control unit 36 may preferentially notify the guidance of a device of the same type as the projector (F) (for example, a device having the same type of function as the projector (F)). For example, when the projector (C) corresponds to the same type of device as the projector (F), notification for guiding the projector (C) as the second device is preferentially performed. In this case, for example, as shown in FIG. 24, the control unit 48 of the terminal device 16 displays an arrow 110 for guiding the projector (F) as a cooperation destination device on the device display screen 68. The arrow 110 is, for example, an image that connects a device image 70 associated with the multifunction device (B) as the first device and a device image 72 associated with the projector (C) as the second device.

  In addition, when a device that is out of order or in use is designated by the user as a cooperation destination device, for example, as shown in FIG. 25, the UI unit 46 of the terminal device 16 is under the control of the control unit 36 of the server 14. A screen 112 showing a message indicating the reason why cooperation is not possible may be displayed.

  Note that when the device that has been out of order becomes available due to repair, or when the device that has been executing the process has completed its processing and is not executing the process, the control unit 36 of the server 14 The device is identified as a device capable of executing the linkage function together with the first device.

  According to the fifth embodiment, devices that are not in failure and devices that are not in use are guided, so that convenience for the user is improved. Further, since the same type of device as the device specified by the user is guided, the device assumed to be used by the user is guided.

(Example 6)
With reference to FIG. 26, the guidance process according to the sixth embodiment will be described. FIG. 26 shows an example of the device selection screen. In the sixth embodiment, a candidate list indicating information on the second device that can execute the cooperation function together with the first device is displayed on the UI unit 46 of the terminal device 16.

  For example, the multifunction device (B) and the dryer (D) are identified as devices, and as shown in FIG. 14A, the device image 70 associated with the multifunction device (B) and the device associated with the dryer (D). It is assumed that the image 76 is displayed on the UI unit 46 of the terminal device 16. In this case, for example, when the multifunction device (B) is selected by the user as the first device and the device image 70 associated with the multifunction device (B) is designated by the user, the identifying unit 38 of the server 14 causes the multifunction device to (B) is identified as the first device. For example, it is assumed that the dryer (D) corresponds to a device that cannot execute the cooperation function together with the multifunction peripheral (B). In this case, for example, the user designates the device image 76 associated with the dryer (D) or connects the device image 70 associated with the multifunction device (B) and the device image 76 associated with the dryer (D). The device image 70 is superimposed on the device image 76, or the device image 70 and the device image 76 are traced from the device image 70 toward the device image 76, thereby allowing the dryer (D) to be linked. Suppose that it is specified as a device.

  If a dryer (D) that cannot execute the linkage function together with the multifunction peripheral (B) as the first device is designated by the user as a linkage destination device, the control unit 36 of the server 14 accepts the designation, The display of a candidate list indicating information related to the second device is controlled as a notification of the guidance of the second device that can execute the cooperation function together with the machine (B). As a result, the device selection screen 114 is displayed on the UI unit 46 of the terminal device 16 as shown in FIG. 26, and the candidate list is displayed in the device selection screen 114. Note that the message screen 78 shown in FIG. 14B may be displayed on the UI unit 46 of the terminal device 16 before the device selection screen 114 is displayed.

  As shown in FIG. 26, the candidate list includes names of devices that can execute the linkage function together with the multifunction peripheral (B), device images associated with the devices, and examples of linkage functions (for example, names of linkage functions). ), Etc. Of course, at least one of these pieces of information may be included in the candidate list. The device image may be an image representing the appearance of an actual device (an image associated with the device in a one-to-one relationship), or may be an image imitating the device (for example, an icon). The image representing the actual appearance of the device is, for example, an image generated by photographing the appearance of the device, and is an image representing the device itself. An image simulating a device corresponds to an image indicating the type of device. As an example of the cooperation function, the names of one or more cooperation functions are included in the candidate list. When the names of a plurality of linkage functions are displayed, the names of the linkage functions may be displayed in a display order corresponding to the designation order of a plurality of devices to be linked. For example, when a multifunction device and a projector can execute a plurality of cooperation functions (for example, cooperation functions A and B), the multifunction device is designated as the first device, and the projector is designated as the first device. The display order of the plurality of linkage functions included in the candidate list may be different. For example, when the multifunction device is designated as the first device, the names of the cooperation functions are displayed in the order of the cooperation functions A and B, and when the projector is designated as the first device, the names of the cooperation functions B and A are displayed. The name of each cooperation function may be displayed.

  The order of arrangement of the second devices in the candidate list may be determined based on, for example, past usage records of the second devices. For example, the control unit 36 of the server 14 manages the use record of each device by acquiring information indicating the use result of each device from each device. For example, the control unit 36 displays the second device with the higher usage frequency (for example, the second device with the higher usage frequency) at the top in the candidate list. The past use record may be a use record of a user who has designated the first image (for example, a user who uses the terminal device 16 or a user who has logged into the server 14), and includes the use record of other users. It may be a usage record.

  As another example, the control unit 36 may display the second device having the larger number of executable cooperation functions in the higher order in the candidate list. For example, when the number of cooperation functions that the projector can execute with the multifunction peripheral (B) is “3” and the number of cooperation functions that the personal computer can execute with the multifunction peripheral (B) is “2”, The projector is displayed above the personal computer.

  When the function of each device is updated and the cooperation function management information 34 is updated, the control unit 36 updates the display of the second device in the candidate list along with the update. For example, if a device that could not execute the cooperation function with the multifunction peripheral before the update becomes capable of executing the cooperation function together with the multifunction peripheral after the update, the control unit 36 identifies the device as the first device. Two devices are displayed in the candidate list. In addition, the usage record of each device is also updated with time, and the control unit 36 updates the display order of each second device in the candidate list based on the updated usage record.

  Further, the control unit 36 of the server 14 may update the second device included in the candidate list according to the operating status and the peripheral status of the first device and the second device, or the second device in the candidate list. The display order may be updated.

  Even if the device is capable of executing the linkage function together with the multifunction device, the device that is out of order or in use may not be displayed in the candidate list. Even in this case, when the failure of the device is resolved or the device is not in use, the device is displayed in the candidate list.

  When the user designates a device name or device image included in the candidate list, the control unit 48 of the terminal device 16 controls the device designated as the first multifunction device under the control of the control unit 36 of the server 14. A list of cooperative functions that can be executed may be displayed on the UI unit 46 of the terminal device 16. For example, when a projector in the candidate list is designated, the control unit 48 of the terminal device 16 causes the UI unit 46 to display a list of cooperation functions that can be executed by the multifunction peripheral and the projector. In addition, when the user designates a cooperation function included in the candidate list, control for executing the designated cooperation function is performed.

  The second device displayed in the candidate list may be included in a device group registered in advance in the server 14, or may be a device included in a device group identified by the AR technology or the like. In addition, a device included in the device group displayed on the UI unit 46 of the terminal device 16 or a device included in the device group displayed in a specific area in the screen of the UI unit 46 may be used. Good. For example, when a user operates a device image associated with a device, the device image is displayed in a specific area. The first device may also be a device included in these device groups. The same applies to the first to fifth embodiments described above and the embodiments described below.

  Similar to the first embodiment, the process according to the sixth embodiment may be applied when a device image with a string is used for a function. For example, when a device or function that cannot execute the cooperation function together with the first function is specified, a device (second device candidate) or function (second function) that can execute the cooperation function together with the first function. A candidate list indicating the candidates) may be displayed. For example, the function list of the second function, the device image associated with the second function, an example of the cooperation function, and the like are included in the candidate list. In addition, a candidate list may be displayed when a function that cannot execute the cooperation function with the first device is designated.

  According to the sixth embodiment, the second device candidates and the second function candidates are displayed as the candidate list, which is convenient for the user.

(Example 7)
Example 7 will be described with reference to FIGS. In the seventh embodiment, similarly to the sixth embodiment, the candidate list is displayed on the UI unit 46 of the terminal device 16.

  Similarly to the sixth embodiment, it is assumed that the multifunction device (B) is designated by the user as the first device, and the device that cannot execute the cooperation function together with the multifunction device (B) is designated by the user. The control unit 36 of the server 14 receives the designation and controls the display of a candidate list indicating information on the second device as a notification of the guidance of the second device that can execute the cooperation function together with the multifunction peripheral (B). . As a result, as shown in FIG. 26, the device selection screen 114 is displayed on the UI unit 46 of the terminal device 16.

  For example, when the user designates a device name, the control unit 48 of the terminal device 16 creates a list of devices of the same type as the designated device (list of second devices) under the control of the control unit 36 of the server 14. It is displayed on the UI unit 46 of the terminal device 16. For example, when the user designates a projector, a device selection screen 116 is displayed on the UI unit 46 of the terminal device 16 as shown in FIG. On the device selection screen 116, a list of projectors as second devices is displayed. For example, when the specifying unit 38 of the server 14 specifies (identifies) the projectors aaa, bbb, and ccc as projectors corresponding to the second device, a list indicating them is displayed. The user selects a projector to be used as the second device from this list.

  In addition, the device selection screen 116 displays a message asking the user to add a device to be linked. For example, after the second device is designated by the user, when the user instructs to add a device to be linked (for example, when “Yes” shown in FIG. 27 is designated by the user), FIG. As illustrated, a device selection screen 118 is displayed on the UI unit 46 of the terminal device 16, and the device selection screen 118 includes a third device that can execute a linkage function together with the first device and the second device. A candidate list showing information about is displayed. This candidate list has the same configuration as the candidate list indicating information on the second device. Note that the display order of the device as the third device may be different from the candidate list for the second device. When the third device is designated by the user, a list of third devices is displayed in the same manner as the device selection screen 116 (see FIG. 27) displayed when the second device is designated. The same applies when adding devices after the fourth device.

  When an image imitating a device (for example, an icon) is included in the candidate list as a device image and the device image is designated by the user, as shown in FIG. 27, the second device associated with the device image. A list of (for example, projectors) is displayed. That is, since the device image represents the type of device and represents the device as a representative, when the device image is designated, a list of second devices associated with the device image is displayed. On the other hand, since an image representing an actual device (an image associated with the device in a one-to-one relationship) is an image representing the device itself, when the device image is included in the candidate list and designated by the user, The device itself is designated by the user. In this case, only the message for inquiring the user to add a device to be linked may be displayed without displaying the list of the second devices shown in FIG.

  When the device to be linked is not added (for example, when “No” shown in FIG. 27 is designated by the user), the control unit 48 of the terminal device 16 controls the control unit 36 of the server 14. 29, the function selection screen 120 is displayed on the UI unit 46 of the terminal device 16 as shown in FIG. The function selection screen 120 displays a list of cooperative functions that can be executed by a plurality of devices specified by the user. For example, when the multifunction device (B) is designated as the first device and the projector aaa is designated as the second device, a list of cooperative functions that can be executed by the multifunction device (B) and the projector aaa is displayed. When the user designates a cooperation function from the list and gives an execution instruction, the multifunction machine (B) and the projector aaa execute the cooperation function.

  On the other hand, when executing a linkage function that is not included in the list of linkage functions, the user requests execution of the linkage function. For example, the UI unit 46 of the terminal device 16 displays a screen 122 for making the request, as shown in FIG. 30, and the user inputs the name of the cooperation function desired to be executed on the screen 122. To do. Information indicating the request is transmitted from the terminal device 16 to the server 14. Upon receiving the request, the control unit 36 of the server 14 determines whether or not the designated device group (for example, the first device and the second device) can execute the cooperation function related to the request. . When the cooperation function related to the request cannot be executed, the control unit 48 of the terminal device 16 cannot execute the cooperation function related to the request as shown in FIG. 31 under the control of the control unit 36 of the server 14. Is displayed on the UI unit 46 of the terminal device 16. When the cooperation function related to the request can be executed, the control unit 48 of the terminal device 16 indicates that the cooperation function related to the request is executed as shown in FIG. 32 under the control of the control unit 36 of the server 14. The message is displayed on the UI unit 46 of the terminal device 16. When the execution instruction is given by the user, the linkage function is executed. Moreover, the cooperation function requested by the user may be registered. For example, as shown in FIG. 32, the control unit 48 of the terminal device 16 inquires of the user whether or not to register the requested cooperation function as a future cooperation function candidate under the control of the control unit 36 of the server 14. Is displayed on the UI unit 46 of the terminal device 16. When “registration” is instructed by the user, the requested information on the cooperation function is registered in the cooperation function management information 34 and will be displayed in the list of cooperation functions in the future. When the user instructs “not register”, the requested information regarding the cooperation function is not registered and is not included in the list of cooperation functions. For example, when the requested cooperation function falls under exceptional processing, when the other cooperation function is used more frequently, or when the number of cooperation functions included in the list increases and becomes complicated. In case of avoidance, it is assumed that the requested cooperation function is not registered.

  Similar to the first embodiment, the processing according to the seventh embodiment may be applied when a device image with a string is used for a function.

  According to the seventh embodiment, the second device candidates and the second function candidates are displayed as the candidate list, which is convenient for the user. In addition, it is possible to easily increase the devices to be linked using the candidate list.

(Example 8)
Example 8 will be described with reference to FIGS. 33 to 35. In the eighth embodiment, as in the sixth and seventh embodiments, the candidate list is displayed on the UI unit 46 of the terminal device 16.

  As in the sixth and seventh embodiments, it is assumed that the multifunction device (B) is designated by the user as the first device, and the device that cannot execute the cooperation function together with the multifunction device (B) is designated by the user. In this case, as shown in FIG. 26, the device selection screen 114 is displayed on the UI unit 46 of the terminal device 16.

  When a cooperation function (for example, “print display screen”) is designated by the user on the device selection screen 114, the control unit 48 of the terminal device 16 is controlled by the control unit 36 of the server 14 as shown in FIG. Then, the device selection screen 124 is displayed on the UI unit 46 of the terminal device 16. In the device selection screen 124, there is a list of devices (second device list) necessary (executable for the link function) necessary to execute a cooperation function (for example, “print display screen”) designated by the user. Is displayed. When a device (second device) is designated by the user from the list and an execution instruction for the cooperation function is given, the cooperation function designated by the user is specified by the first device and the second device designated by the user. Executed.

  When the user instructs the selection of another device not included in the device list displayed on the device selection screen 124 (“Yes” is selected in response to an inquiry “Do you want to make another device?” ), The control unit 48 of the terminal device 16 displays the device selection screen 126 on the UI unit 46 of the terminal device 16 as shown in FIG. 34 under the control of the control unit 36 of the server 14. The device selection screen 126 displays a list of other device candidates that can execute the cooperation function designated by the user. In the device selection screen 126, when a device (second device) is designated by the user and an execution instruction for the cooperation function is given, the cooperation function designated by the user by the first device and the second device designated by the user. Is executed.

  When the user instructs the selection of another device not included in the device list displayed on the device selection screen 126 (when “Yes” is selected in response to an inquiry “other than the above?”) ), The control unit 48 of the terminal device 16 displays a screen 128 for inputting information related to the cooperation target device under the control of the control unit 36 of the server 14 as shown in FIG. 46 is displayed. On this screen 128, the user inputs information related to the device to be linked (for example, the name and type of the device). Information relating to the cooperation target device input by the user is transmitted from the terminal device 16 to the server 14. When the control unit 36 of the server 14 receives the information, the control unit 36 determines whether the first device and the cooperation target device designated by the user can execute the cooperation function designated by the user. to decide. When the cooperation function cannot be executed, the control unit 48 of the terminal device 16 displays a screen for resetting another device as a device to be linked under the control of the control unit 36 of the server 14. 16 UI parts 46 are displayed. When the cooperation function can be executed, when the execution instruction of the cooperation function is given, the cooperation function designated by the user is executed by the first device and the second device (device to be linked) designated by the user. Is done.

  Similar to the first embodiment, the processing according to the eighth embodiment may be applied when a device image with a string is used for a function.

  According to the eighth embodiment, when a cooperation function is specified in the candidate list, devices necessary for executing the cooperation function are displayed, so that convenience for the user when selecting a device is improved.

  The present embodiment may be applied to an environment where a plurality of users use a plurality of devices. For example, even when a user interface such as a touch panel is removed from the device, the terminal device 16 is used as the user interface. As another scene, for example, when a device is temporarily used at a user's destination, etc., a user interface suitable for the user, that is, a function of the device specified by the user and cooperation using the device A user interface on which the function is displayed is realized by the terminal device 16.

  Hereinafter, processes related to the first to eighth embodiments will be described.

(Switching display of information related to linkage function)
In the present embodiment, the display of information related to the cooperation function may be switched according to the order in which the device images associated with the devices are connected. In this case, when the device designated as the device to be linked cannot execute the linkage function together with the first device, the linkage function is executed together with the first device as in the first to eighth embodiments. Notification of guidance of the second device that can be performed is controlled. On the other hand, when the second device capable of executing the cooperation function together with the first device is designated by the user, the display of information regarding the cooperation function is switched according to the order in which the device images are connected. Hereinafter, this processing will be described in detail with reference to FIGS.

  FIG. 36 shows a cooperative function management table as another example of the cooperative function management information 34. In this linkage function management table, as an example, information indicating a combination of device IDs, information indicating a device name to be linked (for example, the type of each device), information indicating a linkage function (linkage function information), and connection Information indicating the order is associated with information indicating the priority order. The connection order corresponds to the order of connecting the device images associated with the devices. The priority order is a display priority order of information related to the linkage function. For example, a device with a device ID “A” is a PC (personal computer), and a device with a device ID “B” is a multifunction device. By linking the PC (A) and the multifunction peripheral (B), for example, a “scan transfer function” and a “print function” are realized as linked functions. The “scan transfer function” is a function for transferring image data generated by scanning by the multifunction peripheral (B) to the PC (A). The “print function” is a function for transmitting data (for example, image data and document data) stored in the PC (A) to the multifunction device (B) and printing the data on the multifunction device (B). For example, when a device is connected from the multifunction device (B) to the PC (A), that is, when a device image is connected from a device image associated with the multifunction device (B) to a device image associated with the PC (A). The priority of the “scan transfer function” is “1st”, and the priority of the “printing function” is “2nd”. In this case, information related to the “scan transfer function” is displayed with priority over information related to the “print function”. Conversely, when a device is connected from the PC (A) to the multifunction device (B), that is, an image is connected from the device image associated with the PC (A) to the device image associated with the multifunction device (B). In this case, the priority of the “print function” is “1st”, and the priority of the “scan transfer function” is “2nd”. In this case, information related to the “print function” is displayed with priority over information related to the “scan transfer function”.

  37 and 38 show an example of a screen displayed on the UI unit 46 of the terminal device 16. For example, it is assumed that the multifunction peripheral (B) and the PC (A) are identified. As shown in FIG. 37A, a device display screen 68 is displayed on the UI unit 46 of the terminal device 16, and the device image 70 and the PC (PC) associated with the multifunction device (B) are displayed on the device display screen 68. A device image 88 associated with A) is displayed. In this state, the user connects device images representing devices to be linked using an indicator (for example, a user's finger, pen, stylus, etc.). The control unit 48 of the terminal device 16 detects the contact of the indicator on the device display screen 68 and detects the movement of the indicator on the device display screen 68. For example, as indicated by an arrow 130, the user touches the device image 70 on the device display screen 68 and operates to the device image 88 using the operation device (for example, traces the operation device on the device display screen 68). Thus, the device image 70 and the device image 88 are connected. As a result, the MFP (B) associated with the device image 70 and the PC (A) associated with the device image 88 are designated as devices to be linked and the order of connection is designated. The order of connecting the device images corresponds to the order of connection. Note that the multifunction device (B) corresponds to the first device, and the PC (A) corresponds to the second device. In the example shown in FIG. 37A, since the image is connected from the device image 70 to the device image 88, the device is connected from the multifunction device (B) to the PC (A). Information indicating the connection order of devices is transmitted from the terminal device 16 to the server 14. Note that the control unit 48 of the terminal device 16 may display an image representing a locus traced by the user on the device display screen 68. After the devices are connected to each other, the locus is set in advance as a straight line or the like. May be displayed on the device display screen 68.

  As described above, when the devices to be linked (for example, the multifunction device (B) and the PC (A)) are specified, the specifying unit 38 of the server 14 uses the PC in the linkage function management table shown in FIG. The cooperation function associated with the combination of (A) and the multifunction peripheral (B) is specified. Thereby, the cooperation function executed by linking the PC (A) and the multifunction peripheral (B) is specified. When the device connection order is specified by the user, the specifying unit 38 specifies the priority order associated with the connection order in the cooperation function management table. A specific example will be described with reference to FIG. 36. Since the PC (A) and the multifunction peripheral (B) are designated as devices to be linked, the linkage function executed by them is “scan transfer function”. “Print function”. In addition, since a device is connected from the MFP (B) to the PC (A) (B → A), the priority of the “scan transfer function” is “first”, and the priority of the “print function” is “2nd place”.

  Information relating to the cooperation function identified as described above and information indicating the priority order are transmitted from the server 14 to the terminal device 16. The control unit 48 of the terminal device 16 causes the UI unit 46 to display information related to the cooperative function as information related to the cooperative function candidate according to the priority order.

  For example, as illustrated in FIG. 37B, the control unit 48 of the terminal device 16 displays the cooperation function display screen 132 on the UI unit 46 and displays information on the cooperation function candidate on the cooperation function display screen 132. Since the priority of the “scan transfer function” is “first” and the priority of the “print function” is “second”, the information related to the “scan transfer function” has priority over the information related to the “print function”. (For example, at the top). For example, as the information related to the “scan transfer function”, an explanatory text of “scan transfer function” “transfer data scanned by the MFP (B) to the PC (A)” is displayed. In addition, as the information related to the “print function”, an explanatory text of “print function” “print data in PC (A)” is displayed.

  When the cooperation function is designated by the user and an execution instruction is given, the designated cooperation function is executed. For example, when a “YES” button is pressed by the user, a cooperation function associated with the “YES” button is executed. In addition, a “return” button is displayed on the cooperation function display screen 132, and when the user presses the “return” button, the device connection processing is stopped.

  It should be noted that the specifying process of the cooperation function and the specifying process of the priority order may be performed by the terminal device 16.

  As an operation other than tracing between device images with an operator, a device to be linked may be specified by a drawing operation such as adding a circle, and a connection order may be specified. For example, the order of drawing operations corresponds to the connection order. As another example, a device to be linked and a connection order may be specified in accordance with a user's voice instruction.

  FIG. 38 shows another example of the operation. For example, as shown in FIG. 38A, the user touches the device image 88 on the device display screen 68 using the operator, and operates up to the device image 70 in the direction indicated by the arrow 134, thereby causing the device image 88. And the device image 70 are connected. As a result, the PC (A) associated with the device image 88 and the multifunction peripheral (B) associated with the device image 70 are designated as devices to be linked and the order of connection is designated. In this example, since the image is connected from the device image 88 to the device image 70, the device is connected from the PC (A) to the multifunction device (B). Referring to the cooperation function management table shown in FIG. 36, the priority of “print function” is “1st”, and the priority of “scan transfer function” is “2nd”. In this case, as shown in FIG. 38B, information related to the “printing function” is displayed on the cooperation function display screen 136 with priority (for example, higher) than information related to the “scan transfer function”.

  As described above, a linkage function that uses a function of a device is specified by connecting device images associated with the device. In addition, the display order of the information related to the cooperation function is changed according to the order of connecting the images, that is, the order of connecting the devices. The connection order of devices also serves as the order of functions used by each device and the movement order of data that moves between linked devices. The operation of connecting devices (that is, the operation of connecting images) is the order of functions. It also serves as an operation to specify the order of data movement. Therefore, by changing the display order of the information related to the cooperation function according to the connection order, the information related to the cooperation function predicted to be used by the user is preferentially displayed. That is, information related to a cooperative function that is highly likely to be used by the user is preferentially displayed. For example, when an image is connected from the multifunction device (B) to the PC (A), “the function of the multifunction device (B) is used before the PC (A), and the multifunction device (B) to the PC (A). It is anticipated that the user will use the link function "transfer data to". Further, when an image is connected from the PC (A) to the multifunction device (B), “the function of the PC (A) is used before the multifunction device (B), and the multifunction device (B) from the PC (A). It is anticipated that the user will use the link function "transfer data to". Therefore, by changing the display order of the information related to the cooperative function according to the order of connecting the images, information related to the cooperative function that is highly likely to be used by the user is preferentially displayed. In addition, the order of the functions to be used and the data movement order are specified without performing any special operation other than the operation for connecting the device images, and information on the cooperation function that the user is expected to use is displayed. .

  The display switching process described above may be applied when a function image associated with a function is used. For example, the display of information related to the cooperation function is switched according to the designation order of the device image associated with the first function and the device image associated with the second function.

  In addition, the display switching process described above may be applied to information related to the cooperation function displayed in the candidate list (see, for example, FIG. 26) according to the sixth embodiment. That is, when the multifunction device (B) is designated as the first device, the PC (A) is included in the candidate list and displayed as the second device candidate (cooperation destination candidate), and is associated with the PC (A). As information related to the cooperative function (information related to the cooperative function that can be executed by the MFP (B) and the PC (A)), for example, information related to a plurality of cooperative functions is displayed in the order shown in FIG. . On the other hand, when the PC (A) is designated as the first device, the multifunction device (B) is displayed as a candidate for the second device (cooperation destination candidate) included in the candidate list, and is linked to the multifunction device (B). For example, information regarding a plurality of cooperation functions is displayed in the order shown in FIG.

(Cooperative processing using partial images)
Depending on the position in the device image associated with the device, the function assigned to the device assigned to the cooperation function may be different. When a specific position is designated by the user in the device image, information related to a cooperation function that uses a function corresponding to the specific position is preferentially displayed. Hereinafter, this process will be described in detail.

  FIG. 39 shows an example of a device function management table. Data of the device function management table is stored in the server 14 as device function management information 32. In this device function management table, as an example, information indicating a device ID, a device name (for example, a device type), information indicating a position in a device image, and information indicating a function corresponding to the position in the device image ( Function information) and an image ID are associated with each other. The position in the device image is a specific position (specific part) in the device image associated with the device, for example, a specific position in the device image schematically representing the device, or in the device image captured by the camera Is a specific position. Different functions are associated with each specific position in the device image.

  FIG. 40 shows an example of a screen displayed on the UI unit 46 of the terminal device 16. For example, it is assumed that the multifunction peripheral (B) and the PC (A) are identified. As shown in FIG. 40A, a device display screen 68 is displayed on the UI unit 46 of the terminal device 16, and device images 70 and 88 are displayed on the device display screen 68. For example, a “print function” is assigned to a specific position (partial image 70 a) corresponding to the main body portion of the multifunction peripheral (B) in the device image 70. In the device image 70, a “scan function” is assigned to a specific position (partial image 70b) corresponding to the document cover, document glass, or automatic document feeder of the multifunction machine (B). A “stapling function” is assigned to a specific position (partial image 70 c) corresponding to the post-processing device in the device image 70. The “stapling function” is a function for stopping the output paper by stapling. In addition, a “data storage function” is assigned to a specific position (partial image 88 a) corresponding to the main part of the PC (A) in the device image 88. A “screen display function” is assigned to a specific position (partial image 88 b) corresponding to the display unit of PC (A) in the device image 88. The “data storage function” is a function for storing data sent from another device on the PC (A). The “screen display function” is a function for displaying data sent from another device on the PC (A).

  Note that the control unit 48 of the terminal device 16 may cause the device display screen 68 to display the name of a function (for example, printing or scanning) assigned to a specific position in the device image. As a result, the user is provided with easy-to-understand information about what function corresponds to a specific position. Of course, the name of the function may not be displayed.

  When the position to which the function is assigned in the device image is designated by the user, the function assigned to the designated position is designated as a function to be linked. The user uses an indicator to connect a specific position (partial image) to which a function is assigned in the device image representing the device to be linked. For example, as indicated by an arrow 138, the user touches the partial image 70b on the device display screen 68 and operates to the partial image 88b using the operator, thereby connecting the partial image 70b and the partial image 88b. As a result, the MFP (B) associated with the device image 70 including the partial image 70b and the PC (A) associated with the device image 88 including the partial image 88b are designated as cooperation target devices, and the partial image 70b. The “scan function” assigned to ”and the“ screen display function ”assigned to the partial image 88b are designated. Moreover, the order of connection may be designated by this connection operation. In this case, the order of connecting the partial images corresponds to the order of connection. In the example shown in FIG. 40A, since the images are connected from the partial image 70b to the partial image 88b, a device is connected from the multifunction peripheral (B) to the PC (A). In addition, “scan function” and “screen display function” are designated as functions used for the linkage function. Information indicating the connection order of the devices and information indicating a specific position designated by the user in the device image are transmitted from the terminal device 16 to the server 14.

  When the devices to be linked (for example, the PC (A) and the multifunction device (B)) are identified, the specifying unit 38 of the server 14 is connected to the PC (A) in the linked function management table shown in FIG. A linkage function realized by linking the multifunction peripheral (B) is specified. Further, the specifying unit 38 specifies the function assigned to the position specified by the user in the device image by referring to the device function management table shown in FIG. And the specific | specification part 38 uses the function allocated to the position designated by the user in the cooperation function group implement | achieved by cooperating PC (A) and a multifunctional machine (B). The priority of the cooperative function that does not use the function is lowered.

  Information relating to the cooperation function identified as described above and information indicating the priority order are transmitted from the server 14 to the terminal device 16. The control unit 48 of the terminal device 16 causes the UI unit 46 to display information related to the cooperative function as information related to the cooperative function candidate according to the priority order.

  For example, as illustrated in FIG. 40B, the control unit 48 of the terminal device 16 displays the cooperation function display screen 140 on the display unit of the UI unit 46, and displays information related to the cooperation function candidates on the cooperation function display screen 140. Let Since the “scan function” and “screen display function” are specified by the user in that order, information on the link function “scan transfer display function” that is executed by linking the “scan function” and the “image display function” However, it is displayed preferentially (for example, at a higher rank) than information related to other linkage functions. For example, information related to the “scan transfer display function” is preferentially displayed over information related to the link function “scan transfer storage function” executed by linking the “scan function” and the “data storage function”. The scan transfer display function is a function for transferring data generated by scanning by the multifunction peripheral (B) to the PC (A) and displaying it on the screen of the PC (A). The scan transfer storage function is a function for transferring data generated by scanning by the multifunction peripheral (B) to the PC (A) and storing it in the PC (A). In the example shown in FIG. 40 (b), explanatory text of each cooperative function is displayed as information relating to each cooperative function.

  According to the collaboration process using partial images, when a device to be linked has multiple functions, the functions are individually specified, and information on the linkage function that uses the specified function is given priority. Is displayed. Thereby, the cooperation function predicted to be used by the user is preferentially displayed.

  Note that the linkage function may be a function that uses a combination of device parts, a function that uses a combination of the entire device and the device part, or a combination of the entire device. It may be a function that uses.

  The cooperation process using the partial image may be applied when a function image associated with the function is used. For example, different functions are assigned depending on the position in the function image, and the cooperation function that uses the function assigned to the position specified by the user is specified.

  The above-described first to eighth embodiments may also be applied to the cooperative processing using partial images. For example, when the partial image in the first image associated with the first device is designated by the user, the second device capable of executing the linkage function together with the function assigned to the portion associated with the partial image. Notification of guidance in whole or in part may be controlled. As another example, when the entire first image associated with the first device is designated by the user, the notification of the guidance of the part of the second device that can execute the cooperation function together with the first device is controlled. Also good. As the notification of the guidance, the whole or part of the second device may be displayed in the candidate list described in the sixth embodiment. When the whole or part of the first device is designated by the user, notification of the guidance of the whole or part of the second device that can execute the linkage function together with the whole or part of the first device may be made, When the whole or part of the second device that cannot be executed with the whole or part of the first device designated by the user is designated, the linkage function may be executed with the whole or part of the first device. Notification of guidance of the whole or part of the second device that can be made may be made. Hereinafter, these processes will be described in detail.

  For example, when the first image (overall image) associated with the first device is designated, the control unit 48 of the terminal device 16 executes the cooperation function together with the first device under the control of the control unit 36 of the server 14. This is a function that can be performed and notifies the guide to one or more functions of the second device. More specifically, the control unit 48 of the terminal device 16 includes one or a plurality of partial images associated with one or a plurality of portions included in the second device (one or a plurality of partial images in the second image associated with the second device). Notification of one or a plurality of partial images associated with one or a plurality of portions to which a function capable of executing the cooperation function with the first device is assigned. For example, as in the first to eighth embodiments, the control unit 48 of the terminal device 16 displays an image representing an arrow for notifying the guidance on the UI unit 46 of the terminal device 16, or the guidance by voice. Or a character string indicating the guidance is displayed on the UI unit 46. Referring to FIG. 40A, when the device image 70 associated with the multifunction device (B) is designated as the first image by the user, the cooperation function can be executed together with the multifunction device (B). The guidance to the partial image associated with the portion to which the function of PC (A) is assigned is notified. For example, when the “screen display function” of the PC (A) corresponds to a function capable of executing the cooperation function together with the multifunction peripheral (B), guidance to the partial image 88b associated with the “screen display function” is provided. Notification is made. For example, an arrow connecting the device image 70 and the partial image 88b associated with the multifunction peripheral (B) is displayed, the “screen display function” is guided by voice, or a character string indicating the “screen display function” is displayed. Or the partial image 88b is displayed in a state where it can be distinguished from other partial images. In addition, when the device image 70 as the first image is designated by the user, the guidance may be notified, or it is a part in the PC (A) and executes the cooperation function together with the multifunction peripheral (B). The guidance may be notified when a part to which a function that cannot be performed is assigned is designated by the user.

  As another example, when a partial image in the first image associated with the first device is designated, the control unit 48 of the terminal device 16 associates with the partial image under the control of the control unit 36 of the server 14. You may notify the guidance to the 2nd apparatus (whole apparatus) which can perform a cooperation function with the function (function which 1st apparatus has) allocated to the part. Referring to FIG. 40A, for example, when the partial image 70a associated with the multifunction peripheral (B) is designated by the user, it is linked with the “print function” assigned to the portion associated with the partial image 70a. The guidance to the second image associated with the second device capable of executing the function is notified. For example, when the PC (A) corresponds to a device that can execute the cooperation function together with the multifunction peripheral (B), guidance to the device image 88 associated with the PC (A) is notified. For example, an arrow connecting the partial image 70a and the device image 88 is displayed, the PC (A) is guided by voice, a character string indicating the PC (A) is displayed, or the device image 88 is another device image. May be displayed in an identifiable state. In addition, when a partial image in the device image 70 is designated by the user, guidance may be notified, or a device image associated with a device that cannot execute the cooperation function together with a function associated with the partial image. When specified by the user, guidance may be notified.

  As another example, when a partial image (referred to as “first partial image”) in the first image associated with the first device is designated, the control unit 48 of the terminal device 16 causes the control unit 36 of the server 14 to A function capable of executing a linkage function together with a function (a function possessed by the first device) assigned to a portion associated with the first partial image under control, and one or a plurality of functions possessed by the second device Notify the guide to. More specifically, the control unit 48 of the terminal device 16 includes one or a plurality of partial images associated with one or a plurality of portions included in the second device (one or a plurality of partial images in the second image associated with the second device). A partial image) (referred to as a “second partial image”), and one or a plurality of functions assigned to a function associated with the function associated with the portion associated with the first partial image. A guide to one or a plurality of second partial images associated with the part is notified. Referring to FIG. 40A, for example, when the partial image 70a associated with the multifunction peripheral (B) is designated by the user, it is linked with the “print function” assigned to the portion associated with the partial image 70a. The guidance to the second partial image associated with the part to which the function of the PC (A) capable of executing the function is assigned is notified. For example, when the “screen display function” of the PC (A) corresponds to a function capable of executing the linkage function together with the “print function”, notification of guidance to the partial image 88b associated with the “screen display function” Is made. For example, an arrow connecting the partial image 70b and the partial image 88b is displayed, the “screen display function” is guided by voice, a character string indicating the “screen display function” is displayed, and the partial image 88b Displayed in a state that can be distinguished from the partial image. In addition, when a partial image in the device image 70 is designated by the user, guidance may be notified, or a partial image associated with a function that cannot execute the cooperation function together with a function associated with the partial image. When specified by the user, guidance may be notified.

  Three or more functions may be designated by the user by designating three or more partial images. For example, when two partial images (that is, two functions (first function and second function)) are designated by the user, a function (third function) capable of executing the linkage function together with the two functions. Guidance to the part to which is assigned may be notified. In this case, the specifying unit 38 of the server 14 may change the guided third function in accordance with the designation order of the partial image associated with the first function and the partial image associated with the second function.

  A plurality of functions that the same device has may be designated as functions to be linked. For example, the “screen display function” and the “data storage function” of the PC (A) may be designated by the user as functions to be linked, the “scan function” of the multifunction machine (B), the “scan function” of the PC (A), The “screen display function” and the “data storage function” may be designated by the user as functions to be linked. Even in this case, the guidance to the part to which the function (for example, the second function) capable of executing the linkage function is assigned together with the previously specified function (for example, the first function) is notified. In addition, the specifying unit 38 of the server 14 may specify a cooperation function that uses each function according to the specified order, and increase the priority of the cooperation function.

(Another example of cooperative processing using partial images)
Hereinafter, another example of the cooperation process using the partial images will be described with reference to FIGS. 41 and 42.

  FIG. 41 shows an example of the device function management table. Data of the device function management table is stored in the server 14 as device function management information 32. In the device function management table, as an example, the device ID, information indicating a device name (for example, the type of device), information indicating the name of the device portion (for example, the type of portion), and the part are identified. A part ID as part identification information, information indicating a function assigned to the part (function possessed by the part), and a part image ID for identifying a part image associated with the part are associated with each other. It has been. The partial image is an image representing the appearance of the part of the device obtained by photographing with the camera. Of course, a partial image that schematically represents the part of the device may be associated with the part. For example, a different function is assigned to each part of the device.

  A specific example will be described. A screen display function is assigned to the display unit of PC (A), and information indicating the screen display function is associated with the partial image ID of the partial image associated with the display unit. It has been. The screen display function is a function for displaying information on the PC (A). A data storage function is assigned to the main body of the PC (A), and information indicating the data storage function is associated with the partial image ID of the partial image associated with the main body. The data storage function is a function for storing data in the PC (A).

  Also, a print function is assigned to the main body of the multifunction peripheral (B), and information indicating the print function is associated with the partial image ID of the partial image associated with the main body. A scanning function is assigned to the reading unit of the multifunction device (B) (for example, a portion corresponding to the document cover, document glass, or automatic document feeder of the multifunction device (B)), and a portion associated with the reading unit Information indicating the scan function is associated with the partial image ID of the image. The stapling function is assigned to the post-processing apparatus of the multifunction peripheral (B), and information indicating the stapling function is associated with the partial image ID of the partial image associated with the processing apparatus thereafter. The stapling function is a function of binding the output paper by stapling.

  The function assigned to the device part is specified (identified) by using, for example, a markerless AR technology. For example, when a device portion is photographed by a camera (for example, the camera 42 of the terminal device 16), appearance image data representing the portion is transmitted from the terminal device 16 to the server 14. The specifying unit 38 of the server 14 specifies (identifies) a function associated with the appearance image data in the device function management table. Thereby, the function assigned to the photographed part is specified (identified). For example, when the main body of the multifunction peripheral (B) is photographed by the camera 42, appearance image data representing the main body of the multifunction peripheral (B) is transmitted from the terminal device 16 to the server 14. The specifying unit 38 of the server 14 specifies the print function associated with the appearance image data in the device function management table. Thereby, it is specified that the function assigned to the main body of the multifunction peripheral (B) is the print function.

  Of course, the function assigned to the part of the device may be specified (identified) by using the marker type AR technology. For example, each part of the device is provided with a marker such as a two-dimensional barcode in which part identification information (for example, part ID) for identifying the part is encoded. When a marker provided in a part is photographed by a camera and the marker type AR technology is applied, partial identification information (for example, a partial ID) of the part is acquired. The application of the marker type AR technology may be performed by the terminal device 16 or the server 14. When the partial identification information is acquired in this way, the specifying unit 38 of the server 14 specifies (identifies) the function associated with the partial identification information (for example, the partial ID) in the device function management table.

  FIG. 42 shows an example of the cooperation function management table. The data of this linkage function management table is stored in the server 14 as linkage function management information 34. This cooperation function management table is information indicating a cooperation function that uses the functions of a plurality of parts. In the cooperation function management table, for example, information indicating a combination of device parts and a combination of part IDs are included. The information indicating and the information indicating the cooperation function that uses the functions of a plurality of parts included in the combination are associated with each other. Of course, in the linkage function management table, information indicating the combination of the device portion and the entire device, and information indicating the linkage function using the function of the device portion and the function of the entire device are: It may be associated.

  A specific example will be described. A print function as a cooperation function is assigned to the combination of the display unit of the PC (A) and the main unit of the multifunction peripheral (B). Information indicating the combination of the partial ID and the partial ID of the main body of the multifunction peripheral (B) is associated with information indicating a print function as a cooperation function. The print function as the linkage function is a function that, for example, transmits data stored in the PC (A) to the multifunction device (B) and prints the data by the multifunction device (B).

  A print function as a cooperation function is assigned to the combination of the main body of the multifunction peripheral (B) and the main body of the projector (C). The information indicating the print function as the cooperation function is associated with the information indicating the combination with the partial ID of the main body portion of C). The print function as the cooperation function is, for example, a function of transmitting data projected by the projector (C) to the multifunction device (B) and printing the data by the multifunction device (B).

  Further, a scan projection function as a cooperation function is assigned to the combination of the reading unit of the multifunction device (B) and the main body of the projector (C), and the partial ID of the reading unit of the multifunction device (B) and the projector The information indicating the combination with the partial ID of the main body part in (C) is associated with information indicating the scan projection function as the cooperation function. The scan projection function as the cooperation function is a function of, for example, transmitting data generated by scanning by the multifunction peripheral (B) to the projector (C) and projecting the data by the projector (C).

  Note that the linkage function may be a function that uses a function of a plurality of parts included in the same device, or a function that uses a function of a part of a plurality of different devices. In addition, the cooperation function may be a function that uses functions of three or more parts.

  For example, when a plurality of parts of a device (for example, a plurality of parts of a plurality of different devices or a plurality of parts of the same device) are specified (identified) using the marker type AR technology or the markerless type AR technology. The specifying unit 38 of the server 14 specifies (identifies) a cooperative function associated with a combination of a plurality of identified parts in the cooperative function management table. Thereby, the cooperation function using the function which the some part identified (for example, imaging | photography) has is pinpointed (identification). For example, the main body of the multifunction peripheral (B) and the main body of the projector (C) are photographed by the camera 42 of the terminal device 16, and the main body of the multifunction peripheral (B) and the main body of the projector (C) are identified. In this case, the specifying unit 38 of the server 14 specifies a print function or the like as a link function associated with a combination of the main body unit of the multifunction peripheral (B) and the main body unit of the projector (C) in the link function management table. To do.

  In such a cooperative process, the first to eighth embodiments may be applied. For example, when the first part of the device is specified (identified), the control unit 48 of the terminal device 16 performs the cooperation function together with the function assigned to the first part under the control of the control unit 36 of the server 14. Guide the second part of the instrument that can be performed.

(Designation of devices to be linked by overlaying device images)
A device group to be linked may be designated by superimposing a plurality of device images. Hereinafter, this process will be described with reference to FIGS. 43 and 44. FIGS. 43 and 44 show an example of a screen displayed on the UI unit 46 of the terminal device 16.

  For example, it is assumed that the multifunction peripheral (B) and the PC (A) are identified. As shown in FIG. 43A, a device display screen 68 is displayed on the UI unit 46 of the terminal device 16, and device images 70 and 88 associated with the identified device are displayed on the device display screen 68. Has been. In this state, the user uses an indicator (for example, the user's finger, pen, stylus, etc.) to superimpose the device image associated with the first device on the device image associated with the cooperation destination device (second device). For example, as shown in FIG. 43 (b), the user designates the device image 70 using the operator, and superimposes the device image 70 on the device image 88 as indicated by an arrow 142. For example, device images are overlapped by a drag and drop operation. That is, the user performs a drag operation on the device image 70 and performs a drop operation on the device image 70 at a position overlapping the device image 88. This drag and drop operation itself is a known technique, for example. Note that a device image to be superimposed may be designated in accordance with a user's voice instruction. For example, the device images 70 and 88 may be designated as device images to be superimposed in accordance with a user's voice instruction, and may be superimposed.

  By superimposing the device images 70 and 88 on each other, the multifunction device (B) associated with the device image 70 and the PC (A) associated with the device image 88 are designated as devices to be linked. For example, the multifunction device (B) specified earlier corresponds to the first device. If the PC (A) specified later is a device that cannot execute the linkage function with the multifunction peripheral (B), the linkage function with the multifunction peripheral (B) is the same as in the first to eighth embodiments. Information on devices that can be executed is notified.

  Note that the control unit 48 of the terminal device 16 may display the device image being dragged on the UI unit 46 in a display mode that can be identified. For example, the device image being dragged may be displayed semi-transparently or displayed in a specific color.

  When the device image 70 is superimposed on the device image 88 and the PC (A) can execute the cooperation function together with the multifunction peripheral (B), as shown in FIG. A confirmation screen 144 is displayed on the UI unit 46. The confirmation screen 144 is a screen for confirming whether or not a device group designated as a device to be linked is to be linked. When a cooperation instruction is given by the user on the confirmation screen 144 (for example, when the “YES” button is pressed by the user), information related to the cooperation function is displayed on the UI unit 46 of the terminal device 16.

  For example, as illustrated in FIG. 44A, the control unit 48 of the terminal device 16 displays the cooperation function display screen 146 on the UI unit 46 and displays information on the cooperation function candidate on the cooperation function display screen 146. By linking the PC (A) and the multifunction peripheral (B), for example, a “scan transfer function” and a “print function” are realized, and the link function display screen 146 includes information on the “scan transfer function” and Information about “printing function” is displayed.

  When the cooperation function is designated by the user and an execution instruction is given, a connection request is made from the terminal device 16 to the cooperation target device. As shown in FIG. 44B, a standby screen 148 is displayed on the UI unit 46 of the terminal device 16 during the connection request. When the connection between the terminal device 16 and the device to be linked is successful, the designated linkage function is executed.

  As described above, a cooperation function that uses a function of a device is specified by superimposing device images associated with the device. Therefore, the functions can be linked without performing an operation other than the image operation, and the functions are linked with a simple operation. Even in this case, since the guidance of the second device capable of executing the linkage function together with the first device is notified, the convenience of the user when using the linkage function is improved compared to the case where the guidance is not notified. improves.

  The cooperation function may be specified by superimposing the partial image on the device image or the partial image. This process will be described with reference to FIG. FIG. 45 shows an example of a screen displayed on the UI unit 46 of the terminal device 16.

  Similar to the cooperation process using the partial image described above, the function of the device differs depending on the position in the device image associated with the device. By superimposing the partial images in the device images on the partial images in the same or different device images, a cooperation function that uses a function associated with both partial images is specified. Hereinafter, this process will be described in detail.

  For example, it is assumed that the multifunction peripheral (B) and the PC (A) are identified. As shown in FIG. 45A, a device display screen 68 is displayed on the UI unit 46 of the terminal device 16, and device images 70 and 88 are displayed on the device display screen 68. For example, the partial images 70a, 70b, 70c, 88a, and 88b are displayed as images that can be moved separately from other partial images.

  When a partial image is designated by the user and the partial image is superimposed on another partial image, a cooperation function that uses a function associated with both partial images is specified, and information regarding the cooperation function is stored in the UI of the terminal device 16. Displayed on the unit 46. This specifying process may be performed by the specifying unit 38 of the server 14 or may be performed by the terminal device 16.

  For example, as indicated by an arrow 150 in FIG. 45B, when the user drags the partial image 70b using the operator and performs a drop operation on the partial image 88b, the partial image 70b is included. The multifunction device (B) associated with the device image 70 and the PC (A) associated with the device image 88 including the partial image 88b are designated as devices to be linked and are assigned to the partial image 70b by the “scan function” "And the" screen display function "assigned to the partial image 88b are designated as functions to be linked.

  In the server 14, functions assigned to the partial images are managed. For example, identification information for identifying a partial image, functional information indicating a function associated with the partial image, and cooperative function information indicating a cooperative function executed by linking the functions are associated with each other. Is stored in the server 14. When partial images are selected on the device display screen 68 and superimposed on other partial images, identification information indicating these partial images is transmitted from the terminal device 16 to the server 14. In the example shown in FIG. 45B, identification information indicating the partial images 70 b and 88 b is transmitted from the terminal device 16 to the server 14. Based on the identification information, the specifying unit 38 of the server 14 specifies a function assigned to each of the superimposed partial images 70b and 88b, and specifies a cooperation function that uses the function. Information regarding the cooperation function is transmitted from the server 14 to the terminal device 16 and displayed.

  According to the above processing, when a device to be linked has a plurality of functions, the functions are individually designated, and information related to the linked function using the designated function is preferentially displayed. Thereby, the cooperation function predicted to be used by the user is preferentially displayed.

  Further, the display priority of the cooperation function may be changed according to the order in which the partial images are superimposed. In this case, information related to the cooperation function that uses the function associated with the superimposed partial images is preferentially displayed.

  In the case of overlapping partial images, the previously specified partial image corresponds to the first image, and the function associated with the partial image corresponds to the first function. The partial image specified later (partial image on which the first image is superimposed) corresponds to the second image, and the function associated with the partial image corresponds to the second function. If the function specified later is a function that cannot execute the linkage function together with the function specified earlier, the linkage function is executed together with the function specified earlier as in the first to eighth embodiments. Information on functions that can be performed is notified. In the above example, when the “screen display function” associated with the partial image 88b specified later cannot execute the linkage function together with the “scan function” associated with the partial image 70b previously specified, The function guide capable of executing the linkage function is notified together with the previously designated “scan function”. In this case, the whole device (for example, the device image itself) capable of executing the linkage function together with the previously specified function (for example, “scan function”) may be guided, or together with the previously specified function. A part of a device (for example, a partial image) to which a function capable of executing the cooperation function is assigned may be guided.

(Switching between single function and linkage function display)
In the present embodiment, switching control between a display of a function used by the device alone (hereinafter referred to as “single function”) and a display of the cooperation function may be performed.

  For example, when only one device is identified within a preset identification time, the control unit 48 of the terminal device 16 displays information regarding the function of the one device (for example, the image forming apparatus 10) as a single function. As the information regarding, it is displayed on the UI unit 46 of the terminal device 16. The length of the identification period may be changed by the user. The device may be identified by applying the AR technology, or the device may be identified by applying a technology other than the AR technology. The device identification process may be performed by the server 14 or the terminal device 16. The start point (starting point) of the identification time may be, for example, a time point when one device is identified, or may be a time point specified by the user (for example, the identification process start time point).

  For example, when another device is not identified within the identification time from the time when one device is identified, the control unit 48 of the terminal device 16 uses information regarding the function of the one device as information regarding a single function. It is displayed on the UI unit 46 of the terminal device 16. In this case, the one device is treated as a device identified within the identification period. The information related to the device may be information transmitted from the server 14 to the terminal device 16 or information stored in the terminal device 16 in advance.

  As another example, when only one device is identified within the identification time from the start time when the time specified by the user is the start time of the identification time, the control unit 48 of the terminal device 16 Information related to the functions possessed is displayed on the UI unit 46 of the terminal device 16 as information related to the single use device.

  As another example, when a single function display instruction is given after one device is identified, the control unit 48 of the terminal device 16 uses information related to the function of the one device as information related to the single function. The information may be displayed on the UI unit 46 of the terminal device 16. The control unit 48 of the terminal device 16 displays, for example, a button image for instructing the display of a single function when always or when one device is photographed (or when one device is identified). 16 UI parts 46 are displayed. When the button image is pressed by the user, the control unit 48 causes the UI unit 46 to display information related to the function of the one device.

  When the identification time has elapsed, the control unit 48 of the terminal device 16 may display a confirmation screen on the UI unit 46 of the terminal device 16. The confirmation screen is, for example, a screen for the user to instruct extension of the identification time. When the user instructs the extension of the identification time via the confirmation screen, when another device is not photographed within the extension time, the control unit 48 of the terminal device 16 displays information on the function of the identified device. It is displayed on the UI unit 46 of the terminal device 16.

  The display control of information related to a single function will be described in more detail. For example, it is assumed that the device is identified by the marker type AR technology or the markerless type AR technology. For example, when only one device is photographed within a preset photographing time, the control unit 48 of the terminal device 16 uses the information regarding the function of the one device as information regarding the single function. Are displayed on the UI part 46 of FIG. The start point (starting point) of the shooting time may be the time when one device is shot, or may be the time point designated by the user (for example, the shooting start time). The length of the shooting time may be changed by the user. When the one device is photographed, the one device is identified by the marker type AR technology or the markerless type AR technology. This identification process may be performed by the server 14 or the terminal device 16.

  For example, when another device is not photographed within the photographing time from the time when one device is photographed, the control unit 48 of the terminal device 16 uses information regarding the function of the one device as information regarding a single function. It is displayed on the UI unit 46 of the terminal device 16. In this case, the one device is treated as a device photographed within the photographing time.

  As another example, when the time point specified by the user is set as the start time of the shooting time, and only one device is shot within the shooting time from the start time, the control unit 48 of the terminal device 16 causes the one device to Information related to the function that the user has is displayed on the UI unit 46 of the terminal device 16 as information related to the single function.

  As another example, when a single function display instruction is given after one device is photographed, the control unit 48 of the terminal device 16 uses information related to the function of the one device as information related to the single function. The information may be displayed on the UI unit 46 of the terminal device 16.

  For example, when only one device is photographed within the photographing time (for example, when another device is not photographed within the photographing time from the time when the first device was photographed, or from the start time specified by the user) When only one device is photographed in time), the control unit 48 of the terminal device 16 transmits the image data generated by the photographing to the server 14. The shooting time may be measured by the control unit 48 or may be measured by a timer. The specifying unit 38 of the server 14 specifies (identifies) a device based on the image data, and specifies the function of the device. Information regarding the function is transmitted from the server 14 to the terminal device 16 and displayed on the UI unit 46 of the terminal device 16. Of course, the terminal device 16 may not manage the time, but the server 14 may manage the time and transmit information about the functions of the identified device to the terminal device 16.

  When the shooting time has elapsed, the control unit 48 of the terminal device 16 may display a confirmation screen on the UI unit 46 of the terminal device 16. The confirmation screen is, for example, a screen for the user to instruct extension of the shooting time. When the user instructs to extend the shooting time via the confirmation screen, if another device is not shot within the extended time, the control unit 48 of the terminal device 16 sends the image data obtained by shooting to the server 14. In response, the function of the one device is displayed on the UI unit 46 of the terminal device 16. The length of the extension time may be changed by the user.

  As another example, when an instruction to display a single function is given after one device has been photographed, the control unit 48 of the terminal device 16 transmits the image data generated by the photographing to the server 14, and Accordingly, information regarding the functions of the photographed device may be received from the server 14.

  As yet another example, the control unit 48 of the terminal device 16 transmits the image data to the server 14 each time the device is photographed and image data is generated, and the function of the photographed device accordingly. Information regarding the server 14 may be received. In this case, when only one device is photographed within the photographing time, the control unit 48 of the terminal device 16 causes the UI unit 46 of the terminal device 16 to display information regarding the one device as information regarding a single function. .

  On the other hand, when a plurality of devices are identified within the identification time, the control unit 36 of the server 14 executes the cooperation function mode. In the cooperation function mode, the guidance to the second device capable of executing the cooperation function together with the first device is notified in the same manner as in the first to eighth embodiments.

  For example, when another device is identified within the identification time from when the first device is identified, the cooperation function mode is executed. In this case, the first device is also treated as a device identified within the identification time. In addition, when another device is identified within the identification time from the time when the first device is identified, the control unit 36 of the server 14 starts a new time starting from the time when the other device is identified. An identification time may be set. The same applies to the subsequent processes. When another device is identified within the new identification time, a new identification time is set.

  As another example, when the time point specified by the user is set as the start time of the identification time, and a plurality of devices are photographed within the identification time from the start time, the cooperation function mode may be executed.

  As another example, the cooperation function mode may be executed when a display instruction of the cooperation function is given after a plurality of devices are identified. For example, when a plurality of devices are photographed (or when a plurality of devices are identified), the control unit 48 of the terminal device 16 displays a button image for instructing display of the cooperation function on the UI unit 46. To display. When the button image is pressed by the user, the cooperation function mode is executed.

  As another example, after the first device is identified, when another device is identified while the execution instruction for the function of the first device is not given, the cooperative function mode is executed. May be.

  The execution of the cooperation function mode will be described in more detail. For example, it is assumed that a plurality of devices are identified by the marker type AR technology or the markerless type AR technology. For example, when a plurality of devices are photographed within a preset photographing time, the cooperation function mode is executed. For example, when another device is photographed within the photographing time from the time when the first device is photographed, the cooperation function mode is executed. In this case, the first device is also treated as a device that has been photographed within the photographing time. Further, when another device is photographed within the photographing time from the time when the first device is photographed, the control unit 36 of the server 14 starts a new time from the time when the other device is photographed. The shooting time may be set. The same applies to the following, and when another device is photographed within the new photographing time, a new photographing time is set.

  As another example, when the time point specified by the user is set as the start time of the shooting time, and a plurality of devices are shot within the shooting time from the start time, the cooperation function mode may be executed.

  As another example, the cooperation function mode may be executed when a display instruction of the cooperation function is given after a plurality of devices are photographed.

  As another example, after the first device is photographed, when another device is photographed while the execution instruction for the function of the first device is not given, the cooperative function mode is executed. May be.

  As described above, according to the display switching process of the single function and the cooperation function, when one device is identified (for example, photographed), information on the function of the one device is displayed, and a plurality of devices are identified ( For example, when shooting is performed, the cooperation function mode is executed. Accordingly, information on functions that can be executed using the identified (eg, photographed) device is provided to the user, which is convenient for the user.

  In addition, since it is possible to use a single function or a linked function simply by identifying the device by applying the AR technology, it is simpler than the case where the user performs the setting for executing the function manually. This makes it possible to use each function, reducing the user's trouble.

  In the present embodiment, the device image associated with the identified device or the superimposed device image may be displayed in three dimensions so as not to be buried in the background image. That is, those images may be displayed as a three-dimensional image. For example, the background image is displayed in two dimensions and the device image is displayed in three dimensions. As a result, the visibility of the device image is improved. Further, the color of the designated device image may be changed, or the designated device image may be vibrated so that the device image designated by the user is identified from other device images.

  In addition, according to the present embodiment, by applying the AR technology, a cooperation function that uses a function of a cooperation target device is specified, and information regarding the cooperation function is displayed. Thereby, even if it is not clear from the appearance what kind of cooperation function can be executed by the cooperation target device, information regarding the cooperation function is provided to the user. In addition, functions that cannot be executed individually by individual devices can be executed by linking a plurality of devices, which is convenient. In addition, since the cooperation function can be used simply by identifying the device to be linked by applying the AR technology, it is simpler than the case where the user performs settings for executing the cooperation function manually. The cooperation function can be used by operation, and the user's trouble is reduced.

  Each of the image forming apparatus 10, the server 14, and the terminal device 16 is realized by cooperation of hardware and software, for example. Specifically, each of the image forming apparatus 10, the server 14, and the terminal device 16 includes one or a plurality of processors such as a CPU (not shown). The function of each unit of the image forming apparatus 10, the server 14, and the terminal device 16 is realized by the one or more processors reading and executing a program stored in a storage device (not shown). The program is stored in the storage device via a recording medium such as a CD or DVD, or via a communication path such as a network. As another example, each unit of the image forming apparatus 10, the server 14, and the terminal device 16 may be realized by hardware resources such as a processor, an electronic circuit, and an ASIC (Application Specific Integrated Circuit). In the realization, a device such as a memory may be used. As another example, each part of the image forming apparatus 10, the server 14, and the terminal device 16 may be realized by a DSP (Digital Signal Processor), an FPGA (Field Programmable Gate Array), or the like.

  10, 12 devices, 14 servers, 16 terminal devices, 32 device function management information, 34 linkage function management information, 38 identification unit.

Claims (25)

  When the first function associated with the first device necessary for executing the cooperation function is designated when the cooperation function is not notified, the second device can execute the cooperation function together with the first device. An information processing apparatus having control means for controlling notification of guidance. The first device and the second device are devices identified by being photographed by photographing means.
The information processing apparatus according to claim 1.   If the first image associated with the first function necessary for executing the cooperation function is designated when the cooperation function is not notified, the second function capable of executing the cooperation function together with the first function. An information processing apparatus having control means for controlling notification of guidance.   When the first image associated with the first device necessary for executing the cooperation function is designated, the second image associated with the second device capable of executing the cooperation function together with the first device from the first image. An information processing apparatus having control means for controlling the display of a guidance instruction to the user. The control means controls display of an image indicating the second image with the first image as a starting point as control of display of a guidance instruction.
The information processing apparatus according to claim 4. The control means controls display of an image connecting the first image and the second image as control of display of a guidance instruction.
The information processing apparatus according to claim 4. The control means further controls display of a guidance instruction to a third device capable of executing the cooperation function together with the first device and the second device when the first image and the second image are designated. ,
The information processing apparatus according to claim 4. The control means controls display of the guidance instruction by displaying an image indicating the second image starting from the first image and displaying a third image pointing the third device starting from the second image. Control,
The information processing apparatus according to claim 7. The control means controls display of an image connecting the first image and the second image and display of an image connecting the second image and the third image as control of display of a guidance instruction.
The information processing apparatus according to claim 7. The control means controls the display of the guidance instruction by changing the third device according to the designation order of the first image and the second image.
The information processing apparatus according to claim 7, wherein the information processing apparatus is an information processing apparatus.   When the first image associated with the first device necessary for executing the cooperation function is designated, guidance to a device capable of executing the cooperation function together with the first device at the first time point when the designation is performed. When the first image is designated at a second time point different from the first time point, guidance to a device that can execute the cooperation function according to a change in the surrounding environment of the first device Information processing apparatus having control means for changing the notification. When the surrounding environment changes to an environment that requires the device for the first device at the second time point, the control unit controls notification of guidance to the device.
The information processing apparatus according to claim 11. The surrounding environment includes the status of the device,
The information processing apparatus according to claim 11 or 12, characterized in that: When the first image associated with the first device necessary for executing the cooperation function is designated, the notification of guidance to one or a plurality of second devices that can execute the cooperation function together with the first device is controlled. Having control means,
The control means controls notification of guidance to each second device according to a priority according to a combination of the first device and each second device.
Information processing device. The priority is a priority of the cooperation function that can be executed using the first device and the second device.
The information processing apparatus according to claim 14. When the first image associated with the first function necessary for executing the cooperation function is designated, the notification of guidance to one or a plurality of second functions capable of executing the cooperation function together with the first function is controlled. Having control means,
The control means controls notification of guidance to each second function according to a priority according to a combination of the first function and each second function.
Information processing device. The priority is a priority of the cooperation function that can be executed using the first function and the second function.
The information processing apparatus according to claim 16. When the first image associated with the first device necessary for executing the cooperation function and the second image associated with the second device capable of executing the cooperation function together with the first device are designated, the first image Control means for notifying a third device capable of executing the linkage function together with the device and the second device;
The control means controls the notification of the guidance by changing the third device according to the designation order of the first image and the second image.
Information processing device. Computer
When the first function associated with the first device necessary for executing the cooperation function is designated when the cooperation function is not notified, the second device can execute the cooperation function together with the first device. Control means for controlling notification of
Program to function as. Computer
If the first image associated with the first function necessary for executing the cooperation function is designated when the cooperation function is not notified, the second function capable of executing the cooperation function together with the first function. Control means for controlling notification of
Program to function as. Computer
When the first image associated with the first device necessary for executing the cooperation function is designated, the second image associated with the second device capable of executing the cooperation function together with the first device from the first image. Control means for controlling the display of guidance instructions to
Program to function as. Computer
When the first image associated with the first device necessary for executing the cooperation function is designated, guidance to a device capable of executing the cooperation function together with the first device at the first time point when the designation is performed. When the first image is designated at a second time point different from the first time point, guidance to a device that can execute the cooperation function according to a change in the surrounding environment of the first device Control means to change the notification of
Program to function as. Computer
When the first image associated with the first device necessary for executing the cooperation function is designated, the notification of guidance to one or a plurality of second devices that can execute the cooperation function together with the first device is controlled. Function as a control means,
The control means controls notification of guidance to each second device according to a priority according to a combination of the first device and each second device.
program. Computer
When the first image associated with the first function necessary for executing the cooperation function is designated, the notification of guidance to one or a plurality of second functions capable of executing the cooperation function together with the first function is controlled. Function as a control means,
The control means controls notification of guidance to each second function according to a priority according to a combination of the first function and each second function.
program. Computer
When the first image associated with the first device necessary for executing the cooperation function and the second image associated with the second device capable of executing the cooperation function together with the first device are designated, the first image Function as a control means for notifying a third device capable of executing the linkage function together with the device and the second device;
The control means controls the notification of the guidance by changing the third device according to the designation order of the first image and the second image.
program.