JP2016081312A - Image processing system and interlocking modification method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To notify a modification added to one data set to a user looking at another data set.SOLUTION: An image processing device includes a substituting unit for converting original data into substitution data and an output control unit for outputting the substitution data. A server includes; an associated portion reply unit 559 that, upon receipt of a transmission request containing an AR tag and acquired data from a first portable information device, transmits an associated portion associated with the AR tag to the first portable information device; an added portion extraction unit 563 that extracts an added portion that is present in the acquired data but not present in the substitution data; a device identification unit 565 that identifies a second portable information device to which a second associated portion associated with second original data, which is associated with first original data, was transmitted; and an interlocking transmission unit 567 that transmits the added portion to the second portable information device.SELECTED DRAWING: Figure 9

Description

  The present invention relates to an image processing system and an interlocking change method, and more particularly to an image processing system that displays an image corresponding to an AR tag in a real space imaged by a camera, and an interlocking change executed by the image processing system. Regarding the method.

  In recent years, technologies related to augmented reality have been developed. For example, Japanese Patent Laid-Open No. 2013-09287 discloses an acquisition unit that acquires information about a plurality of markers included in an image, and a display unit that superimposes a plurality of virtual objects respectively corresponding to the plurality of markers on the image. Display control means for controlling the display to be displayed on the display, and determination means for determining whether or not the plurality of virtual objects are in a specific combination, and the plurality of virtual objects are in a specific combination by the determination means The display control means controls the plurality of virtual objects to be superimposed on the image and displayed on the display unit in a specific arrangement corresponding to the specific combination. An image processing apparatus is described.

In the conventional image processing apparatus, since the virtual object is paired with the marker, if the virtual object is changed, the changed virtual object that is paired with the marker is displayed. However, when making the same change to a plurality of related virtual objects, there is a problem that the same change must be made for each of the plurality of virtual objects.
JP2013-09287A

  The present invention has been made to solve the above-described problems, and one of the objects of the present invention is to provide an image capable of notifying a user who views other data of a change added to one data. It is to provide a processing system.

  Another object of the present invention is to provide a linked display method capable of notifying a user who views other data of a change added to one data.

  In order to achieve the above object, according to one aspect of the present invention, an image processing system includes an image forming apparatus, a plurality of portable information devices, and a server, and the image forming apparatus includes: Replacement means for converting the original data into replacement data obtained by replacing at least a related portion of the original data with an AR tag, original identification information for identifying the original data, an AR tag, An original association that associates the original data with another original data different from the original data, an AR tag registration means for transmitting registration information including the relevant part and replacement data to the server, an output control means for outputting the replacement data Data association registration means for registering the attached information to the server, and the server receives the information from the image forming apparatus. In response to receiving the recording information, transmission including the original identification information included in the registration information, the AR tag, the related part, the AR tag association means for associating the replacement data, and the AR tag and the acquired data Related part reply means for transmitting the related part associated with the AR tag included in the transmission request to the first portable information device in response to receiving the request from the first portable information device among the plurality of portable information devices. And an additional portion extracting means for extracting an additional portion that is present in the acquired data included in the transmission request but is not present in the replacement data associated with the AR tag included in the transmission request, and the additional portion is extracted by the additional portion extracting means. The first original data associated with the AR tag included in the transmission request and the original association information registered by the image forming apparatus. The related original specifying means for specifying the associated second original data, and when the second original data is specified by the related original specifying means, the second original data is associated with the second original data from among the plurality of portable information devices. A plurality of device specifying means for specifying the second portable information device in which the second related part is transmitted by the related part reply means; and an interlocking transmitting means for transmitting the additional portion to the second portable information device. Each of the portable information devices includes: acquisition data acquisition means for acquiring replacement data output by the image forming apparatus or data obtained by changing a part of the replacement data as acquisition data; and AR tag extraction for extracting an AR tag from the acquisition data And a related partial request for transmitting to the server a transmission request including the AR tag extracted from the acquired data and the acquired data Means, a related part receiving means for receiving the related part associated with the AR tag from the server after transmitting the transmission request, and a position determined based on the position in the acquired data of the AR tag extracted from the acquired data, Display control means for displaying an image of the received related portion, and the display control means provided in the second portable information device among the plurality of portable information devices is configured to display the second related portion from the server after displaying the second related portion. In response to receiving the additional portion, it includes interlocking display control means for displaying the received image of the additional portion at a position determined based on the position in the acquired data of the AR tag extracted from the acquired data.

  According to this aspect, replacement data output by the image forming apparatus or data obtained by changing a part of the replacement data is acquired as acquisition data by each of the plurality of portable information devices, and an AR tag is extracted from the acquisition data. When the related part associated with the AR tag is received from the server, an image of the related part is displayed at a position determined based on the position in the acquired data. For this reason, since the related part is displayed at the position defined by the AR tag in the acquired data obtained in the portable information device, the related part can be displayed in the acquired data. In addition, the server extracts the additional portion from the acquired data received from the first portable information device, the first original data associated with the AR tag received from the first portable information device, and the image forming apparatus When the second original data associated with the second original data is specified, the additional portion is transmitted to the second portable information device to which the second related portion associated with the second original data is transmitted. In the second portable information device, the second related part and the additional part are displayed. For this reason, in the acquired data acquired in the first portable information device, an additional portion changed from the replacement data can be displayed on the second portable information device. As a result, it is possible to provide an image processing system capable of notifying a user who views other data of a change added to one data.

  Preferably, the interlocking transmission means included in the server transmits the additional part on condition that a predetermined condition is satisfied.

  According to this aspect, the additional portion can be displayed on the second portable information device only when a predetermined condition is satisfied.

  Preferably, the image forming apparatus further includes priority registration means for registering, in the server, priority information that defines relative priorities for the first original data and the second original data. The interlocking transmission means provided includes priority comparison means for comparing the priority of the first original data and the priority of the second original data, and when the comparison result by the priority comparison means is the first comparison result. , Send the additional part.

  According to this aspect, when the priority of the first original data is equal to or higher than the priority of the second original data, the additional portion can be displayed on the second portable information device.

  Preferably, the server analyzes a user specifying means for specifying a user of each of the first portable information device and the second portable information device, and an additional portion extracted based on the first transmission request by the additional portion extracting means. A security level determination means for determining a security level for the additional portion, and the interlocking transmission means provided in the server determines the security level predetermined for the user of the second portable information device. Security level comparison means for comparing with the security level for the additional part determined by the means is further included, and when the comparison result by the security level comparison means is the second comparison result, the additional part is transmitted.

  According to this aspect, when the confidential level for the user who operates the second portable information device is equal to or higher than the confidential level for the additional portion, the additional portion can be displayed on the second portable information device.

  Preferably, the image forming apparatus further includes a data acquisition unit that acquires data, and the data association registration unit associates each of the plurality of pages included in the acquired data with each other as a plurality of original data. Means.

  According to this aspect, since multiple pieces of original data included in the same data are related, the changes made to the replacement data corresponding to the original data of any page of the same data are replaced with the original data of other pages. Users can be notified of the data.

  Preferably, the data association registration unit includes a plurality of replacement data respectively output by the data output unit by a plurality of operations by the same user or a plurality of operations by a plurality of users belonging to the same group. The same user associating means for associating the original data is included.

  According to this aspect, changes made to any of a plurality of replacement data output by a plurality of operations by the same user or a plurality of operations by one of a plurality of users belonging to the same group are changed to other replacement data. You can notify the viewer.

  Preferably, the data association registration unit selects the candidate data acquisition unit that acquires the original data from the server as candidate data and the original data corresponding to the replacement data output by the data output unit from the candidate data by the user. And operation association means for associating with the candidate data.

  According to this aspect, a change made to the replacement data corresponding to any of the plurality of original data associated by the user operating the image processing apparatus is notified to the user who sees the replacement data corresponding to the other original data. be able to.

  According to another aspect of the present invention, an interlocking display method is an interlocking change method executed in an image processing system including an image forming apparatus, a plurality of portable information devices, and a server. A replacement step for converting the original data into replacement data obtained by replacing at least a related portion in the original data with an AR tag, original identification information for identifying the original data, an AR tag, An original association that associates the original data with another original data different from the original data, an AR tag registration step that transmits registration information including the part and replacement data to the server, an output control step that outputs the replacement data A data association registration step of registering information with the server, and causing the server to execute the image forming apparatus. The registration information is received, and includes an AR tag association step for associating the original identification information included in the registration information, the AR tag, the related part, and the replacement data, and the AR tag and the acquired data. In response to receiving the transmission request from the first portable information device among the plurality of portable information devices, the related portion reply that transmits the related portion associated with the AR tag included in the transmission request to the first portable information device. An additional portion extraction step for extracting an additional portion that is present in the acquired data included in the transmission request but is not present in the replacement data associated with the AR tag included in the transmission request, and the additional portion is included in the additional portion extraction step. When extracted, the first original data associated with the AR tag included in the transmission request and the original registered by the image forming apparatus And a related original specifying step for specifying the second original data related by the association information, and when the second original data is specified in the related original specifying step, the second original is selected from the plurality of portable information devices. A device specifying step for specifying the second portable information device in which the second related portion associated with the data is transmitted in the related portion reply step; and an interlocking transmitting step for transmitting the additional portion to the second portable information device; , And an acquisition data acquisition step for acquiring, as acquisition data, replacement data output by the image forming apparatus or data obtained by changing a part of the replacement data for each of the plurality of portable information devices, and an AR tag from the acquisition data An AR tag extraction step for extracting data, an AR tag extracted from acquired data, and an acquired data A related part requesting step for transmitting a transmission request including a data to the server, a related part receiving step for receiving a related part associated with the AR tag from the server after transmitting the transmission request, and extracted from the acquired data A display control step of displaying an image of the received related portion at a position determined based on the position in the acquired data of the AR tag, and causing the second portable information device to execute the display control step. The display control step is received at a position determined based on the position in the acquired data of the AR tag extracted from the acquired data in response to receiving the additional part from the server after displaying the second related part. And an interlocking display control step for displaying an image of the additional portion.

  According to this aspect, it is possible to provide a linked display method capable of notifying a user who views other data of a change added to one data.

It is a figure which shows an example of the whole outline | summary of the augmented reality display system in embodiment of this invention. 1 is a perspective view showing an external appearance of an MFP in the present embodiment. 2 is a block diagram illustrating an example of a hardware configuration of an MFP according to the present embodiment. FIG. It is a block diagram which shows an example of the outline | summary of the hardware constitutions of the portable information device in this Embodiment. It is a block diagram which shows an example of the hardware constitutions of HMD in this Embodiment. It is a block diagram which shows an example of the hardware constitutions of the server in this Embodiment. 2 is a block diagram illustrating an example of an overview of functions of a CPU provided in an MFP. FIG. It is a figure which shows an example of a priority setting screen. It is a block diagram which shows an example of the outline | summary of the function of CPU with which a server is provided. It is a block diagram which shows an example of the outline | summary of the function of CPU with which the portable information device in this Embodiment is provided. It is a figure which shows an example of 1st original data. It is a figure which shows an example of 1st replacement data. It is a figure which shows an example of 2nd original data. It is a figure which shows an example of 2nd replacement data. It is a figure which shows an example of the paper when a character or a figure is added to the paper on which the image of the replacement data is formed. It is a 1st figure which shows an example of the image displayed on a portable information device. It is a 2nd figure which shows an example of the image displayed on a portable information device. It is a flowchart which shows an example of the flow of a replacement data generation process. It is a flowchart which shows an example of the flow of a registration information generation process. It is a flowchart which shows an example of the flow of an original correlation information production | generation process. It is a flowchart which shows an example of the flow of a virtual reality realization process. It is a flowchart which shows an example of the flow of an interlocking transmission process. It is a flowchart which shows an example of the flow of an augmented reality display process.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings. In the following description, the same parts are denoted by the same reference numerals. Their names and functions are also the same. Therefore, detailed description thereof will not be repeated.

  FIG. 1 is a diagram showing an example of an overall outline of an augmented reality display system according to an embodiment of the present invention. Referring to FIG. 1, an augmented reality display system 1 includes a wireless station 3, an MFP (Multi Function Peripheral) 100, portable information devices 200, 200 </ b> A, 200 </ b> B, and a personal computer (hereinafter referred to as a “computer”). 3, a head mounted display (hereinafter referred to as “HMD”) 400, and a server 500.

  MFP 100 functions as an image processing apparatus, and includes a document reading function for reading a document, an image forming function for forming an image on a recording medium such as paper based on image data, and a facsimile transmission / reception function for transmitting / receiving facsimile data. ing. In this embodiment, the MFP 100 is described as an example of an image processing apparatus. However, instead of the MFP 100, any apparatus having a function of processing an image may be a PC 300, a server 500, or the like. May be.

  The PC 300 is a general computer. The PC 300 is installed with a printer driver program for controlling the MFP 100.

  The portable information devices 200, 200A, and 200B are examples of information processing devices. The portable information devices 200, 200A, and 200B are used by being carried by users, such as PDAs (Personal Digital Assistants) and smartphones, and have a function of storing data and a wireless LAN function. In addition, portable information devices 200, 200A, and 200B can install application programs in the same manner as PC 300. Since the portable information devices 200, 200A, and 200B have the same hardware configuration and functions, the portable information device 200 will be described as an example here unless otherwise specified.

  The HMD 400 has a shape of glasses and is used by being worn by a user. The HMD 400 has an imaging function for imaging a subject, a display function for displaying an image on a lens portion of glasses, and a wireless LAN function. A user wearing the HMD 400 can visually recognize the subject through the lens and at the same time visually recognize the image displayed on the lens.

  The network 2 is a local area network (LAN), and the connection form may be wired or wireless. Further, the network 2 is not limited to a LAN, and may be a network using a public switched telephone network (Public Switched Telephone Networks). Further, the network 2 may be a wide area network (WAN) such as the Internet.

  The MFP 100 can transmit and receive data to and from the wireless station 3, the PC 300, and the server 500 via the network 2. The wireless station 3 is a relay device of the network 2 and communicates with the portable information devices 200, 200A, 200B and the HMD 400 having a wireless LAN communication function to connect the portable information devices 200, 200A, 200B and the HMD 400 to the network 2. To do. Therefore, each of portable information devices 200, 200 </ b> A, 200 </ b> B and HMD 400 can send and receive data to / from MFP 100, PC 300, and server 500 via wireless station 3 and network 2.

  FIG. 2 is a perspective view showing an appearance of the MFP according to the present embodiment. FIG. 3 is a block diagram illustrating an example of the hardware configuration of the MFP according to the present embodiment. 2 and 3, MFP 100 includes a main circuit 110, a document reading unit 130 for reading a document, an automatic document transport device 120 for transporting a document to document reading unit 130, and a document reading unit. An image is formed by an image forming unit 140 for forming an image on a sheet or the like based on image data output by the manuscript 130 reading and outputting a document, a paper feeding unit 150 for supplying the image to the image forming unit 140, and an image. A post-processing unit 155 that processes paper and an operation panel 160 as a user interface are included.

  The post-processing unit 155 executes a sorting process for sorting and discharging one or more sheets on which images are formed by the image forming unit 140, a punching process for punch hole processing, and a staple process for driving staple needles.

  The main circuit 110 includes a CPU 111, a communication interface (I / F) unit 112, a ROM 113, a RAM 114, a hard disk drive (HDD) 115 as a mass storage device, a facsimile unit 116, and a CD-ROM 118. External storage device 117. CPU 111 is connected to automatic document feeder 120, document reading unit 130, image forming unit 140, paper feeding unit 150, post-processing unit 155, and operation panel 160, and controls the entire MFP 100.

  The ROM 113 stores a program executed by the CPU 111 or data necessary for executing the program. The RAM 114 is used as a work area when the CPU 111 executes a program. The RAM 114 temporarily stores read data (image data) continuously sent from the document reading unit 130.

  Operation panel 160 is provided on the upper surface of MFP 100 and includes a display unit 161 and an operation unit 163. The display unit 161 is a display device such as a liquid crystal display (LCD) or an organic ELD (Electro-Luminescence Display), and displays an instruction menu for the user, information about acquired image data, and the like. The operation unit 163 includes a hard key unit 167 including a plurality of keys, and accepts input of various instructions, data such as characters and numbers by user operations corresponding to the keys. The operation unit 163 further includes a touch panel 165 provided on the display unit 161.

  Communication I / F unit 112 is an interface for connecting MFP 100 to network 2. CPU 111 communicates with portable information devices 200, 200 </ b> A, 200 </ b> B, PC 300, HMD 400, and server 500 via communication I / F unit 112 to transmit / receive data. The communication I / F unit 112 can communicate with a computer connected to the Internet via the network 2.

  The facsimile unit 116 is connected to the public switched telephone network (PSTN) and transmits facsimile data to the PSTN or receives facsimile data from the PSTN. The facsimile unit 116 stores the received facsimile data in the HDD 115 or outputs it to the image forming unit 140. The image forming unit 140 prints the facsimile data received by the facsimile unit 116 on a sheet. Further, the facsimile unit 116 converts the data stored in the HDD 115 into facsimile data, and transmits the facsimile data to a facsimile machine connected to the PSTN.

  The external storage device 117 is loaded with a CD-ROM (Compact Disk ROM) 118. The CPU 111 can access the CD-ROM 118 via the external storage device 117. The CPU 111 loads the program recorded on the CD-ROM 118 attached to the external storage device 117 to the RAM 114 and executes it. The medium for storing the program executed by the CPU 111 is not limited to the CD-ROM 118, but an optical disc (MO (Magnetic Optical Disc) / MD (Mini Disc) / DVD (Digital Versatile Disc)), IC card, optical card, It may be a semiconductor memory such as a mask ROM, EPROM (Erasable Programmable ROM), or EEPROM (Electrically EPROM).

  Further, the program executed by the CPU 111 is not limited to the program recorded on the CD-ROM 118, and the program stored in the HDD 115 may be loaded into the RAM 114 and executed. In this case, another computer connected to the network 2 may rewrite the program stored in the HDD 115 of the MFP 100, or may write a new program. Further, MFP 100 may download a program from another computer connected to network 2 and store the program in HDD 115. The program here includes not only a program directly executable by the CPU 111 but also a source program, a compressed program, an encrypted program, and the like.

  In the augmented reality display system 1, data is transmitted and received between the MFP 100, the portable information devices 200, 200 </ b> A, 200 </ b> B, the PC 300, the HDM 400, and the server 500. As a protocol for transmitting and receiving data, any protocol can be used as long as the transmission source can be specified in the receiving apparatus. As a protocol for transmitting and receiving data, for example, HTTP (Hyper Text Transfer Protocol), FTP (File Transfer Protocol), SMPT (Simple Mail Transfer Protocol), POP (Post Office Protocol), and the like are used.

  FIG. 4 is a block diagram illustrating an example of an outline of a hardware configuration of the portable information device according to the present embodiment. Referring to FIG. 4, portable information device 200 is connected to CPU 201 for controlling overall portable information device 200, camera 202, flash memory 203 for storing data in a nonvolatile manner, and call unit 205. A wireless communication unit 204; a display unit 206 that displays information; an operation unit 207 that accepts user operations; a wireless LAN I / F 208; a position detection unit 209; an orientation detection unit 210; and an external storage device 211. Including.

  The wireless communication unit 204 performs wireless communication with a mobile phone base station connected to a telephone communication network. The wireless communication unit 204 connects the portable information device 200 to a telephone communication network and enables a call using the call unit 205. The radio communication unit 204 decodes a voice signal obtained by demodulating the radio signal received from the mobile phone base station and outputs the decoded signal to the call unit 205. Further, the wireless communication unit 204 encodes the voice input from the call unit 205 and transmits it to the mobile phone base station. The call unit 205 includes a microphone and a speaker, outputs the voice input from the wireless communication unit 204 from the speaker, and outputs the voice input from the microphone to the wireless communication unit 204. Further, the wireless communication unit 204 is controlled by the CPU 201, connects the portable information device 200 to an electronic mail server, and transmits and receives electronic mail.

  The camera 202 includes a lens and a photoelectric conversion element. The light focused by the lens is imaged on the photoelectric conversion element. The photoelectric conversion element photoelectrically converts the received light and outputs image data to the CPU 201. The photoelectric conversion element is a CMOS (Complementary Metal Oxide Semiconductor) sensor, a CCD (Charge Coupled Device) sensor, or the like.

  The display unit 206 is a display device such as a liquid crystal display (LCD) or an organic ELD, and displays an instruction menu for a user, information about acquired image data, and the like. The operation unit 207 includes a plurality of keys, and accepts input of various instructions, data such as characters and numbers by user operations corresponding to the keys.

  The wireless LAN I / F 208 is an interface for communicating with the wireless station 3 and connecting the portable information device 200 to the network 2. By registering the IP (Internet Protocol) addresses of HMD 200, server 500, MFP 100, and PC 300 in portable information device 200, portable information device 200 can communicate with HMD 400, server 500, MFP 100, and PC 300. Data transmission / reception becomes possible. In this embodiment, a case where portable information device 200 communicates with MFP 100, PC 300, HMD 400, and server 500 using wireless LAN I / F 208 will be described as an example, but communication is performed using another communication method. You may do it. Specifically, when portable information device 200, MFP 100, PC 300, HMD 400, and server 500 are equipped with a short-range wireless device such as Bluetooth (registered trademark), for example, portable information device 200 is MFP 100, PC 300, HMD 400. , One-to-one communication with any of the servers 500 may be performed. In addition, the portable information device 200 is connected to any one of the MFP 100, the PC 300, the HMD 400, and the server 500 by a wire such as a USB (Universal Serial Bus) cable, and the portable information device 200 is connected to the MFP 100, the PC 300, the HMD 400, the server. You may make it communicate with either of 500 one to one.

  The flash memory 203 stores a program executed by the CPU 201 or data necessary for executing the program. The CPU 201 loads a program recorded in the flash memory 203 to a RAM included in the CPU 201 and executes it.

  The position detection unit 209 detects the current position of the portable information device 200. Specifically, the position detection unit 209 is a global positioning system receiver, and measures the current position by receiving radio waves from a plurality of GPS satellites. The position detection unit 209 outputs a value indicating the measured current position, for example, latitude and longitude to the CPU 201.

  The direction detection unit 210 detects the direction in which the camera 202 of the portable information device 200 faces. Specifically, the orientation detection unit 210 is a geomagnetic sensor that measures geomagnetism. The direction detection unit 210 outputs the detected direction to the CPU 201. The direction detection unit 210 is installed in the portable information device 200 so as to be able to detect the imaging direction of the camera 202, in other words, the direction in which the optical axis of the lens of the camera 202 faces. The direction detection unit 210 is not limited to a geomagnetic sensor, and a gyro sensor or the like may be used.

  The external storage device 211 is detachable from the portable information device 200, and a CD-ROM 211A storing a remote operation program can be attached thereto. The CPU 201 can access the CD-ROM 211A via the external storage device 211. The CPU 201 can load a remote operation program recorded on the CD-ROM 211A attached to the external storage device 211 into a RAM included in the CPU 201 and execute it.

  Note that the program recorded in the flash memory 203 or the CD-ROM 211A has been described as a program executed by the CPU 201, but a program in which another computer connected to the network 2 rewrites the program stored in the flash memory 203, Alternatively, it may be a new program written additionally. Further, the portable information device 200 may be a program downloaded from another computer connected to the network 2. The program here includes not only a program directly executable by the CPU 201 but also a source program, a compressed program, an encrypted program, and the like.

  The medium for storing the program executed by the CPU 201 is not limited to the CD-ROM 211A, and may be a semiconductor memory such as an optical disk (MO / MD / DVD), an IC card, an optical card, a mask ROM, an EPROM, or an EEPROM. Good.

  FIG. 5 is a block diagram showing an example of the hardware configuration of the HMD in the present embodiment. Referring to FIG. 5, HMD 400 in the present embodiment includes a CPU 401 for controlling the entire HMD 400, a camera 402, a flash memory 403 for storing data in a nonvolatile manner, and a display unit 404 for displaying information. And wireless LAN I / F 405.

  The camera 402 includes a lens and a photoelectric conversion element. The light focused by the lens is imaged on the photoelectric conversion element. The photoelectric conversion element photoelectrically converts the received light and outputs image data to the CPU 401. The photoelectric conversion element is a CMOS (Complementary Metal Oxide Semiconductor) sensor, a CCD (Charge Coupled Device) sensor, or the like. Although an example in which the camera 402 captures a still image is described here, a video camera that captures a moving image may be used. When a video camera is used, one of a plurality of frames included in the moving image may be handled as a still image.

  The display unit 404 is a liquid crystal display (LCD) made of a transparent member, and is embedded in the lens portion of the HMD 400. The display surface of the display unit 404 is arranged at a position determined with reference to the optical axis of the camera 402. Specifically, the imaging range of the camera 402 is set to be the same as the field of view when the user wears the HMD 400. For this reason, the image obtained by capturing with the camera 402 is the same as the image in the field of view actually viewed by the user. For this reason, the position in the user's field of view can be specified from the image output by the camera 402. The display surface of the display unit 404 is set to be the same as the field of view when the user wears the HMD 400 in the lens portion. For this reason, an image can be displayed at an arbitrary position in the user's field of view.

  The wireless LAN I / F 405 is an interface for communicating with the wireless station 3 and connecting the HMD 400 to the network 2. By registering the IP (Internet Protocol) address of the server 500 in the flash memory 403, the HMD 400 can communicate with the server 500 and transmit / receive data. In this embodiment, the case where the HMD 400 communicates with the server 500 using the wireless LAN I / F 405 will be described as an example, but communication may be performed using another communication method. Specifically, when the HMD 400 and the server 500 are mounted with a short-range wireless device such as Bluetooth (registered trademark), for example, the HMD 400 may communicate with the server 500 on a one-to-one basis.

  The flash memory 403 stores a program executed by the CPU 401 or data necessary for executing the program. The CPU 401 loads the program recorded in the flash memory 403 to the RAM provided in the CPU 401 and executes it.

  FIG. 6 is a block diagram illustrating an example of a hardware configuration of the server according to the present embodiment. Referring to FIG. 6, server 500 has a CPU 501 for controlling server 500 as a whole, ROM 502 for storing a program to be executed by CPU 501, RAM 503 used as a work area for CPU 501, and non-volatile storage of data. HDD 504 for storing information, a communication unit 505 for connecting CPU 501 to network 2, a display unit 506 for displaying information, an operation unit 507 for accepting input of user operations, and an external storage device 508.

  The CPU 501 loads a program stored in the ROM 502 or the HDD 504 to the RAM 502 and executes it.

  The external storage device 508 can be loaded with a CD-ROM 508A storing a program. The CPU 501 can access the CD-ROM 508A via the external storage device 508. The CPU 501 can load a program recorded on the CD-ROM 508A into the RAM 502 and execute it.

  In addition, although the program recorded on ROM502, HDD504, or CD-ROM508A was demonstrated as a program which CPU501 performs, the other computer connected to the network 2 rewritten the program memorize | stored in HDD504, or It may be a new program written additionally. Further, the server 500 may be a program downloaded from another computer connected to the network 2. The program here includes not only a program directly executable by the CPU 501 but also a source program, a compressed program, an encrypted program, and the like.

  The medium for storing the program executed by the CPU 501 is not limited to the CD-ROM 508A, and may be a semiconductor memory such as an optical disk (MO / MD / DVD), IC card, optical card, mask ROM, EPROM, or EEPROM. Good.

  FIG. 7 is a block diagram illustrating an example of an overview of the functions of the CPU provided in the MFP. The functions shown in FIG. 7 are functions formed in CPU 111 when CPU 111 provided in MFP 100 executes a program stored in ROM 113, HDD 115, or CD-ROM 118. Referring to FIG. 7, CPU 111 includes a data acquisition unit 51 that acquires original data, a replacement unit 53 that converts original data into replacement data, an AR tag registration unit 55, and an output control unit 57 that outputs replacement data. A user authentication unit 59 that authenticates a user who operates MFP 100, a data association registration unit 61 that associates original data with other original data, and a priority registration unit 63.

  The data acquisition unit 51 acquires original data, and outputs the acquired original data to the replacement unit 53, the AR tag registration unit 55, and the data association registration unit 61. The original data may be data stored in the HDD 115, or data stored in the PC 300 or the server 500. The format of the data is not limited, but may be application data generated by the PC 300 executing the application program or image data. The application program is not particularly limited, and includes, for example, a document editing program for generating a document, an image editing program for editing an image, a spreadsheet program, a program for generating presentation data, a database program, and the like.

  The replacement unit 53 generates replacement data in which the part specified by the user in the original data is used as a related part, and the related part is replaced with an AR (Augmented Reality) tag. Since the related portion is a portion specified by the user in the original data, it includes characters, graphics or photographs, or a combination thereof, although it varies depending on the original data. The AR tag is a predetermined image, and can be distinguished from other images and other AR tags. Note that the AR tag may be a two-dimensional barcode such as a QR code (registered trademark) (Quick Response Code). The related part includes position information indicating a relative position of the related part with reference to the AR tag arranged in the replacement data.

  For example, if the user designates a part of the original data as the related part, the original data is converted into replacement data. Even if the replacement data is output, the related portion is replaced by the AR tag, and therefore the related portion cannot be known from the replacement data. By specifying a part of the original data as the related part, the user can prevent another user who sees the replacement data from understanding the related part. The replacement unit 53 outputs the replacement data to the output control unit 57 and outputs a set of the related part, the AR tag in which the related part is replaced, and the replacement data to the AR tag registration unit 55.

  User authentication unit 59 authenticates a user who operates MFP 100. If the login screen is displayed on the display unit 161 and the user identification information and password input by the user to the operation unit 163 are received and they match the combination of the user identification information and password stored in the HDD 115 in advance, Authenticate the user. The user authentication unit 59 outputs user identification information of the authenticated user to the output control unit 57.

  Output control unit 57 outputs the replacement data input from replacement unit 53 to an output destination determined by an operation input to operation unit 163 by a user operating MFP 100. The output control unit 57 outputs an output completion signal including the user identification information input from the user authentication unit 59 to the data association registration unit 61 in response to outputting the replacement data. The output control unit 57 includes an image formation control unit 65, a data transmission unit 67, and a facsimile transmission control unit 69. The image formation control unit 65 controls the image formation unit 140 to form an image of replacement data on a sheet.

  The data transmission unit 67 transmits the replacement data to another computer via the communication I / F unit 112. Although the transmission protocol is not particularly limited, for example, file transfer by FTP (File Transfer Protocol), transmission of e-mail with attached replacement data or changed replacement data. Here, as an example of the case where the data transmission unit 67 transmits the replacement data, a case where the replacement data is transmitted to the portable information device 200 will be described as an example.

  The facsimile transmission control unit 69 controls the facsimile unit 116 to transmit the replacement data to another facsimile apparatus. In a facsimile apparatus that receives replacement data, an image of replacement data is formed on a sheet.

  The AR tag registration unit 55 receives original data from the data acquisition unit 51, and receives a set of related portions, AR tags, and replacement data from the replacement unit 361. The AR tag registration unit 55 transmits registration information including original identification information for identifying original data, a related portion, an AR tag, and replacement data to the server 500. Specifically, the registration information is transmitted to the server 500 via the communication I / F unit 112.

  The data association registration unit 61 receives original data from the data acquisition unit 51. The data association registration unit 61 associates the original data with another original data different from the original data. Specifically, original association information including two or more pieces of original identification information for identifying two or more pieces of original data is transmitted to server 500 via communication I / F unit 112. Further, the data association registration unit 61 outputs the original association information to the priority registration unit 63. The data association registration unit 61 includes an identical data association unit 71, an identical user association unit 73, a candidate data acquisition unit 75, and an operation association unit 77.

  When the original data input from the data acquisition unit 51 is one page of data consisting of a plurality of pages, the same data association unit 71 converts the original data input from the data acquisition unit 51 into data including the original data. It associates with the original data of one or more other pages among the plurality of pages. In other words, the same data association unit 71 associates a plurality of original data, which are a plurality of pages included in one data, with each other. For example, when the data consists of the first page, the second page, and the third page, the original data of the first page and the original data of the second page are associated with each other, and the original data of the first page and the original data of the third page And the original data on the second page and the original data on the third page.

  The same user association unit 73 associates a plurality of pieces of original data respectively corresponding to a plurality of pieces of replacement data output based on an instruction from the same user by the output control unit 57. After the original data is input from the data acquisition unit 51 and the output completion signal including the user identification information is input from the output control unit 57, the same user association unit 73 receives the user identification information included in the output completion signal. Based on the operation by the user, it is determined that replacement data obtained by replacing the original data has been output. The same user association unit 73 stores history information including original identification information for identifying original data and user identification information, and in response to an output completion signal being input from the output control unit 57. The original data specified by the original identification information included in the history information including the user identification information included in the output completion signal with reference to the history information, and the original data related to the original data input from the data acquisition unit 51 To decide. For example, when replacement data in which original data A is replaced based on an operation by user A is formed and replacement data in which original data B is replaced based on an operation by user A is formed, original data A and original data Associate with B. Further, the same user association unit 73 associates a plurality of original data respectively corresponding to a plurality of replacement data output based on instructions from a plurality of users belonging to the same group by the output control unit 57. For example, when user A and user B belong to the same group, replacement data in which original data A is replaced based on an operation by user A is formed as an image, and replacement data in which original data B is replaced based on an operation by user B When the image is formed, the original data A and the original data B are associated with each other.

  The candidate data acquisition unit 75 acquires one or more original data as candidates from the server 500. Since the registration information is transmitted to the server 500 by the AR tag registration unit 55, the server 500 stores original data included in the registration information. The candidate data acquisition unit 75 requests the server 500 to transmit candidate data, and acquires one or more original data stored in the server 500 as candidate data. The candidate data acquisition unit 75 outputs the acquired one or more candidate data to the operation association unit 77.

  The operation association unit 77 associates the original data input from the data acquisition unit 51 with candidate data selected by the user from one or more candidate data input from the candidate data acquisition unit 75.

  The priority registration unit 63 receives the original association information from the data association registration unit 61. The priority registration unit 63 gives a priority to the original data specified by two or more pieces of original identification information included in the original association information. The priority determines relative superiority or inferiority with respect to the two original data. For example, with respect to two original data, there are three ways: one has a higher priority than the other, one has a lower priority than the other, and both have the same priority. When the original data is 3 or more, the relationship between the priorities of the two sets of original data is the same as the above three types. The priority registration unit 63 transmits priority information including original identification information and priority of each of two or more original data to the server 500 via the communication I / F unit 112.

  The priority registration unit 63 displays a priority setting screen on the display unit 163, and sets the priority according to an operation input by the user to the operation unit 163.

  FIG. 8 is a diagram illustrating an example of a priority setting screen. The priority setting screen 700 illustrated in FIG. 8 illustrates an example of a case where the priority setting screen 700 is displayed when the first original data and the second original data are associated with each other. Here, the original identification information “document X” of the first original data is used, and the original identification information “document Y” of the second original data is used.

  Referring to FIG. 8, priority setting screen 700 includes first original data thumbnail 701, second original data thumbnail 702, selection buttons 711 to 714, and determination button 715. The thumbnail is a reduced image of the data image.

  The original identification information “document X” of the first original data is arranged in the vicinity of the thumbnail 701 of the first original data, and the original identification information “document of the second original data in the vicinity of the thumbnail 702 of the second original data. Y "is arranged.

  Any one of the selection buttons 711 to 714 can be selected. The selection button 711 is a button for setting the priority of the first original data and the priority of the second original data to be the same. The selection button 712 is a button for setting the priority of the first original data to a value higher than the priority of the second original data. The selection button 713 is a button for setting the priority of the first original data to a value lower than the priority of the second original data. The selection button 714 is a button for not setting the priority of the first original data and the priority of the second original data.

  The determination button 715 is a button for inputting an operation for determining priority. When the determination button 715 is instructed in a state where any of the selection buttons 711 to 714 is selected, the priorities of the first original data and the second original data are determined.

  FIG. 9 is a block diagram illustrating an example of an overview of the functions of the CPU provided in the server. The functions shown in FIG. 9 are formed in the CPU 501 when the CPU 501 provided in the server 500 loads the program stored in the ROM 502 or the HDD 504 to the RAM 502 and executes it. 9, CPU 501 provided in server 500 includes an AR tag association unit 551, a registration information extraction unit 553, a transmission request reception unit 555, an original association unit 557, an associated partial reply unit 559, A related original identification unit 561, an additional part extraction unit 563, a device identification unit 565, an interlocking transmission unit 567, a priority setting unit 569, and a user identification unit 571 are included.

  AR tag associating unit 551 receives registration information from MFP 100, associates original data included in registration information, AR tag, related portion, and replacement data with the received registration information as registration information extraction unit It outputs to 553. The AR tag association unit 551 controls the communication unit 505, receives registration information transmitted from the MFP 100, and stores the received registration information in the HDD 504, whereby the original data included in the registration information, the AR tag And the related part and the replacement data are associated with each other, and the registration information is output to the registration information extraction unit 553.

  In the transmission request receiving unit 555, the communication unit 505 receives a transmission request from any one of the portable information devices 200, 200 </ b> A, 200 </ b> B, or the HMD 400. Here, a case where a transmission request is received from portable information device 200 will be described as an example. The transmission request includes an AR tag and acquired data. The acquired data is data acquired by the portable information device 200 that has transmitted the transmission request. The acquired data is data obtained, for example, by the portable information device 200 capturing an image of a sheet on which the MFP 100 forms an image of replacement data using the camera 202 as a subject. The acquired data is replacement data when portable information device 200 receives replacement data transmitted by MFP 100. In response to receiving the transmission request, the transmission request receiving unit 55 outputs the AR tag included in the transmission request to the registration information extracting unit 553, and outputs the acquired data included in the transmission request to the additional portion extracting unit 563. , Device identification information for identifying the device that has transmitted the transmission request, here, device identification information of portable information device 200 is output to related partial reply section 559.

  The registration information extraction unit 553 receives the AR tag input from the transmission request receiving unit 555 in response to the input of the AR tag from the transmission request receiving unit 555, the original data associated by the AR tag association unit 551, Determine relevant parts and replacement data. Specifically, registration information including the same AR tag as the AR tag input from the transmission request receiving unit 555 is extracted from the registration information stored in the HDD 504, and the original identification information included in the extracted registration information Get part and replacement data. The registration information extraction unit 553 outputs the pair of the original identification information and the related part to the related part reply unit 559, outputs the original identification information to the related original specifying unit 561, and adds the pair of the original identification information and the replacement data The data is output to the partial extraction unit 563.

  The related part reply unit 559 receives a set of the original identification information and the related part from the registered information extraction unit 553 and transmits a transmission request from the transmission request receiving unit 555, here, the device identification information of the portable information device 200 Is entered. The related part reply unit 559 controls the communication unit 505 to transmit the related part to the portable information device 200 specified by the device identification information. In addition, the related part reply unit 559 outputs a set of original identification information and device identification information to the device identification unit 565.

  Original association unit 557 receives original association information from MFP 100, associates two or more original data respectively specified by two or more original identification information included in the received original association information, and A set of original association information is output to the related original identification unit 561. The original association unit 557 controls the communication unit 505 to receive the original association information transmitted from the MFP 100 and store the received original association information in the HDD 504, so that 2 included in the original association information. Two or more original data respectively specified by the above original identification information are associated with each other, and the original association information is output to the related original specifying unit 561.

  The related original identification unit 561 receives original identification information from the registered information extraction unit 553 and receives original related information from the original association unit 557. The original related information includes two or more pieces of original identification information. The related original specifying unit 561 specifies the original data specified by the original identification information input from the registered information extraction unit 553 and the original data related by the original related information as related original data. The related original identification unit 561 extracts original related information including original identification information input from the registration information extraction unit 553 from the original related information stored in the HDD 504, and is included in the extracted original related information 2. The above original identification information is acquired. The related original specifying unit 561 specifies one or more pieces of original identification information excluding the original identification information input from the registered information extraction unit 553 among the two or more pieces of acquired original identification information as the original identification information of the related original data. To the unit 565.

  The device specifying unit 565 receives a set of original identification information and device identification information from the related part reply unit 559, and receives original identification information of one or more related original data from the related original specifying unit 561. When the combination of the original identification information and the device identification information is input from the related part reply unit 559, the device specifying unit 565 stores them in the HDD 504. When the original identification information of one or more related original data is input from the related original specifying unit 561, the device specifying unit 565 refers to the set of the original identification information and the device identification information stored in the HDD 504 and refers to the related original. The device identification information paired with the original identification information of the related original data input from the specifying unit 561 is specified. Specifically, the device specifying unit 565 has the same original identification as the original identification information of the related original data input from the related original specifying unit 561 from the set of the original identification information and the device identification information stored in the HDD 504. A set including information is extracted, and device identification information included in the extracted set is specified. When a plurality of pieces of original identification information are input from the related original identification unit 561, the device identification unit 565 identifies device identification information paired with the original identification information for each of the plurality of original identification information. The device specifying unit 565 outputs the pair of the original identification information of the related original data and the device identification information paired therewith to the interlocking transmission unit 567 and the user specifying unit 571.

  The additional portion extraction unit 563 receives the combination of original identification information and replacement data of the original data from the registration information extraction unit 553, and receives the acquired data from the transmission request reception unit 555. The additional portion extraction unit 563 extracts a portion that exists in the acquired data but does not exist in the replacement data as an additional portion. What is necessary is just to extract an additional part from the difference of replacement data and acquisition data. The additional part extraction unit 56 outputs a pair of the extracted additional part and the original identification information of the original data input from the registration information extraction unit 553 to the interlocking transmission unit 567. The additional portion includes image information included in the acquired data and position information indicating the relative position of the additional portion based on the AR tag included in the acquired data.

  A priority setting unit 569 receives priority information from the MFP 100, sets priority for each of two or more original data respectively specified by two or more original identification information included in the priority information, and sets priority information. Is output to the interlocking transmission unit 567. The priority information defines original identification information and priority for each of two or more original data. The priority setting unit 569 controls the communication unit 505, receives priority information transmitted from the MFP 100, and stores the received priority information in the HDD 504, whereby two or more originals included in the priority information are stored. Priorities are set for each of two or more original data specified by the identification information, and the priority information is output to the interlocking transmission unit 567.

  The user specifying unit 571 receives a set of original identification information and device identification information of related original data from the device specifying unit 565. The user specifying unit 571 specifies a user who operates the device identification information input from the device specifying unit 565. The HDD 504 stores in advance the user identification information of the user, the device identification information of the device assigned to the user, and the confidential level assigned to the user. In response to the input of the device identification information from the device identification unit 565, the user identification unit 571 identifies the user of the user identification information paired with the device identification information. The user specifying unit 571 outputs the specified user security level to the interlocking transmission unit 567.

  The interlocking transmission unit 567 receives the device identification information from the device identification unit 565, and receives the combination of the original identification information of the original data and the additional portion from the additional portion extraction unit 563. The interlocking transmission unit 567 controls the communication unit 505 to transmit the additional part to the device specified by the device identification information input from the device specifying unit 565.

  The interlocking transmission unit 567 includes a rank comparison unit 581, a security level determination unit 583, and a security level comparison unit 585. The rank comparison unit 581 receives a set of the original identification information of the related original data and the device identification information from the device specifying unit 565, and receives a set of the additional portion and the original identification information of the original data from the additional portion extraction unit 563. The Here, the original identification information of the original data input from the additional portion extraction unit 563 is referred to as first original identification information, and the original identification information of the related original data input from the device specifying unit 565 is the second original identification information. That's it. The rank comparison unit 581 refers to the priority set for each original data by the priority setting unit 569, and uses the priority of the original data specified by the first original identification information and the second original identification information. The priority of the related original data identified is compared. The interlocking transmission unit 567 sets the second priority on the condition that the priority of the original data specified by the first original identification information is equal to or higher than the priority of the related original data specified by the second original identification information. The additional part is transmitted to the device specified by the device identification information paired with the original identification information. When the priority of the original data specified by the first original identification information is lower than the priority of the original data specified by the second original identification information, the interlocking transmission unit 567 combines with the second original identification information. The additional part is not transmitted to the device specified by the device identification information. Also, the interlocking transmission unit 567 sets the priority of the original data specified by the first original identification information and the priority of the related original data specified by the second original identification information by the priority setting unit 569. If not, the additional part is not transmitted to the device specified by the device identification information paired with the second original identification information.

  The security level determination unit 583 analyzes the additional part input from the additional part extraction unit 563 and determines the security level for the additional part. The security level determination unit 583 extracts character information included in the image of the additional portion by recognizing the character of the image of the additional portion. Then, the security level determination unit 583 determines the security level by comparing the extracted character information with a predetermined character string. The predetermined character string is a character string determined for each security level. For example, the character string “confidential” for the highest confidential level “1”, the character string “confidential” for the confidential level “2” lower than the confidential level “1”, and the like. Further, the character string “Project A” which is the project name for the highest confidential level “1”, the character string “Project B” which is the project name for the confidential level “2” lower than the confidential level “1”, and the like. is there. The security level determination unit 583 outputs the security level determined for the added portion to the security level comparison unit 585.

  The security level comparison unit 585 receives the security level for the additional portion from the security level determination unit 583 and receives the user security level from the user identification unit 571. The confidential level comparison unit 585 compares the confidential level for the added portion with the confidential level of the user. The interlocking transmission unit 567 transmits the additional part to the device specified by the device identification information input from the device specifying unit 565 on the condition that the user's security level is equal to or higher than the security level for the additional part. When the confidential level of the user is lower than the confidential level for the additional portion, the interlocking transmission unit 567 does not transmit the additional portion.

  FIG. 10 is a block diagram illustrating an example of an outline of functions of a CPU provided in the portable information device according to the present embodiment. The functions shown in FIG. 10 are functions formed in the CPU 201 when the CPU 201 included in the portable information device 200 executes the augmented reality display program stored in the flash memory 203. Referring to FIG. 10, the CPU 201 obtains an acquisition data acquisition unit 253, an AR tag extraction unit 255, a related part request unit 257 that requests the server 500 to transmit a related part, and a relation that receives a related part from the server 500. A partial receiving unit 259 and a display control unit 251 for controlling the display unit 206 are included.

  The acquisition data acquisition unit 253 acquires data. Here, the data acquired by the acquisition data acquisition unit 253 is referred to as acquisition data. The acquisition data acquisition unit 253 outputs the acquired acquisition data to the AR tag extraction unit 255, the related part request unit 257, and the display control unit 251.

  Acquired data acquisition unit 253 controls wireless LAN I / F 208 and receives data transmitted by MFP 100 as acquired data. When MFP 100 transmits the acquired data by e-mail, the MFP 100 receives the e-mail and acquires the acquired data included in the received e-mail. The acquired data includes data obtained by changing the data received by the wireless LAN I / F 208 from the MFP 100. For example, when the user who operates MFP 100 edits data received from MFP 100 using an application program, acquisition data acquisition unit 253 acquires data after editing data received from MFP 100 as acquisition data. To do. The acquired data acquisition unit 253 includes an imaging control unit 261 that controls the camera 202. The imaging control unit 261 controls the camera 202 and acquires a captured image output by the camera 202 imaging the subject. When the camera 202 captures and outputs a sheet on which a replacement data image is formed by the MFP 100 as a subject, the acquisition data acquisition unit 253 acquires a captured image output by the camera 202 imaging the sheet as acquisition data. . The user may draw handwritten characters, figures, and the like on the sheet on which the replacement data image is formed by the MFP 100. In this case, the captured image that the camera 202 captures and outputs the subject includes images of handwritten characters and graphics in addition to the replacement data image.

  The AR tag extraction unit 255 extracts an AR tag from the acquisition data input from the acquisition data acquisition unit 253. The AR tag extraction unit 255 outputs the extracted AR tag to the related part request unit 257.

  The related part request unit 257 receives the acquisition data from the acquisition data acquisition unit 253 and the AR tag from the AR tag extraction unit 255. The related part request unit 257 controls the wireless LAN I / F 208 and transmits a transmission request to the server 500. The transmission request includes acquired data and an AR tag. The IP address of server 500 may be stored in advance in flash memory 203 of portable information device 200. When receiving the transmission request, the server 500 that receives the transmission request returns a related part associated with the AR tag included in the transmission request.

  The related part receiving unit 259 controls the wireless LAN I / F 208 to receive the related part from the server 500. The related part receiving unit 259 outputs the related part to the display control unit 251. The related part includes an image and position information indicating a relative position of the related part with reference to the AR tag.

  The display control unit 251 displays an image of the acquisition data on the display unit 206 in response to the acquisition data input from the acquisition data acquisition unit 253. When the related part is input from the related part receiving unit 259, the display control unit 251 displays an image of the related part on the display unit 206. Based on the position information included in the related portion, the display control unit 251 specifies the position where the image of the related portion is arranged based on the AR tag included in the image of the acquired data displayed on the display unit 206. The display control unit 251 superimposes and displays the image of the related portion at a position specified with reference to the AR tag in the acquired data image.

  The wireless LAN I / F 208 may receive the additional part from the server 500 after the related part is displayed by the display control unit 251. The display control unit 251 includes an interlocking display control unit 263. When receiving the additional portion from the server 500, the interlocking display control unit 263 displays an image of the additional portion on the display unit 206. The additional portion includes an image and position information indicating a relative position of the additional portion with reference to the AR tag. Based on the position information included in the additional portion, the interlocking display control unit 263 specifies the position where the image of the additional portion is to be arranged with reference to the AR tag included in the acquired data image displayed on the display unit 206. . The interlocking display control unit 263 superimposes and displays the image of the additional portion at the position specified on the basis of the AR tag in the acquired data image.

  Here, a case where MFP 100 converts data received from PC 300 into replacement data and forms an image will be described as an example. The data that MFP 100 receives from PC 300 includes first original data that is the first page and second original data that is the second page. In the first original data and the second original data, the case where the priority of the first original data is set higher than the priority of the second original data will be described as an example.

  FIG. 11 is a diagram illustrating an example of the first original data. Referring to FIG. 11, first original data 610 includes a character string 611 of “Confidential” and an area 613 where a bar graph is represented. Here, a case where the area 613 is designated as a related part by the user will be described as an example.

  FIG. 12 is a diagram illustrating an example of the first replacement data. The first replacement data is data obtained by replacing the related portion 613 of the first original data 610 with an AR tag. Referring to FIG. 12, first replacement data 610A includes a character string 611 of “Confidential” and an AR tag 613A.

  FIG. 13 is a diagram illustrating an example of second original data. Referring to FIG. 13, second original data 620 includes a character string 621 of “Confidential” and an area 623 where a band graph is represented. Here, a case where the area 623 is designated as a related part by the user will be described as an example.

  FIG. 14 is a diagram illustrating an example of second replacement data. The second replacement data is data in which the related portion 623 of the second original data 620 is replaced with an AR tag. Referring to FIG. 14, first replacement data 620A includes a character string 621 of “Confidential” and an AR tag 623A.

  When MFP 100 forms the image of first replacement data 610A on a sheet, MFP 100 includes first original identification information of first original data 610, AR tag 613A, related portion 613, and replacement data 610A. The registration information is transmitted to the server 500. When MFP 100 forms the image of second replacement data 620A on a sheet, MFP 100 includes second original identification information of first original data 620, AR tag 623A, related portion 623, and replacement data 620A. The second registration information is transmitted to the server 500. Further, since the first original data and the second original data are the first page and the second page included in the same data, the MFP 100 includes the first original identification information of the first original data, the second original data, and the second page. Original association information including the second original identification information of the original data is transmitted to the server 500. Further, since the priority of the first original data is set by the user higher than the priority of the second original data, the MFP 100 can identify the original identification information of each of the first original data and the second original data, Priority information in which the priority of one original data is higher than the second original data priority is transmitted to the server 500. Server 500 stores first registration information, second registration information, original association information, and priority information in HDD 504.

  At this stage, when the user A operates the portable information device 200 to capture a sheet on which the image of the first replacement data 610A is formed by the camera 202, the portable information device 200 uses the first replacement data 610A. A captured image obtained by capturing an image with the camera 202 is used as first acquired data, and a first transmission request including the AR tag 613A extracted from the first acquired data and the first acquired data is sent to the server 500. Send to. The server 500 extracts the first registration information including the same AR tag 613A as the AR tag 613A included in the first transmission request, and returns the related portion 613 included in the first registration information to the portable information device 200. . Therefore, the portable information device 200 displays the related portion 613 returned from the server 500 at a position determined based on the AR tag 613A in the first acquired data (captured image). Therefore, the same image as the first original data 610 shown in FIG. 11 is displayed on the display unit 206 of the portable information device 200.

  On the other hand, when the user B operates the portable information device 200A to capture a sheet on which the image of the second replacement data 620A is formed by the camera 202, the portable information device 200A displays the image of the second replacement data 620A. A captured image obtained by capturing with the camera 202 is used as second acquired data, and a second transmission request including the AR tag 623A extracted from the second acquired data and the second acquired data is transmitted to the server 500. To do. The server 500 extracts the second registration information including the same AR tag 623A as the AR tag 623A included in the second transmission request, and returns the related portion 623 included in the second registration information to the portable information device 200A. . Therefore, portable information device 200A displays related portion 623 returned from server 500 at a position determined based on AR tag 623A in the second acquired data (captured image). Therefore, the same image as the second original data 620 shown in FIG. 13 is displayed on the display unit 206 of the portable information device 200A.

  Next, a case will be described as an example where user A draws characters or figures by handwriting on a sheet on which an image of the first replacement data 610A is formed.

  FIG. 15 is a diagram illustrating an example of a sheet when characters or figures are added to the sheet on which the replacement data image is formed. Here, a case where the user A draws a character by handwriting on a sheet on which an image of the first replacement data 610A is formed is shown. Referring to FIG. 15, paper 610 </ b> B includes a character string 611 of “Confidential”, a related portion 613, and a region 615 where handwritten characters are represented.

  When the user A operates the portable information device 200 and images the paper 610B with the camera 202, the portable information device 200 uses the captured image obtained by imaging the paper with the camera 202 as the third acquisition data, and The third transmission request including the AR tag 613A extracted from the third acquired data and the third acquired data is transmitted to the server 500. The server 500 extracts the first registration information including the same AR tag 613A as the AR tag 613A included in the third transmission request, and returns the related portion 613 included in the first registration information to the portable information device 200. . Therefore, portable information device 200 displays related portion 613 returned from server 500 at a position determined based on AR tag 613A in the third acquired data (captured image). FIG. 16 shows an example of an image displayed on the portable information device 200.

  FIG. 16 is a first diagram illustrating an example of an image displayed on the portable information device. Referring to FIG. 16, an image 610 </ b> C includes a character string 611 of “Confidential”, a related portion 613, and a region 615 where handwritten characters are represented.

  On the other hand, the server 500 compares the third acquisition data included in the third transmission request with the first replacement data included in the first registration information, and exists in the third acquisition data. Determine additional parts that do not exist in the replacement data. Here, an area 615 in which handwritten characters that are present in the third acquired data 610B but not present in the first replacement data 610A are represented is determined as an additional portion. Furthermore, the server 500 specifies the first original data 610 specified by the first original identification information included in the first registration information and the second original data 620 related by the original association information. Further, the server 500 sets the second registration information including the second original identification information of the second original data 620 on the condition that the priority of the first original data is equal to or higher than the priority of the second original data. And the device that transmitted the second related portion 623A included in the second registration information is identified. Here, since the priority of the first original data is higher than the priority of the second original data and the second related portion 623A is transmitted to the portable information device 200A, the portable information device 200A is specified. Then, server 500 transmits additional portion 615 to portable information device 200A.

  When user B adds a handwritten character to the sheet on which the second replacement data 620 is formed, the server 500 specifies the second original identification information included in the second registration information. Original data 620 and the first original data 610 associated with the original association information are specified, but the priority of the second original data is lower than the priority of the first original data. The additional part is not transmitted to the portable information device 200 without extracting the first registration information including the first original identification information of the data 610. Therefore, by setting priority to a plurality of related original data, it is possible to determine whether or not to display an additional part added to one side on the other side.

  When the portable information device 200A receives the additional portion 615 from the server 500 while displaying the second related portion 623 received from the server 500, the portable information device 200A converts the additional portion 615 into the second acquired data (captured image). ) Is displayed at a position determined based on the AR tag 623A. FIG. 17 shows an example of an image displayed on portable information device 200A.

  FIG. 17 is a second diagram illustrating an example of an image displayed on the portable information device. Referring to FIG. 17, an image 620 </ b> C includes a character string 621 of “Confidential”, a related portion 623, and an additional portion 615 that is an area where handwritten characters are represented. Therefore, an image obtained by adding an additional portion 615 to the same image as the second original data 620 illustrated in FIG. 13 is displayed on the display unit 206 of the portable information device 200A.

  Thus, the user B can visually recognize the additional portion 615 added by the user A by handwriting. For this reason, the additional part added to at least one among the some original data relevant to each other can be notified to the user who browses other relevant original data.

  FIG. 18 is a flowchart illustrating an example of the flow of replacement data generation processing. The replacement data generation process is a process executed by CPU 111 when CPU 111 provided in MFP 100 executes a replacement data generation program stored in ROM 113, HDD 115, or CD-ROM 118. Referring to FIG. 18, CPU 111 authenticates a user who operates MFP 100 (step S01). A login screen is displayed on the display unit 161, and user identification information and a password input by the user to the operation unit 163 are accepted. If they are stored in the HDD 115 in advance, the user is authenticated.

  In the next step S02, data is acquired. The data acquired in step S02 may be data stored in advance in the HDD 115, print data received from the PC 300, or application data. Application data is data generated when the PC 300 executes an application program.

  In the next step S03, it is determined whether or not the data acquired in step S02 includes a plurality of pages. If the data includes a plurality of pages, the process proceeds to step S04. If the data acquired in step S02 does not include a plurality of pages, the data is set as original data to be processed, and the process proceeds to step S08.

  In step S04, one of the plurality of pages is selected as original data to be processed, and the process proceeds to step S05. In step S05, a registration information generation process is executed, and the process proceeds to step S06. Although details of the registration information generation process will be described later, it includes a process of converting original data set as a processing target into converted replacement data.

  In step S06, an original association information generation process is executed, and the process proceeds to step S07. Details of the original association information generation processing will be described later. In step S07, it is determined whether or not there is a page not selected as a processing target. If there is a page that is not selected as a processing target, the process returns to step S04. If there is no page, the process proceeds to step S10.

  In step S08, a registration information generation process is executed for the original data set as a processing target, and the process proceeds to step S09. In step S09, an original association information generation process is executed, and the process proceeds to step S10.

  In step S10, it is determined whether an output instruction has been accepted. When the operation unit 163 receives an operation of an output instruction by the user, it is determined that the output instruction has been received. For example, when a user presses a key to which a print instruction or a transmission instruction is assigned in advance among a plurality of keys provided in the hard key unit 167, the output instruction is accepted. The process waits until an output instruction is accepted (NO in step S10). If an output instruction is accepted (YES in step S10), the process proceeds to step S11. In step S11, the process branches depending on the output destination specified by the output instruction. If the output instruction indicates printing, the process is output to step S12. If the output instruction indicates data transmission, the process proceeds to step S14. For example, when a user presses a key assigned with a print instruction among a plurality of keys included in the hard key unit 167 in step S10, the process proceeds to step S12, and the key assigned a transmission instruction is pressed by the user. If so, the process proceeds to step S14.

  In step S12, the replacement data is printed, and the process proceeds to step S13. Specifically, the replacement data image generated in step S05 is formed on a sheet. In step S14, the replacement data is transmitted, and the process proceeds to step S13. Specifically, the replacement data generated in step S05 is transmitted to another computer via the communication I / F unit 112. The destination for transmitting the replacement data may be specified by the user using the operation unit 163. For example, when the portable information device 200 is designated by the user, the replacement data is transmitted to the portable information device 200.

  In the next step S13, the history information is stored and the process is terminated. The history information includes user identification information and original identification information. The user identification information is user identification information of the user authenticated in step S01. The original identification information is data identification information for identifying original data to be processed in the registration information generation process executed in step S05 or step S08. Specifically, the original data identification information is data identification information of each of a plurality of original data respectively corresponding to a plurality of pages when the data acquired in step S02 includes a plurality of pages, and is acquired in step S02. When the data includes one page, it is data identification information of the data.

  FIG. 19 is a flowchart illustrating an example of the flow of registration information generation processing. The registration information generation process is a process executed in step S05 or step S08 in FIG. Before the registration information generation process is executed, the original data to be processed is specified. Referring to FIG. 19, CPU 111 accepts designation of related parts in the original data (step S21). For example, if an image of original data is displayed on the display unit 161 and the user inputs an operation for designating an arbitrary region in the displayed image to the operation unit 163, the designated region is accepted as a related portion. The process waits until designation of a related part is accepted (NO in step S21). If designation of a related part is accepted (YES in step S21), the process proceeds to step S22. The related part includes position information in the original data.

  In step S22, an AR tag is determined. An arbitrary AR tag is determined from the AR tags prepared in advance. In the next step S23, the AR tag is associated with the related part. In step S21, the related part which is the area designated by the user in the original data is associated with the AR tag determined in step S22.

  In the next step S24, replacement data is generated. The replacement data is data obtained by replacing the relevant part of the original data with the AR tag. Specifically, the CPU 111 generates replacement data by replacing the related portion specified by the user in step S21 in the original data with the AR tag determined in step S22.

  In step S25, registration information is generated. The registration information includes original identification information for identifying original data, a related portion, an AR tag, and replacement data. Then, the registration information is registered in the server, and the process ends. The registration information generated in step S25 is transmitted to the server 500 via the communication I / F unit 112.

  FIG. 20 is a flowchart illustrating an example of the flow of the original association information generation process. The original association information generation process is a process executed in step S06 or step S09 in FIG. Before the original association information generation process is executed, the original data to be processed and the user who operates the MFP 100 are specified. Referring to FIG. 20, CPU 111 determines whether or not the original data is a part of data including a plurality of pages (step S31). If the original data is part of the data, the process proceeds to step S32. If not, the process proceeds to step S34. In step S32, another page is set as related original data, and the process proceeds to step S33. Among a plurality of pages included in the same data, data of each of one or more pages other than the original data page is set as related original data. In step S33, original association information is generated, and the process proceeds to step S34. The original association information includes two pieces of original identification information and associates two pieces of original data. Here, the original data is associated with the original data set as the related original data in step S32. When a plurality of related original data are set, the same number of pieces of original association information as the plurality of related original data are generated.

  In step S34, the same history information is extracted by the user or group. History information with the same user is history information including user identification information of the user who operates MFP 100 among the history information stored in HDD 115. The history information with the same group is history information including any of user identification information of each of one or more users belonging to the same group as the user who operates MFP 100 among the history information stored in HDD 115. The HDD 115 stores group data obtained by classifying a plurality of users into one or more groups, and uses the group data to identify one or more users belonging to the same group as the user who operates the MFP 100. For example, consider a case where user A who operates MFP 100 designates first original data and causes MFP 100 to execute a process of printing replacement data obtained by replacing the first original data with an AR tag. Prior to user A, user B belonging to the same group as user A designates the second original data, and causes MFP 100 to execute a process of printing replacement data in which the second original data is replaced with an AR tag. In this case, history information including the user identification information of the user B and the original identification information of the second original data is stored in the HDD 115. Therefore, when user A designates the first original data and causes MFP 100 to execute the process of printing the replacement data obtained by replacing the first original data with the AR tag, the user identification information of user B and the second The history information including the original identification information of the original data is extracted.

  In step S35, it is determined whether history information has been extracted. If the history information is extracted, the process proceeds to step S36. If not, the process proceeds to step S38. In step S36, the output original data is set as related original data. The output original data is the original data specified by the original identification information included in the history information extracted in step S34. In step S37, original association information is generated, and the process proceeds to step S38. The original data is associated with the original data set as the related original data in step S36. When a plurality of pieces of history information are extracted, the same number of original association information as the plurality of pieces of history information is generated.

  In step S38, the server is requested for candidate data. Transmission of candidate data is transmitted to the server 500 via the communication I / F unit 112. The server 500 that is requested to transmit candidate data returns the candidate data. In the server 500, candidate data can be arbitrarily determined. In the next step S39, when the communication I / F unit 112 receives one or more candidate data from the server 500, a candidate list from which one or more candidate data can be selected is displayed on the display unit 161. It is determined whether candidate data has been selected by the user (step S40). The process waits until one or more candidate data items are selected from the candidate list by the user. When one or more candidate data items are selected, the process proceeds to step S41. In step S41, the selected candidate data is set as related original data. In step S42, original association information is generated, and the process proceeds to step S43. The original data is associated with the candidate data set as the related original data in step S41. When a plurality of candidate data is extracted, the same number of original association information as the plurality of selected candidate data is generated.

  In step S43, it is determined whether or not original association information has been generated. If at least one piece of original association information has been generated, the process proceeds to step S44. If none has been generated, the process returns to the replacement data generation process. In step S44, a priority setting screen is displayed on display 161. The priority setting screen is, for example, the priority setting screen shown in FIG.

  In the next step S45, priority is accepted. If the user instructs one of the selection buttons 711 to 714 according to the priority setting screen and indicates the decision button 715, the priority is accepted. Then, priority information is generated (step S46). The priority information includes two sets of original identification information and priority. Specifically, the priority information includes a set of first original identification information and priority for the first original data, and a set of second original identification information and priority for the second original data. And including.

  In the next step S47, the original association information generated in step S33, step S37 or step S42 is transmitted to the server 500, and the process proceeds to step S48. In step S48, the priority information generated in step S46 is transmitted to server 500, and the process returns to the replacement data generation process.

  FIG. 21 is a flowchart illustrating an example of the flow of the virtual reality realization process. The virtual reality realization process is a process executed by the CPU 501 when the CPU 501 provided in the server 500 loads the virtual reality realization program stored in the ROM 502 or the HDD 504 to the RAM 502 and executes it. Referring to FIG. 21, CPU 501 determines whether or not registration information has been received. It is determined whether communication unit 505 has received registration information from MFP 100. If registration information is received, the process proceeds to step S52; otherwise, the process proceeds to step S53. In step S52, the received registration information is stored in HDD 504, and the process proceeds to step S53.

  In step S53, it is determined whether or not the original association information has been received. It is determined whether communication unit 505 has received the original association information from MFP 100. If the original association information is received, the process proceeds to step S54; otherwise, the process proceeds to step S55. In step S54, the received original association information is stored in HDD 504, and the process proceeds to step S55.

  In step S55, it is determined whether priority information has been received. It is determined whether communication unit 505 has received priority information from MFP 100. If priority information is received, the process proceeds to step S56; otherwise, the process proceeds to step S57. In step S56, the received priority information is stored in HDD 504, and the process proceeds to step S57.

  In step S57, it is determined whether a transmission request has been received. It is determined whether communication unit 505 has received a transmission request from any of portable information devices 200, 200A, 200B or HMD 400. The transmission request includes acquired data and an AR tag. If a transmission request is received, the process proceeds to step S58; otherwise, the process returns to step S51. Here, a case where a transmission request is received from portable information device 200 will be described as an example.

  In step S58, registration information including the AR tag included in the received transmission request is extracted from the registration information stored in HDD 115 in step S52. In the next step S59, it is determined whether or not registration information has been extracted. If the registration information is extracted, the process proceeds to step S60. If not, the process returns to step S51. In step S60, the original data is specified. The original data specified by the original identification information included in the extracted registration information is specified.

  In the next step S61, the related part is returned, and the process proceeds to step S62. Via the communication unit 505, the related part included in the extracted registration information is transmitted to the device that has transmitted the transmission request, here, the portable information device 200. In step S62, the transmission history is stored in HDD 504, and the process proceeds to step S63. The transmission history includes the original identification information of the original data specified in step S60 and the device identification information of the device that has transmitted the transmission request received in step S57.

  In step S63, the acquired data included in the transmission request is compared with the replacement data included in the registration information. And it is judged whether the additional part was extracted (step S64). The additional portion is a portion that exists in the acquired data but does not exist in the replacement data. If the additional part is extracted, the process proceeds to step S65. If the additional part cannot be extracted, the process returns to step S51. In step S65, an interlocked transmission process is executed, and the process returns to step S51.

  FIG. 22 is a flowchart illustrating an example of the flow of linked transmission processing. The linked transmission process is a process executed in step S65 of FIG. Before the linked transmission process is executed, the original data to be processed is specified and the additional part is determined. Referring to FIG. 22, CPU 501 determines whether or not original association information exists (step S71). Original association information including original identification information of original data to be processed is extracted from the original association information stored in the HDD 115. If such original association information is extracted, the process proceeds to step S72. If not, the process returns to the virtual reality implementation process. In step S72, related original data is specified. Of the two pieces of original identification information included in the original association information extracted in step S71, the original data specified by the original identification information different from the original identification information of the original data to be processed is specified as the related original data. In the next step S73, it is determined whether or not a transmission history exists. It is determined whether or not a transmission history including original identification information of related original data is stored in HDD 504. If there is a transmission history, the process proceeds to step S74; otherwise, the process returns to the virtual reality implementation process.

  In step S74, the related device is specified. The device specified by the device identification information included in the transmission history including the original identification information of the related original data is specified as the related device. Here, a case where portable information device 200A is specified as a related device will be described as an example.

  In step S75, it is determined whether priority information exists. From the priority information stored in the HDD 115, priority information including original identification information of the original data to be processed and the related original data is extracted. If such priority information is extracted, the process proceeds to step S76. If not, the process proceeds to step S78.

  In step S76, the priorities are compared. The priority of the original data to be processed determined by the priority information extracted in step S75 is compared with the priority of the related original data (step S77). Then, it is determined whether or not the comparison result is the first comparison result. Here, the first comparison result is a case where the priority of the original data to be processed is equal to or higher than the priority of the related original data. If the comparison result of the two priorities is the first comparison result, the process proceeds to step S78; otherwise, the process returns to the virtual reality implementation process.

  In step S78, the security level of the additional part is determined. Specifically, the character information included in the image of the additional portion is extracted by recognizing the image of the additional portion. Then, the security level is determined by comparing the extracted character information with a predetermined character string. The predetermined character string is a character string determined for each security level. For example, the character string “confidential” for the highest confidential level “1”, the character string “confidential” for the confidential level “2” lower than the confidential level “1”, and the like.

  In the next step S79, the security level of the user of the related device is determined. The server 500 stores in advance in the HDD 504 user information including a user identification information of a user, device identification information of a device assigned to the user, and a confidential level set assigned to the user. First, the user who operates portable information device 200A, which is the related device identified in step S74, is identified. The user information including the device identification information of the portable information device 200A is extracted, and the user identification information and the security level included in the extracted user information are acquired, thereby acquiring the security level of the user of the related device.

  In the next step S80, the confidential levels are compared. The security level of the additional part determined in step S78 is compared with the security level of the user of the related apparatus determined in step S79. Then, it is determined whether or not the comparison result is the second comparison result (step S81). Here, the second comparison result is a case where the security level of the user of the related apparatus is equal to or higher than the security level of the additional portion. If the comparison result of the two confidential levels is the second comparison result, the process proceeds to step S82; otherwise, the process returns to the virtual reality implementation process.

  In step S82, the additional part is transmitted to the related device, and the process returns to the virtual reality implementation process. Via the communication unit 505, the additional part is transmitted to the related device identified in step S74, here, the portable information device 200A.

  FIG. 23 is a flowchart illustrating an example of the flow of the augmented reality display process. The augmented reality display process is a process executed by the CPU 201 when the CPU 201 included in the portable information device 200 executes the augmented reality display program stored in the flash memory 203. Referring to FIG. 23, CPU 201 determines whether or not a document has been imaged (step S201). If the camera 202 has captured the subject, the process proceeds to step S202. If not, the process proceeds to step S203. Here, the subject of the camera 202 is a sheet on which an image of replacement data is formed by the MFP 100. In step S202, the captured image output by the camera 202 imaging the subject is set as acquired data, and the process proceeds to step S205.

  In step S203, it is determined whether wireless LAN I / F 208 has received data from the outside. If data is received, the process proceeds to step S204; otherwise, the process returns to step S201. Here, a case where replacement data is received from MFP 100 will be described as an example. In step S204, the replacement data received by wireless LAN I / F 208 from MFP 100 is set as acquired data, and the process proceeds to step S205.

  In step S205, the acquired data image set in step S202 or step S204 is displayed on display 206, and the process proceeds to step S206. In step S206, it is determined whether an AR tag has been extracted from the acquired data. If the AR tag is extracted, the process proceeds to step S207; otherwise, the process ends.

  In step S207, a transmission request is transmitted to server 500. The transmission request includes acquired data and an AR tag. Then, the wireless LAN I / F 208 determines whether or not the relevant part has been received from the server 500 (step S208). The process waits until the related part is received. If the related part is received, the process proceeds to step S209. In step S209, the AR tag included in the acquired data is replaced with the related part, and the process proceeds to step S210. The related part that replaces the AR tag is the related part acquired in step S208. Specifically, using the position information included in the related part, the replacement target area is determined based on the position and size of the AR tag placed in the acquired data, and the related part is placed in the determined replacement target area.

  In step S210, the related part is displayed, and the process proceeds to step S211. Specifically, the acquired data in which the related portions are arranged is displayed on the display unit 206. As a result, the original data image in which the AR tag portion of the acquired data is replaced with the related portion is displayed on the display unit 206.

  In step S211, it is determined whether an additional portion has been received. If the wireless LAN I / F 208 receives an additional portion from the server 500, the process proceeds to step S212. If not, the process ends. In step S212, the additional part is displayed on display unit 206, and the process ends. Using the position information included in the additional part, the replacement target area is determined based on the position and size of the AR tag arranged in the acquired data, and the additional part is placed in the determined replacement target area.

<Modification>
In the embodiment described above, the portable information device 200 has been described as an example of the augmented reality display device. However, the HMD 400 may function as the augmented reality display device. In this case, the CPU 401 included in the HMD 400 has the same function as that of the CPU 201 included in the portable information device 200 illustrated in FIG. 10, but the display control unit 251 is different.

  The display control unit 251 included in the CPU 401 included in the HMD 400 displays a related portion on the display unit 404 without displaying the captured image on the display unit 404 in response to the input of the captured image from the imaging control unit 261. The user wearing the HMD 400 actually sees the same image as the captured image through the display surface of the display unit 404. For this reason, the display control unit 251 specifies an area determined by the position information included in the related portion on the display surface of the display unit 404 with reference to the AR tag included in the captured image, and displays the display surface of the display unit 404. The image of the relevant part is displayed in the area specified in the area. Similarly, the interlocking display control unit 263 specifies an area determined by the position information included in the additional portion on the display surface of the display unit 404 with reference to the AR tag included in the captured image. The image of the additional portion is displayed in the area specified on the display surface.

  In the modification, the CPU 401 provided in the HMD 400 executes the augmented reality display process shown in FIG. 23, but steps S205, S210, and S212 are different. In step S205, the CPU 401 provided in the HMD 400 does not display the acquired data image. In step S210, the CPU 401 specifies an area determined by position information included in the related part on the display surface of the display unit 404 with reference to the AR tag included in the captured image, and specifies an image of the related part. And the process proceeds to step S211. In step S212, the CPU 401 specifies an area determined by the position information included in the additional portion on the display surface of the display unit 404 with reference to the AR tag included in the captured image, and specifies an image of the additional portion. Display in the selected area and finish the process.

  As described above, in the image processing system 1 according to the present embodiment, the MFP 100 converts the original data into replacement data in which a related part, which is at least a part of the original data, is replaced with an AR tag, An original in which registration information including original identification information for identifying original data, an AR tag, related parts, and replacement data is registered in the server, and the original data is associated with another original data different from the original data. Register the association information in the server and output the replacement data. In response to receiving registration information from MFP 100, server 500 associates original identification information included in the registration information, an AR tag, a related portion, and replacement data, and includes an AR tag and acquired data. In response to receiving the request from the first portable information device of portable information devices 200, 200A, 200B or HMD 400, the related part associated with the AR tag included in the transmission request is transmitted to the first portable information device. To do. Furthermore, the server 500 extracts an additional portion that exists in the acquired data included in the transmission request but does not exist in the replacement data associated with the AR tag included in the transmission request, and is associated with the AR tag included in the transmission request. The first original data and the second original data associated with the original association information are identified, and the second original data associated with the second original data from the portable information device 200, 200A, 200B or HMD 400 is identified. The second portable information device to which the related portion is transmitted is specified, and the additional portion is transmitted to the second portable information device. Each of portable information devices 200, 200A, 200B or HMD 400 acquires replacement data output from MFP 100 or data obtained by changing a part of replacement data as acquisition data, extracts an AR tag from the acquisition data, When the related portion associated with the AR tag is received, an image of the related portion is displayed at a position determined based on the position of the AR tag in the acquired data. For this reason, for example, since the related portion is displayed at the position determined by the AR tag in the acquired data obtained in the portable information device 200, the related portion can be displayed in the acquired data. When the acquired data is replacement data, original data corresponding to the replacement data can be displayed. Further, when the acquired data is data in which the replacement data is changed, the original data corresponding to the replacement data can be displayed in the state where the same change as the change to the replacement data is made.

  Further, the server 500 is a case where an additional part is extracted from acquired data received from the first portable information device, for example, the portable information device 200, and is associated with the AR tag received from the portable information device 200. When the first original data and the second original data related by the MFP 100 are specified, the second portable information device to which the second related part related to the second original data is transmitted, for example, The additional part is transmitted to portable information device 200A. In portable information device 200A, the second related part and the additional part are displayed. For this reason, in the acquired data acquired in the portable information device 200, an additional portion changed from the replacement data can be displayed on the portable information device 200A. As a result, the change added to the replacement data corresponding to the first original data can be notified to the user who views the replacement data corresponding to the second original data.

  In addition, server 500 transmits the additional portion to portable information device 200A, which is the second portable information device, on condition that a predetermined condition is satisfied. For this reason, an additional part can be displayed on a 2nd portable information device, only when a predetermined condition is materialized.

  If the predetermined condition is that the priority of the first original data is equal to or higher than the priority of the second original data, the change added to the replacement data corresponding to the first original data is changed to the second original data. The user viewing the replacement data corresponding to the data is notified, but the change added to the replacement data corresponding to the second original data is not notified to the user viewing the replacement data corresponding to the first original data. Therefore, whether or not to display the additional portion can be determined based on the priority of the related original data.

  Further, if the predetermined condition is that the confidential level for the user who operates the second portable information device is equal to or higher than the confidential level for the additional portion, the user who operates the second portable information device according to the confidential level of the additional portion. It can be determined whether or not to notify.

  In addition, MFP 100 associates each of a plurality of pages included in the acquired data with each other as a plurality of original data. For this reason, it is possible to notify a user who sees the replacement data of the original data of another page of a change made to the replacement data corresponding to the original data of any page of the same data.

  Further, MFP 100 associates a plurality of original data respectively corresponding to a plurality of replacement data output by a plurality of operations by the same user or a plurality of operations by a plurality of users belonging to the same group. For this reason, changes made to any of multiple replacement data output by multiple operations by the same user or multiple operations by any of multiple users belonging to the same group can be made available to users who view other replacement data. You can be notified.

  In addition, MFP 100 acquires original data as candidate data from the server, and associates original data corresponding to the output replacement data with candidate data selected by the user from the candidate data. Therefore, a change made to the replacement data corresponding to any of the plurality of original data associated by the user who operates MFP 100 can be notified to the user who sees the replacement data corresponding to the other original data.

  In the above-described embodiment, the image processing system 1 has been described. However, the replacement data generation process illustrated in FIGS. 18 to 20 is executed by the MFP 100, and the virtual reality implementation process illustrated in FIGS. 21 and 22 is performed. Needless to say, the present invention can be understood as an interlocking display method which is executed by the server 500 and causes the portable information device 200, 200A, 200B or HMD 400 to execute the augmented reality display processing shown in FIG.

  The embodiment disclosed this time should be considered as illustrative in all points and not restrictive. The scope of the present invention is defined by the terms of the claims, rather than the description above, and is intended to include any modifications within the scope and meaning equivalent to the terms of the claims.

<Appendix>
(1) The output control unit included in the image forming apparatus includes an image formation control unit that forms an image of the replacement data on a sheet, and the portable information device further includes an imaging unit that images a subject, The acquisition data acquisition unit included in each of the plurality of portable information devices is an imaging control unit that acquires, as acquisition data, a captured image obtained by the imaging unit imaging a sheet on which an image is formed by the image forming apparatus as a subject. The image processing system according to claim 1, wherein the AR tag extraction unit included in each of the plurality of portable information devices analyzes the acquired data and extracts an AR tag.
(2) The image processing system according to any one of claims 1 to 7, wherein the output control unit included in the image forming apparatus further includes a data transmission unit configured to transmit the replacement data to the portable information device.
(3) The image processing system according to (1), wherein the data transmission unit includes an e-mail transmission unit that transmits an e-mail including the replacement data.
(4) The image processing system according to any one of claims 1 to 7, wherein the output control means included in the image forming apparatus further includes a facsimile transmission control means for transmitting the replacement data by facsimile.

1 augmented reality display system, 2 network, 3 wireless station, 100 MFP, 200, 200A, 200B portable information device, 300 PC, 400 HMD, 500 server, 110 main circuit, 111 CPU, 112 communication I / F unit, 113 ROM 114 RAM, 115 HDD 115 Facsimile unit, 117 External storage device, 118 CD-ROM, 120 Automatic document feeder, 130 Document reading unit, 140 Image forming unit, 150 Paper feed unit, 155 Post-processing unit, 160 Operation panel, 161 Display unit, 163 operation unit, 201 CPU, 202 camera, 203 flash memory, 204 wireless communication unit, 205 call unit, 206 display unit, 207 operation unit, 208 wireless LAN I / F, 209 position detection unit, 210 direction detection unit, 211 External storage device, 401 CPU 402 Camera 403 Flash memory 404 Display unit 405 Wireless LAN I / F 501 CPU, 502 ROM, 503 RAM, 504 HDD, 505 Communication unit 506 Display unit 507 Operation unit 51 Data acquisition unit 53 Replacement unit 55 tag registration unit 55 transmission request receiving unit 56 additional part extraction unit 57 output control unit 59 user authentication unit 61 data association registration unit 63 priority registration unit 65 image formation control unit 67 data transmission 69, facsimile transmission control unit, 71 identical data association unit, 73 identical user association unit, 75 candidate data acquisition unit, 77 operation association unit, 251 display control unit, 253 acquisition data acquisition unit, 255 tag extraction unit, 257 related part request unit, 259 related part receiving unit, 261 imaging control unit, 263 interlocking display control unit, 2 67 linked transmission unit, 361 replacement unit, 551 tag association unit, 553 registration information extraction unit, 555 transmission request reception unit, 557 original association unit, 559 related part reply unit, 561 related original identification unit, 563 additional part extraction unit 565 Device identification unit, 567 Interlock transmission unit, 568 Priority setting unit, 571 User identification unit, 581 Rank comparison unit, 583 Security level determination unit, 585 Security level comparison unit.

Claims (8)

An image processing system including an image forming apparatus, a plurality of portable information devices, and a server,
The image forming apparatus includes a replacement unit that converts the original data into replacement data in which a related part that is at least a part of the original data is replaced with an AR tag
AR tag registration means for transmitting registration information including original identification information for identifying the original data, the AR tag, the related portion, and the replacement data to the server;
Output control means for outputting the replacement data;
Data association registration means for registering in the server, original association information that associates original data with different original data different from the original data, and
The server includes an AR tag association means for associating original identification information, an AR tag, a related portion, and replacement data included in the registration information in response to the registration information being received from the image forming apparatus. When,
In response to receiving a transmission request including an AR tag and acquired data from the first portable information device among the plurality of portable information devices, a related portion associated with the AR tag included in the transmission request is A related partial reply means for transmitting to one portable information device;
An additional part extracting means for extracting an additional part that exists in the acquisition data included in the transmission request but does not exist in the replacement data associated with the AR tag included in the transmission request;
When the additional part is extracted by the additional part extraction unit, the first original data associated with the AR tag included in the transmission request is associated with the original association information registered by the image forming apparatus. Related original specifying means for specifying the second original data;
When the second original data is specified by the related original specifying unit, a second related part associated with the second original data is transmitted from the plurality of portable information devices by the related part reply unit. Device specifying means for specifying the second portable information device,
Linked transmission means for transmitting the additional portion to the second portable information device,
Each of the plurality of portable information devices includes acquisition data acquisition means for acquiring replacement data output by the image forming apparatus or data obtained by changing a part of the replacement data as acquisition data;
An AR tag extracting means for extracting an AR tag from the acquired data;
Related portion requesting means for transmitting a transmission request including the AR tag extracted from the acquired data and the acquired data to the server;
Associated part receiving means for receiving an associated part associated with the AR tag from the server after transmitting the transmission request;
Display control means for displaying the received image of the related portion at a position determined with reference to the position in the acquired data of the AR tag extracted from the acquired data;
The display control means included in the second portable information device among the plurality of portable information devices displays the second related part, and then receives the additional part from the server in response to the acquisition. An image processing system comprising linked display control means for displaying the received image of the additional portion at a position determined based on a position in the acquired data of the AR tag extracted from data.   The image processing system according to claim 1, wherein the linked transmission unit included in the server transmits the additional portion on condition that a predetermined condition is satisfied. The image forming apparatus further includes priority registration means for registering, in the server, priority information that defines relative priorities for the first original data and the second original data.
The linked transmission means included in the server includes priority comparison means for comparing the priority of the first original data with the priority of the second original data,
The image processing system according to claim 2, wherein when the comparison result by the priority comparison unit is a first comparison result, the additional portion is transmitted. The server includes user specifying means for specifying users of the first portable information device and the second portable information device;
Security level determination means for analyzing the additional part extracted based on the first transmission request by the additional part extraction means and determining a security level for the additional part, and
The linked transmission means included in the server compares a security level predetermined for the user of the second portable information device with a security level for the additional portion determined by the security level determination means. A comparison means,
The image processing system according to claim 2, wherein the additional portion is transmitted when the comparison result by the security level comparison unit is a second comparison result. The image forming apparatus further includes data acquisition means for acquiring data,
The image processing according to claim 1, wherein the data association registration unit includes the same data association unit that associates each of a plurality of pages included in the acquired data as a plurality of original data. system.   The data association registration means includes a plurality of originals respectively corresponding to a plurality of replacement data output by the data output means by a plurality of operations by one user or a plurality of operations by a plurality of users belonging to the same group. The image processing system according to claim 1, further comprising a same-user association unit that associates data. The data association registration means includes candidate data acquisition means for acquiring original data as candidate data from the server;
The operation associating means for associating original data corresponding to the replacement data output by the data output means with candidate data selected by the user from among the candidate data. The image processing system described. A linked change method executed in an image processing system including an image forming apparatus, a plurality of portable information devices, and a server,
A replacement step of converting the original data into replacement data obtained by replacing the related data, which is at least a part of the original data, with an AR tag;
An AR tag registration step of transmitting registration information including original identification information for identifying the original data, the AR tag, the related portion, and the replacement data to the server;
An output control step for outputting the replacement data;
A data association registration step of registering original association information, which associates original data with another original data different from the original data, in the server,
An AR tag associating step associating the original identification information, the AR tag, the related portion, and the replacement data included in the registration information in response to receiving the registration information from the image forming apparatus to the server. When,
In response to receiving a transmission request including an AR tag and acquired data from the first portable information device among the plurality of portable information devices, a related portion associated with the AR tag included in the transmission request is A related partial reply step to be transmitted to one portable information device;
An additional part extracting step of extracting an additional part that exists in the acquisition data included in the transmission request but does not exist in the replacement data associated with the AR tag included in the transmission request;
When the additional portion is extracted in the additional portion extraction step, the first original data associated with the AR tag included in the transmission request is associated with the original association information registered by the image forming apparatus. A related original identification step for identifying the second original data;
When the second original data is specified in the related original specifying step, a second related part associated with the second original data is transmitted from the plurality of portable information devices in the related part returning step. A device specifying step for specifying the second portable information device,
Causing the second portable information device to perform an interlocking transmission step of transmitting the additional portion;
An acquisition data acquisition step of acquiring, as acquisition data, replacement data output by the image forming apparatus, or data obtained by changing a part of the replacement data, in each of the plurality of portable information devices;
An AR tag extraction step of extracting an AR tag from the acquired data;
An associated partial requesting step of transmitting a transmission request including the AR tag extracted from the acquired data and the acquired data to the server;
Receiving a related part associated with the AR tag from the server after transmitting the transmission request; and
A display control step of displaying an image of the received related portion at a position determined with reference to a position in the acquired data of the AR tag extracted from the acquired data;
The display control step of causing the second portable information device to be executed among the plurality of portable information devices displays the second related portion and then receives the additional portion from the server. An interlocking display method including an interlocking display control step of displaying an image of the received additional portion at a position determined based on a position in the acquisition data of the AR tag extracted from the acquisition data.

Images