JP5231799B2 - Control device, display device and control method thereof, mixed reality system, and computer program - Google Patents

Description

  The present invention relates to a control device, a display device, a control method thereof, a mixed reality system, and a computer program that can be used in a mixed reality technology using a video see-through HMD (Head Mounted Display).

  In recent years, a mixed reality, so-called MR (Mixed Reality) technology has been known as a technology for seamlessly combining the real world and the virtual world in real time. As one of the MR technologies, a video see-through HMD (Head Mounted Display) is used to capture a subject that substantially matches the subject observed from the pupil position of the HMD user with a video camera or the like. A technique is known in which an HMD user can observe a mixed reality image in which (Computer Graphics) is superimposed and displayed.

  FIG. 13 is a functional block diagram of a general video see-through mixed reality system. The outline will be described with reference to this figure. The video see-through mixed reality system is roughly composed of an HMD 1301, a control device 1302, and an image processing device 1303 as display devices.

  First, the HMD 1301 is a video see-through HMD. The HMD 1301 includes an imaging unit 1304 that images the outside, a display unit 1306 that displays an MR image on which CG is superimposed, and a communication I / F 1305 that performs data transmission and reception with the control device 1302.

  More specifically, the imaging unit 1304 is an imaging unit that captures an external observation image that substantially matches the line-of-sight position of the HMD user. It comprises two sets of image sensors for the right eye and left eye for generating a stereo image, an optical system, and a signal processing circuit for performing image processing at the subsequent stage. An interface (IF) 1305 is a communication interface for transmitting an image captured by the imaging unit 1304 to the control device 1302 and receiving a combined MR image. Here, it is assumed an optical fiber or LVDS that requires real-time performance and is capable of large-capacity transmission. The display unit 1306 displays the combined MR image, and includes two sets of display devices for right eye and left eye and an optical system. As the display device, a small liquid crystal display or a scan type device using MEMS is used.

  Next, the control device 1302 is a control device for controlling the HMD 1301. The communication I / F 1307, the I / F conversion unit 1308, the communication I / Fs 1309 and 1310 that perform data transmission / reception with the image processing apparatus 1303, and the optical distortion correction unit 1311 are included.

  More specifically, the interface (I / F) 1307 is a communication I / F on the control device 1302 side for performing data transmission / reception with the HMD 1301. This I / F is the same as that on the HMD 1301 side. When the received captured image data is transmitted to the image processing apparatus 1303, the I / F conversion unit 1308 converts the received captured image data into a data format handled by the I / F included in the image processing apparatus. An interface (I / F) 1309 is a communication I / F for transmitting captured image data that has undergone format conversion or protocol conversion in some cases to the image processing apparatus 1303. In this case, optical fibers such as USB and IEEE 1394 metal wires and Gigabit Ethernet (registered trademark), which are required to have real-time properties and can transmit a large capacity, are used. An interface (I / F) 1310 is an I / F for receiving a display image, and an I / F for display image transmission such as DVI or HDMI is used. The optical distortion correction unit 1311 is a correction processing unit that electronically corrects the optical distortion of the lens of the display unit in the HMD 1301. The details of the correction process will be described later.

  Furthermore, the image processing device 1303 measures the position and orientation for CG drawing based on the captured image received from the HMD 1301 via the control device 1302, draws the CG based on the content DB, and performs the combined processing with the captured image. Do. The image processing apparatus 1303 includes communication I / Fs 1312, 1317, a position / orientation information generation unit 1313, a content DB 1314, a CG drawing unit 1315, and an image composition unit 1316. The image processing apparatus 1303 can be configured using an apparatus having a high-performance arithmetic processing function or a graphic display function such as a personal computer or a workstation.

  More specifically, the interface (I / F) 1312 is a communication I / F on the image processing apparatus 1303 side for receiving a captured image, and the same one as that on the control apparatus 1302 side is used. The position and orientation information generation unit 1313 is a position and orientation information generation unit that generates position and orientation information of the wearer of the HMD 1301 from the received captured image. Although not shown here, the output signal of the three-dimensional position and orientation sensor attached to the HMD 1301 may be used supplementarily. There is also a form in which an identifier such as a marker is extracted from an image captured by an objective viewpoint camera (not shown) and used as correction information.

  The content DB 1314 is a database that stores virtual image content for rendering CG. The CG rendering unit 1315 renders a CG based on the position / orientation information and the content stored in the content DB 1314. The image composition unit 1316 synthesizes the received captured image and the virtual image generated by the CG drawing unit 1315. The interface (I / F) 1317 transmits an MR image (composite image) that is a composite image generated by the image composition unit 1316 to the control device 1302. The control device 1302 then sends the MR image to the HMD 1301 via the I / F 1307 and displays it on the display unit 1306.

  By mounting the video see-through HMD with the above configuration and processing process, it is possible to experience a mixed real world where the real world and the virtual world are seamlessly fused in real time. A general MR technique and system configuration are disclosed in Patent Document 1.

  In the MR system, in order to further increase the realism of the MR space, the resolution and gradation of captured images and CG images (virtual images) are improved, and smoothness in moving image display is realized by increasing the frame rate. Is required. In response to this requirement, the amount of data of captured images and MR images has increased dramatically, and an extremely wide transmission band for transmitting and receiving data has been demanded. Optical transmission of display image data by DVI is one of the solutions.

In the HMD used by the user, it is inevitable that the cable to the control device should be light and thin and easy to handle. Patent Document 2 discloses a technique for serializing a DVI RGB color signal, a parallel signal obtained by bundling a synchronization signal and a control signal by time-division multiplexing, and providing a flag in the packet field for serial transmission. Has been.
Japanese Patent Laid-Open No. 11-88913 (FIG. 7, paragraph [0035]) JP 2005-258579 A

  However, the conventional techniques described above have the following problems.

  First, in the above mixed reality system, in order to correct the optical distortion of the display system, the optical distortion correction unit 1311 performs a reverse correction as shown in FIG. 14 to generate a display image without deterioration. At that time, since the upper and lower sides of the display image are areas that cannot actually be used, it is not necessary to transmit the data in that portion, but it is not necessary to simply serialize and transmit the parallel image display data input from the DVI. Some parts are also transmitted.

  Also, if the image size of the imaging system is different from that of the display system, especially when the aspect ratio is different, the desired resolution conversion is performed, or after all the image data is transmitted, the unnecessary part is masked and combined immediately before display. Therefore, it is difficult to effectively use the bandwidth.

  Furthermore, when data transmission is performed by wireless communication, the transmission band is limited compared to the wired system. Therefore, it is very difficult to line up the resolution, gradation, frame rate at the time of moving image display, and displaying them with a reduced delay time in real time.

  The present invention has been made in view of such problems, and an object of the present invention is to reduce the total data transmission amount in data transmission between the display device and the control device, and to effectively use the bandwidth.

The present invention for solving the above problems is a control device for controlling transmission of display image data for display on an external display device,
When transmitting the transmission packet to the display image data to be transmitted to the line unit, it determines the display and transmitted as to line and transmitted not be good line of image data, based on a predetermined set value A determination means;
Packet generation means for generating a transmission packet for each line of the display image data, wherein at least the display image data is transmitted for the line of the display image data determined not to be transmitted by the determination means. includes a flag indicating not to perform and the synchronization signal of the line in the header, a packet generation unit that generates the transmission packet excluding the display image data,
And transmitting means for transmitting the transmission packets generated by the packet generator to the external display device.

The present invention for solving the above problems is a display device connected to the control device,
Receiving means for receiving a transmission packet from the control device;
Display means for displaying display image data included in the received transmission packet,
The display means displays Null data in a display area corresponding to the transmission packet when the received transmission packet includes a flag indicating that the display image data is not transmitted.

In addition, the display device
Imaging means for imaging real space and generating captured image data;
A position and orientation sensor for measuring the position and orientation of the display device;
Prediction means for predicting the position and orientation of the display device based on the measurement result of the position and orientation sensor;
A determination unit that determines a cut-out position of the captured image data based on a prediction result of the prediction unit;
The packet generating means for generating said transmission packet when transmitting the transmission packet to the captured image data in a line unit, per line of the captured image data to the transmission is not made based on the extraction position, at least the captured image A packet generation means for generating a transmission packet including a flag indicating that data transmission is not performed and a synchronization signal of the line in the header, and excluding the captured image data;
Transmission means for transmitting the transmission packet to the control device can be further provided.

  The present invention for solving the above problems includes the control device and the display device, and the display device displays a display image data obtained by combining the real space image and the virtual image. realizable.

  ADVANTAGE OF THE INVENTION According to this invention, the total amount of data transmission in the data transmission between a display apparatus and a control apparatus can be reduced, and a band can be utilized effectively.

  Embodiments of the invention will be described below with reference to the drawings.

[First Embodiment]
First, a first embodiment of the invention will be described with reference to the drawings. FIG. 1 is a functional block diagram illustrating a configuration example of an image forming system corresponding to the present embodiment. The image forming system includes a display device 101, a control device 102, and an image processing device 103.

  The display device 101 is assumed to be a video see-through HMD having a display function, and hereinafter, the display device 101 is referred to as an HMD 101. The HMD 101 includes an imaging unit 104, a packetization processing unit 105, an optical transmission communication I / F 106, a display image adjustment unit 107, and a display unit 108.

  The control device 102 is connected to an external HMD 101 and controls the HMD 101. Also, optical transmission communication I / F 109 that performs data transmission / reception with the HMD 101, an I / F conversion unit 110, and communication I / Fs 111 and 1112 that perform data transmission / reception with the image processing apparatus 103 are provided. Furthermore, an optical distortion correction unit 113 and a packetization mask processing unit 114 that is a feature of the present embodiment are provided.

  The image processing apparatus 103 measures the position and orientation for CG rendering based on the captured image received from the HMD 101 via the control apparatus 102, renders the CG based on the content DB, and captures the captured image that is a real space image and The synthesis process is performed. The image processing apparatus 103 includes communication I / Fs 115 and 120, a position / orientation information generation unit 116, a content DB 117, a CG drawing unit 118, and an image composition unit 119. The image processing apparatus 103 can be configured using an apparatus having a high-performance arithmetic processing function or a graphic display function such as a personal computer or a workstation.

  Hereinafter, components constituting each device will be described.

  First, in the HMD 101, the imaging unit 104 captures a real space observation image that substantially matches the line-of-sight position of the HMD wearer. The imaging unit 104 includes two sets of imaging elements and optical systems for right eye and left eye for generating a stereo image, and a signal processing circuit for performing subsequent image processing.

  The packetization processing unit 105 converts the captured image output from the imaging unit 104 into a packet configuration that allows serial transmission using an optical fiber. The captured image data output in parallel is serialized and packetized according to a predetermined transmission protocol.

  The optical transmission interface (I / F) 106 is a communication interface for transmitting an image captured by the imaging unit 104 to the control device 102 and receiving a synthesized MR image. Here, optical transmission using an optical fiber that requires real-time performance and capable of large-capacity transmission is assumed. Although the present embodiment has been described by taking wired data transmission by optical transmission as an example, broadband wireless technology such as UWB may be used. Also, it is not a method of preparing and transmitting transmission cables for transmission of RGB and sync signals, but serializing each color in time series, packetizing sync signals, and the number of cables required for transmission A configuration that reduces the number is assumed. The mask process when data is not transmitted will be described later.

  The display image adjustment unit 107 adjusts the display image data received from the control device 102 according to the display format of the display unit 108. The display image data serialized for optical transmission is converted into parallel data, and data of an area having no data to be displayed is generated in accordance with the display size of the display unit. Details will be described later.

  The display unit 108 displays the synthesized MR image. Similar to the imaging unit 104, the display unit 108 also includes two sets of display devices for the right eye and left eye and an optical system. As the display device, a small liquid crystal display or a scan type device using MEMS is used.

  Next, in the control apparatus 102, an optical transmission interface (I / F) 109 is a communication interface on the control apparatus 102 side. The optical transmission I / F 109 can have the same configuration as the optical transmission I / F 106 on the HMD 101 side.

  An interface (I / F) conversion unit 110 converts received captured image data into a data format handled by an interface included in the image processing apparatus 103 when transmitting the captured image data to the image processing apparatus 103.

  The interface (I / F) 111 is a communication interface for transmitting captured image data that has undergone format conversion, or protocol conversion in some cases, to the image processing apparatus 103. Optical fibers such as USB and IEEE 1394 metal wires and Gigabit Ethernet (registered trademark), which are required to be real-time and can transmit a large capacity, are used.

  The interface (I / F) 112 is a communication interface for receiving a display image. An I / F for display image transmission such as DVI or HDMI is used. The optical distortion correction unit 113 electronically corrects the optical distortion of the lens of the display unit in the HMD 101.

  The packetization mask processing unit 114 functions as a packet generation unit that converts a display image subjected to reverse distortion that cancels distortion generated in the optical system by optical distortion correction processing into a packet configuration that can be serially transmitted by an optical fiber. To do. In addition, it functions as a mask processing unit that masks data in a region that is not displayed as a result of the optical distortion correction processing in units of lines. When data transmission is not performed by mask processing, a flag that clearly indicates that mask processing has been performed is displayed as a flag in the header of the packet, and a packet that also sets a flag that clearly indicates the synchronization signal (horizontal synchronization signal) is transmitted. To do. Details will be described later.

  Next, in the image processing apparatus 103, an interface (I / F) 115 is a communication interface on the image processing apparatus 103 side for receiving a captured image. The I / F 115 can have the same configuration as the I / F 111 on the control device 102 side. The position / orientation information generation unit 116 generates position / orientation information of the wearer of the HMD 101 from the captured image received from the control device 102 via the I / F 115. The content DB 117 is a database stored as virtual image content. The CG drawing unit 118 draws a CG based on the position and orientation information and the content. The image composition unit 119 combines the captured image that is the received real space image and the virtual image generated by the CG drawing unit 118 to generate a composite image.

  The image forming system having the above configuration operates as follows.

  First, captured image data generated by the imaging unit 104 of the HMD 101 is converted into serial data by the packetization processing unit 105 and converted into a packet configuration according to a predetermined protocol. The converted captured image data is transmitted to the control device 102 via the optical transmission I / F 106.

  The control device 102 performs protocol conversion and format conversion on the captured image data received from the HMD 101 for transmission to the image processing device 103 by the I / F conversion unit 110, and then transmits the data to the image processing device 103.

  The image processing apparatus 103 generates position and orientation information of the wearer of the HMD 101 from the received captured image data, and generates a CG image. After synthesizing with the captured image data, the synthesized MR image is transmitted to the control device 102.

  In the control apparatus 102, reverse distortion is applied by the optical distortion correction unit 113 in order to cancel the influence of optical distortion on the display image, and the image is transmitted to the HMD 101 via the optical transmission I / F 109. At this time, if there is non-transmitted data due to mask processing at the same time as data packetization, this fact is clearly indicated in the header of the packet.

  In the HMD 101, the display image adjustment unit 107 converts the received display image into an appropriate format. At that time, the area where the masked data exists is filled with Null data on the HMD 101 side, sent to the display unit 108, and displayed to the user.

  As described above, in the image forming system of the present embodiment, data mask processing is performed to generate a non-display area by optical distortion correction and to adjust the resolution and aspect ratio of the imaging system and the display system. When the mask process is performed, the masked data is not transmitted to the HMD 101, but a mask flag is transmitted as a packet together with the synchronization signal. As a result, the amount of transmission data can be reduced and the bandwidth can be used effectively.

  Next, the configuration of the transmission packet corresponding to this embodiment will be described with reference to FIG. FIG. 2 is a diagram illustrating an example of a field component of a transmission packet corresponding to the embodiment of the invention.

  The transmission packet shown in FIG. 2 shows an example of the configuration of a packet used for transmission of stream data of an imaging system or a display system. The transmission packet 200 is composed of a plurality of component fields. Hereinafter, each component field will be described.

  The pcode field 201 is a code indicating the type of packet. There are types of image data of the imaging system, image data of the display system, control data, and the like. In the present embodiment, as will be described later, a vertical or horizontal synchronization signal is specified as a flag in the header of a packet that transmits image data. However, only the synchronization signal is transmitted as a packet, and the synchronization signal is transmitted on the receiving side. It is good also as a structure which produces | generates.

  The mask field 202 is a field that is a feature of this embodiment, and has a role of a flag for clearly indicating that there is no data to be transmitted. The maximum value of the image size that can be displayed on the display device of the display unit 108 is determined in advance, and there is a case where there is no data to be displayed due to a mismatch in size and aspect ratio with the imaging system, distortion correction, or the like. In such a case, it is more efficient to take a configuration in which data that is not displayed is not sent intentionally, data is not present, and data is generated on the receiving side. Actually, since there is no data to be displayed, it is filled with Null data. Therefore, the mask field 202 is set to “ON” when the data of the corresponding line is not transmitted. On the other hand, when transmitting the data of the corresponding line, this mask field 202 is set to “OFF”. Thus, the presence or absence of data transmission can be notified to the receiving side by the flag of the mask field.

  If the synchronization signal is not configured as a separate packet, a flag that clearly indicates the rising edge of the vertical synchronization signal and the horizontal synchronization signal may be provided in this field. Furthermore, an offset value can be included when there is a horizontal offset as described in the second embodiment.

  The rl field 203 is a field for clearly indicating whether the data of the corresponding packet is for the right eye or for the left eye. In the HMD 101, the image system and display system for the right eye, the left eye, and the display system handle two systems of image data. The reserved 204 is a field for reservation.

  The index (index) field 205 is an index number when one frame of image data is divided into a plurality of packets. In this example, since it is configured as one packet for each horizontal synchronization signal, that is, for each line, the index number is synonymous with the number of the horizontal synchronization signal or the number of lines.

  The data_length (data length) field 206 is a field indicating the data length of the subsequent data field. In this example, 1 bit indicates a data length of 1 byte. A data_field (data field) 207 is a field for storing display image data to be transmitted. If it is assumed that display image data having an image size of SXGA (1280 × 1024) is transmitted, the data size of one line is 1280 (pixels) × 3 bytes (8 bits for each RGB), for a total of 3840 bytes.

  Next, the relationship between the display images before and after transmission will be described with reference to FIG. FIG. 3 a is a diagram for explaining the relationship between display images before and after transmission, corresponding to the first embodiment of the invention.

  In FIG. 3a, an area 300 of the display image having the number of pixels of length L1 and width N is shown as an example. For example, it is assumed that the display image size received by the control apparatus 102 via DVI is the L1 × N image size. However, depending on the design conditions, as a result of the optical distortion correction process, the image size becomes an area 303 excluding the upper and lower areas 301 and 302 indicated by the hatched portion in the figure. In this case, the areas 301 and 302 are invalid areas and are areas in which a display image is not displayed in the display area of the display unit 108. Note that the vertical size at this time is L2 (<L1).

  When the display image 300 is transmitted from the control device 102 to the HMD 101, it is efficient to transmit the valid data 303 excluding the hatched area. On the other hand, the display size of the display unit 108 of the HMD 101 is L1 × N including an invalid area. Therefore, when displaying on the display unit 108, a non-transmitted area is filled with null data.

  Specifically, if L1 = 1024, N = 1280, L2 = 960, and the widths of the hatched areas above and below are equal, the position of a indicated by the arrow can be 32. In this case, the 0th to 31st lines are masked. The number of lines to be masked is held by the packetization mask processing unit 114 as a setting value for mask processing.

  Next, a data flow between the HMD 101 and the control device 102 will be described with reference to FIG. FIG. 3B is a sequence diagram illustrating an example of a data flow between the HMD 101 and the control apparatus 102 corresponding to the first embodiment of the invention. In the data flow of FIG. 3B, as an example, it is assumed that the display image illustrated in FIG. 3A is transmitted from the control device 102 to the HMD 101.

  First, a V packet representing a vertical synchronization signal indicating the head of a frame is transmitted from the control device 102 to the HMD 101. In this example, for the sake of simplicity, the vertical and horizontal synchronization signals are different from the packets for transmitting the stream data, but may be sent as stream data packets provided with a field for clearly indicating the synchronization signal.

  After transmitting the vertical synchronization signal V, an H packet representing the horizontal synchronization signal is transmitted continuously. In this example, another packet configuration is the same as the vertical synchronization signal.

  Next, the display stream packet is transmitted. At this time, since the 32 lines from the top are not transmitted, the stream packet itself is transmitted, but the mask field flag in the header described above is set, the value of the data length field 206 is 0, and the data that follows There is also no field 207. The HMD 101 that receives this packet knows that no data exists because the mask bit is ON. The same sequence continues for a while while the data is masked.

  Since stream data exists from the 33rd line (a line in FIG. 3a), normal packet transmission is performed. The contents and flow of the processing on the control device 102 side and the processing on the HMD 101 side will be described later with reference to separate drawings.

  First, the processing on the control device 102 side will be described with reference to FIG. FIG. 4 is a flowchart showing an example of processing of the control device 102 corresponding to the first embodiment of the invention. The processing corresponding to the flowchart is realized by each functional block of the control device 102 executing a corresponding processing program.

  In FIG. 4, first, in step S401, the packetization mask processing unit 114 determines whether or not mask setting is performed as an initial setting when the system is operating. If the mask is set (“YES” in step S401), the process proceeds to step S402. On the other hand, if the mask is not set ("NO" in step S401), the process of step S402 is skipped and the process proceeds to step S403 to transmit all data.

  In step S402, the packetization mask processing unit 114 reads the set value in response to the mask setting. More specifically, the number of lines to be masked is determined and set up and down. In step S403, the control apparatus 102 receives an input of a display image from the image processing apparatus 103 via DVI. In step S <b> 404, the optical distortion correction unit 113 performs an optical distortion correction process that applies reverse distortion so that normal display is performed as a result of the optical system of the display unit 108.

  In step S405, the packetization mask processing unit 114 performs reading in units of lines in order to process the correction processing results in units of lines. In step S <b> 406, the packetization mask processing unit 114 determines whether or not the processing target line is a line on which mask processing is performed in light of the set value. If the line number corresponds to the mask process (“YES” in step S406), the process proceeds to step S407. If the line does not correspond to the mask process (“NO” in step S406), the process proceeds to step S408.

  In step S407, in response to the line to be masked, the packetization mask processing unit 114 sets the mask bit (flag) in the mask field of the packet header to ON. In this case, since the data is not transmitted, the data body to be transmitted is not added to the data field 207, and the data length 206 is also set to zero. In step S408, the mask bit (flag) in the mask field 202 is turned OFF in response to the line performing normal processing without performing mask processing.

  In step S409, the packetization mask processing unit 114 reflects the value of each mask field 202 to form a packet. If the synchronization signal is not in a separate packet configuration, the synchronization signal flag is also set in the mask field 202. In step S410, the generated packet is transmitted to the HMD 101 via the optical transmission I / F 109.

  Next, the processing on the HMD 101 side will be described with reference to FIG. FIG. 5 is a flowchart showing an example of processing of the HMD 101 corresponding to the first embodiment of the invention. The processing corresponding to the flowchart is realized by each functional block of the HMD 101 executing a corresponding processing program.

  First, in step S501, a packet is received from the control apparatus 102 via the optical transmission I / F 106. The received packet is a stream data packet, and the synchronization signal is also included in the header. In step S502, the display image adjustment unit 107 checks the synchronization signal flag in the mask field in the packet header. In step S <b> 503, the display image adjustment unit 107 receives the confirmation of the type of the synchronization signal, and generates a synchronization signal used by the display unit 108.

  In step S504, the display image adjustment unit 107 determines whether the mask bit in the mask field in the header is ON. If the mask bit is ON, that is, if the data is not transmitted by the mask process (“YES” in step S504), the process proceeds to step S505. On the other hand, when the mask bit is OFF, that is, when normal data transmission is performed (“NO” in step S504), the process proceeds to step S506.

  In step S <b> 505, the display image adjustment unit 107 receives the fact that the mask processing is being performed, and writes Null data to the memory used by the display unit 108 in the corresponding area. Thereafter, the process proceeds to step S508. In step S506, in response to the mask process not being performed, the display image adjustment unit 107 acquires the display image data in the data field of the packet. In the subsequent step S507, the display image data acquired by the display image adjustment unit 107 is written in the memory.

  In step S508, the display unit 108 performs display at the timing when the storage of the image for one frame is completed.

  As described above, in this embodiment, a flag that clearly indicates non-transmission of data is prepared together with serialization of the display image and packetization of the horizontal and vertical synchronization signals, and the receiving side performs image reconstruction. As a result, the total amount of data transmission can be reduced and the bandwidth can be used effectively.

[Second Embodiment]
Next, a second embodiment for realizing the invention will be described. In the first embodiment described above, the mask area is determined based on the result of the optical distortion correction process and the size and aspect ratio of the imaging system and the display system. On the other hand, the image forming system corresponding to the second embodiment is different in that the result of the adjustment process at the time of factory shipment is reflected. Specifically, when the display image displayed on the left and right eyes requires fine adjustments in the vertical and horizontal directions, the same function is realized by electronically correcting the accuracy during assembly to some extent. . This feature will be described in more detail below.

  First, a system configuration corresponding to the second embodiment of the invention will be described with reference to FIG. FIG. 6 is a functional block diagram illustrating an example of a system configuration corresponding to the present embodiment. As shown in FIG. 6, the system corresponding to this embodiment includes an HMD 601, a control device 602, a personal computer (PC) 603, and an inspection camera 604.

  The HMD 601 displays an MR image. Since the imaging, display, and image transmission functions of the HMD 601 are the same as those in the first embodiment, description thereof is omitted here. In the following description, when referring to the configuration block of the HMD 601, reference is made using the reference numbers in FIG. 1. For example, the display unit in the HMD 601 is referred to as the display unit 108.

  The control device 602 controls the HMD 601. The function of the control device 602 is the same as that of the control device 102 described in the first embodiment, but is different in that it further includes an adjustment value storage unit 610 that stores values at the time of adjustment. In the following description, when referring to the constituent blocks of the control device 602, the reference numerals in FIG. 1 are used as in the case of the HMD 601.

  The PC 603 is a processing device having a function for outputting an adjustment image and grasping an adjustment value. The PC 603 includes an image generation unit 605, an image output unit 606, an image input unit 607, a comparison / adjustment function unit 608, and an adjustment value setting unit 609.

  The inspection camera 604 is an inspection camera for confirming the display output of the HMD. A display image captured by the camera is taken into the PC 603 and used for adjustment.

  The image generation unit 605 generates an adjustment image to be displayed on the HMD 601. The image output unit 606 outputs the display image generated by the image generation unit 605 from a graphics board having an output such as DVI. The image input unit 607 inputs a camera image captured by the inspection camera 604 to the PC 603 via an I / F such as USB or IEEE1394.

  The comparison / adjustment function unit 608 compares the image output as the display image with the image captured by the inspection camera 604. If the display position is shifted as a result of the comparison, the amount of deviation is calculated, an image is generated again, and the image is output. This process is repeated, and when it falls within the desired tolerance range, the process is finished, and an adjustment value indicating how much the deviation is finally sent is sent to the adjustment value setting unit 609. Details of processing in the comparison / adjustment function unit 608 will be described later.

  The adjustment value setting unit 609 sets an adjustment value for the control device 602. The set value of the deviation amount calculated by the comparison / adjustment function unit 608 is transmitted to the control device 602. A serial I / F such as RS-232C or USB, or a debugging I / F (for example, J-TAG) of the control device is used.

  With this configuration, in this embodiment, an adjustment value for automatically adjusting the display position at the factory or the user side is calculated and set for the control device 602.

  Next, image positions before and after adjustment will be described with reference to FIG. FIG. 7 is a diagram for explaining the relationship between image positions before and after adjustment, corresponding to the second embodiment of the invention.

  FIG. 7 is a schematic diagram of both display images of the left eye image 700L and the right eye image 700R. In the left eye image 700L and the right eye image 700R, the outermost frames 701L and 701R indicate the imaging range of the inspection camera. Rectangles 702L and 702R indicated by inner solid lines indicate the position of the display image in an ideal state. On the other hand, dotted lines 703L and 703R indicate display positions that are actually observed.

  In the left-eye image 700L, it can be seen that the dotted line portion 703L is shifted diagonally downward to the right. Also, it can be seen that the dotted line portion 703R is shifted downward in the right eye image 700R. Although schematically shown here, an image for inspection having a high contrast and allowing easy confirmation of the display position is displayed and compared with an ideal position.

  Next, processing in the PC 603 corresponding to the present embodiment will be described with reference to FIG. FIG. 8 is a flowchart showing an example of processing corresponding to the second embodiment of the invention. In FIG. 8, steps S801 to S809 are the contents of processing performed on the PC 603 side, and only step S810 is the contents of processing performed by the control device 602. Therefore, the processing corresponding to the flowchart is realized by the functional blocks of the control device 602 and the PC 603 executing the corresponding processing program.

  First, in step S <b> 801, the image generation unit 605 generates an image so that the standard image is displayed at the reference position, and outputs the image from the image output unit 606. In step S <b> 802, the image displayed on the display unit 108 of the HMD 601 is captured by the inspection camera 604, and the camera image is captured into the PC 603.

  In step S803, the comparison / adjustment function unit 608 compares the positional relationship between the output standard image and the input image captured from the inspection camera 604. In step S804, it is determined whether the display position shift is within an allowable range as a result of the comparison. This allowable range can be set, for example, as “N pixels in the vertical direction and M pixels in the horizontal direction”. The specific value is a design matter and is not particularly limited here. If it is not within the allowable range, the process proceeds to step S805. If it is within the allowable range, the process proceeds to step S809.

  In step S805, the comparison / adjustment function unit 608 determines whether the amount of deviation exceeds a specified value that cannot be electrically corrected, in response to the positional deviation being greater than or equal to the allowable value. If the amount of deviation exceeds the specified value, the process proceeds to step S806, and if it is within the specified value, the process proceeds to step S807. In step S806, it is determined that mechanical adjustment is necessary in response to the amount of deviation being greater than or equal to the specified value, and the electronic adjustment process is terminated.

  In step S <b> 807, the image generation unit 605 acquires information on the amount of deviation from the comparison / adjustment function unit 608, and newly generates an image to be displayed so that the amount of deviation falls within the allowable range. Actually, a display image shifted so as to cancel out the shift amount obtained from the comparison result is generated. In step S808, the image output unit 606 outputs the display image for confirmation of adjustment generated in step S807 to the control device 602.

  Thereafter, the display image is captured by the inspection camera 604, and the process is repeated until it falls within the allowable value range. In step S809, the final adjustment value is determined in response to the positional deviation falling within the allowable value range. As shown in FIG. 7, the adjustment value in this case can be adjusted not only in the upper and lower line directions but also in the left and right pixel units.

  In step S810, the adjustment value storage unit 610 of the control device 602 stores the determined adjustment value. The processing when the system is actually operated after saving will be described with reference to another drawing.

  Next, processing in the control device 602 will be described with reference to FIG. FIG. 9 is a flowchart showing an example of processing of the control device 602 corresponding to the second embodiment of the invention. The processing corresponding to the flowchart is realized by each functional block of the control device 602 executing a corresponding processing program.

  In FIG. 9, in step S <b> 901, an adjustment value for correcting the display position when the system is activated is read from the adjustment value storage unit 610. In step S902, the I / F 112 receives the synthesized MR image from the image processing apparatus 103 as a display image to the HMD 601.

  In step S903, the optical distortion correction unit 113 performs optical distortion correction on the input display image. In step S904, the packetized mask processing unit 114 reads the display image after distortion correction in units of lines. In step S905, it is determined whether the processing target line is a corresponding line for mask processing.

  Although not specifically described here, the process is performed including the optical distortion correction process and the mask process associated with the aspect ratio conversion described in the first embodiment. That is, the determination of the corresponding line to be masked is performed based on a setting value obtained by combining a setting value on the assumption that there is no deviation amount and an adjustment value determined at the time of adjustment. The adjustment value of the deviation amount actually acts as a change or replacement of the line number, and the mask process is applied to the line where data does not exist due to the shift.

  For example, when the adjustment value of the deviation amount is “two lines below”, the upper two lines are masked and the lower two lines are not adopted. The two lines are out of the display area of the display unit 108 in the display image. If it is a corresponding line of the mask process, the process proceeds to step S906, and if it is a non-relevant line, the process proceeds to step S907.

  In step S <b> 906, the packetization mask processing unit 114 performs mask processing in response to the fact that the line is a mask processing target line. The contents of the mask process are the same as those in the first embodiment, and a description thereof is omitted. In step S907, the packetization mask processing unit 114 determines whether a horizontal adjustment value is set. If adjustment is necessary (“YES” in step S907), the process proceeds to step S908. On the other hand, when adjustment is not necessary (“NO” in step S907), the process proceeds to step S909.

  In step S908, the packetization mask processing unit 114 determines the offset value of the first pixel to be used as the display pixel in the horizontal line. As a result, the position of the image to be displayed can be adjusted by adjusting the reading position by the offset position on the HMD 601 side.

  In step S909, the packet is configured by reflecting the value of each mask field. At this time, for a line for which an offset value is set, the offset value is included in the mask field 202. In addition, the data field 207 does not include data of pixels that are out of the display area of the display unit 108 of the HMD 601 due to the offset. If the synchronization signal is not in a separate packet configuration, the synchronization signal flag is also set in the mask field. In step S910, the generated packet is transmitted.

  As described above, it is possible to transmit only the data necessary for display even when the display position is misaligned at the time of shipment from the factory or at the time of adjustment by the user and the display position is shifted during transmission of the display image. . As a result, it is possible to effectively use the transmission band.

[Third Embodiment]
Next, a third embodiment for realizing the invention will be described. In the first embodiment described above, image composition is performed by the image processing apparatus 103. On the other hand, the present embodiment is characterized in that image synthesis is performed on the control device or HMD side, and the delay of a real image as a background image is suppressed by synthesizing with the latest captured image. In addition, the response when the captured image used for composition is not transmitted is also included.

  First, a system configuration corresponding to the third embodiment of the invention will be described with reference to FIG. FIG. 10 is a functional block diagram illustrating an example of a system configuration corresponding to the present embodiment. As shown in FIG. 10, the system corresponding to this embodiment includes a display device 1001, a control device 1002, and an image processing device 1003.

  The display device 1001 is assumed to be a video see-through HMD having a display function. Hereinafter, the display device 1001 is referred to as an HMD 1001. The HMD 1001 includes an imaging unit 1004, a three-dimensional position / orientation sensor 1005, a position / orientation information generation unit 1006, a packetized mask processing unit 1007, an optical transmission communication I / F 1008, a display image adjustment unit 1009, and a display unit 1010.

  The functions of the imaging unit 1004, the optical transmission communication I / F 1008, the display image adjustment unit 1009, and the display unit 1010 are the same as the corresponding blocks shown in FIG.

  The control device 1002 controls the HMD 1001. The control device 1002 performs communication I / F 1011 for optical transmission that performs data transmission / reception with the HMD 1001, communication I / F 1011 that performs data transmission / reception with the data conversion unit 1012, the captured image conversion unit 1013, the I / F conversion unit 1014, and the image processing device 1003. / F1015, 1016 are provided. Further, an image composition unit 1017, an optical distortion correction unit 1018, and a packetization mask processing unit 1019 are further provided. The optical transmission communication I / F 1011, the I / F conversion unit 1014, the communication I / Fs 1015 and 1016, the optical distortion correction unit 1018, and the packetization mask processing unit 1019 have the same functions as the corresponding blocks in FIG. Description is omitted.

  The image processing apparatus 1003 corrects the position and orientation for CG rendering based on the captured image received from the HMD 1001 via the control device 1002 and the position and orientation information generated as a measurement result in the position and orientation information generation unit 1006, and the content DB CG is drawn based on the above. Since the functions other than the position / orientation correction unit 1021 are the same as those in FIG. 1 except that the image composition function is omitted, the description thereof is omitted here.

  The three-dimensional position / orientation sensor 1005 is a sensor for assisting the position / orientation measurement of the wearer of the HMD 1001, and assumes a magnetic sensor or a gyro sensor (acceleration, angular velocity). The position / orientation information generation unit 1006 generates the position / orientation information of the wearer of the HMD 1001 using the output of the three-dimensional position / orientation sensor 1005 and the identification marker extracted from the captured image by image processing.

  In the present embodiment, the three-dimensional position / orientation sensor 1005 is used for measurement assistance. However, the use and mounting of the sensor are not essential, and the measurement may be performed only by identifying the marker included in the captured image.

  The packetization mask processing unit 1007 acquires a captured image having a size larger than the display area, predicts the position at the display timing based on the position and orientation information at the time of acquiring the captured image, and determines the size to be cut out based on the prediction result. decide. In addition, data in an area unnecessary for display is masked in units of lines. Thereby, the delay of the captured image used as the background of a synthesis | combination with CG can be suppressed to the minimum.

  When data transmission is not performed by mask processing, a flag that clearly indicates that mask processing has been performed is indicated by a flag in the header of the packet, and a packet that has a flag that clearly indicates a synchronization signal (horizontal synchronization signal) is transmitted. . As in the second embodiment, the horizontal offset value is also set, and necessary data on the horizontal line is selected to perform packetization.

  When the image processing apparatus 1003 requires the captured image to extract the identifier from the image captured by the HMD 1001 and perform the position / orientation measurement, the data conversion unit 1012 captures the image up to the image size necessary for the position / orientation measurement. Reduce processing. Further, when a captured image is necessary to correct the value already calculated on the HMD 1001 side, the image is reduced to the image size necessary for correction.

  The captured image conversion unit 1013 converts the captured image data received from the HMD 1001 according to the data format at the time of image composition. The captured image data serialized for optical transmission is converted into parallel data.

  The image composition unit 1017 synthesizes each input captured image and the virtual image generated by the CG rendering unit 1023 of the image processing apparatus 1003. There are several methods for synthesizing images. Here, image synthesis using chroma key processing and an α channel is assumed.

  The position / orientation correction unit 1021 is arranged on the image processing apparatus 1003 side, and has the same function as that of the HMD 1001 or a function of correcting the position / orientation information generated by the HMD 1001 in accordance with the timing of image composition. It doesn't matter. In the correction, a process of calculating position / orientation information at the time of image composition from past position / orientation information grasped in time series is performed. This process may be a general prediction method, and a well-known method uses a Kalman filter. Here, the timing is the same as the timing at the time of image synthesis. In that case, although there is a slight time lag from the acquisition to the display for the captured image, the CG image can appear to guarantee real-time characteristics as long as the prediction is not significantly shifted.

  Next, image positions before and after adjustment will be described with reference to FIG. FIG. 11 is a conceptual diagram for explaining processing for cutting out a display area from a captured image, corresponding to the third embodiment of the invention.

  In FIG. 11, an outer frame 1101 is an image size of a captured image captured by the imaging unit 1004. A rectangle 1102 indicated by a solid line on the inner side indicates the position of the captured image in an ideal state where there is no spatial displacement between the time of image capturing and the time of image display or image composition. A hatched area 1103 surrounded by a dotted line portion indicates a cut-out position of a captured image that is cut out in accordance with timing at the time of image display or image synthesis based on position and orientation information. In this example, it is predicted that the head is moving in the upper right.

  Next, processing in the HMD 1001 will be described with reference to FIG. FIG. 12 is a flowchart showing an example of processing of the HMD 1001 corresponding to the third embodiment of the invention. The processing corresponding to the flowchart is realized by each functional block of the HMD 1001 executing a corresponding processing program.

  In FIG. 12, in step S1201, the captured image captured by the imaging unit 1004 is output. This captured image is an image larger than the image synthesis size. In step S <b> 1202, the position / orientation information generation unit 1006 acquires the output value of the three-dimensional position / orientation sensor 1005. The position / orientation information generation unit 1006 holds a past sensor output value. In step S1203, the sensor output value at the timing of imaging and the imaging at the time of image synthesis or image display based on the past sensor output value. Predict location. The past sensor output value is updated with a new sensor output value.

  In step S1204, the packetization mask processing unit 1007 sets the cutout position of the captured image based on the predicted imaging position at the time of image synthesis or image display. In step S1205, it is determined based on the set value whether the processing target line is a line to be masked. If the line number corresponds to the mask process (“YES” in step S1205), the process proceeds to step S1206. If it does not correspond to the mask process (“NO” in step S1205), the process proceeds to step S1207.

  In step S1206, the packetization mask processing unit 1007 turns ON the mask bit (flag) in the mask field 202 of the packet header when the line corresponds to the mask processing. At this time, since the data is not transmitted, no data is added. In step S1207, the packetization mask processing unit 1007 turns off the mask bit (flag) of the mask field 202 in response to the line performing normal processing without performing mask processing.

  In step S1208, the packetization mask processing unit 1007 configures a transmission packet based on the value of the mask field 202. If the synchronization signal is not in a separate packet configuration, the synchronization signal flag is also set in the mask field. In step S1209, the generated packet is transmitted.

  As described above, it is possible to effectively use the band by clearly indicating that data is not transmitted when transmitting a captured image as well as a display image. Further, by adopting such a configuration, it is possible to provide a mixed reality system in which the delay time from imaging to display can be felt in terms of experience.

  As described above, according to the present invention, the flag for clearly indicating the serialization of the image data of the display system and the imaging system and the packetization of the horizontal and vertical synchronization signals and the non-transmission of the data is prepared, and the image is reconstructed on the receiving side. I do. As a result, the total data transmission amount can be reduced and the band can be used effectively without performing resolution conversion or mask processing on the receiving side. This is more effective during wireless communication where the transmission band is limited compared to the wired system.

  In addition, by incorporating this control system in the adjustment process, the accuracy of adjustment at the time of factory assembly can be reduced, and the time and cost for adjustment can be reduced. In addition, since adjustment is possible on the user side, a certain correction effect can be obtained even with respect to a shift of a display image that normally requires mechanical adjustment.

[Other Embodiments]
Note that the present invention can be applied to a system (for example, a copier, a facsimile machine, etc.) consisting of a single device even when applied to a system composed of a plurality of devices (for example, a host computer, interface device, reader, printer, etc.). You may apply.

  The object of the present invention can also be achieved by supplying, to a system, a storage medium that records the code of a computer program that realizes the functions described above, and the system reads and executes the code of the computer program. In this case, the computer program code itself read from the storage medium realizes the functions of the above-described embodiments, and the storage medium storing the computer program code constitutes the present invention. In addition, the operating system (OS) running on the computer performs part or all of the actual processing based on the code instruction of the program, and the above-described functions are realized by the processing. .

  Furthermore, you may implement | achieve with the following forms. That is, the computer program code read from the storage medium is written into a memory provided in a function expansion card inserted into the computer or a function expansion unit connected to the computer. Then, based on the instruction of the code of the computer program, the above-described functions are realized by the CPU or the like provided in the function expansion card or function expansion unit performing part or all of the actual processing.

  When the present invention is applied to the above storage medium, the computer program code corresponding to the flowchart described above is stored in the storage medium.

1 is a functional block diagram illustrating a configuration example of an image forming system corresponding to a first embodiment of the invention. It is a figure which shows an example of the field component of the transmission packet corresponding to the 1st Embodiment of invention. It is a figure for demonstrating the relationship of the display image before and behind transmission corresponding to the 1st Embodiment of invention. It is a sequence diagram which shows an example of the data flow between HMD101 and the control apparatus 102 corresponding to the 1st Embodiment of invention. It is a flowchart which shows an example of the process of the control apparatus 102 corresponding to the 1st Embodiment of invention. It is a flowchart which shows an example of a process of HMD101 corresponding to the 1st Embodiment of invention. It is a functional block diagram which shows an example of the system configuration corresponding to the 2nd Embodiment of invention. It is a figure for demonstrating the relationship of the image position before and behind adjustment corresponding to the 2nd Embodiment of invention. It is a flowchart which shows an example of the process corresponding to the 2nd Embodiment of invention. It is a flowchart which shows an example of a process of the control apparatus 602 corresponding to the 2nd Embodiment of invention. It is a functional block diagram which shows an example of the system configuration corresponding to the 3rd Embodiment of invention. It is a conceptual diagram for demonstrating the process which extracts the display area from the captured image corresponding to the 3rd Embodiment of invention. It is a flowchart which shows an example of a process of HMD1001 corresponding to the 3rd Embodiment of invention. The functional block diagram of the video see-through mixed reality system in a prior art example. The schematic diagram of the image which performed the reverse correction for correct | amending the optical distortion in a prior art example.

Explanation of symbols

101 ... HMD (Head Mounted Display)
DESCRIPTION OF SYMBOLS 102 ... Control apparatus 103 ... Image processing apparatus 104 ... Imaging unit 105 ... Packetization process part 106 ... Optical transmission I / F (HMD apparatus)
107 ... Display image adjustment unit 108 ... Display unit 109 ... Optical transmission I / F (control device)
110: I / F conversion unit 111: Communication I / F unit (for captured image transmission)
112 ... Communication I / F part (for display image transmission)
113 ... Optical distortion correction processing unit 114 ... Packetization mask processing unit 115 ... Communication I / F unit (for captured image transmission)
116: Position / attitude information generation unit 117 ... Content DB
118 ... CG rendering unit 119 ... Image composition unit 120 ... Communication I / F unit (for display image transmission)

Claims (13)

A control device that controls transmission of display image data for display on an external display device, and when the display image data to be transmitted is transmitted in transmission packets in line units, the display image data should be transmitted Determination means for determining a line and a line that may not be transmitted based on a predetermined set value;
Packet generation means for generating a transmission packet for each line of the display image data, wherein at least the display image data is transmitted for the line of the display image data determined not to be transmitted by the determination means. A packet generation means for generating a transmission packet including a flag indicating that the display is not performed and a synchronization signal of the line in the header and excluding the display image data;
A control device comprising: transmission means for transmitting the transmission packet generated by the packet generation means to an external display device.   The set value defines at least one of a range in which the display image data is not displayed in a display area of the external display device and a range outside the display area in the display image data. The control apparatus according to 1.   The range in which the display image data is not displayed in the display area is a result of correction processing for canceling optical distortion of the external display device, or a difference in display size between the display image data and the display area. The control device according to claim 2, wherein the control device is determined based on the determination.   The control device according to claim 2 or 3, wherein a range outside the display area in the display image data is determined based on a display shift amount in the external display device. The packet generation means, for the line of the display image data determined to be transmitted by the determination means,
When there is a horizontal offset value based on the shift amount, generating a transmission packet including the offset value and the synchronization signal of the line in a header and including display image data corresponding to the offset value. The control device according to claim 4, wherein A display device,
Receiving means for receiving the transmission packet from the transmitting means of the control device according to any one of claims 1 to 5,
Display means for displaying display image data included in the received transmission packet,
The display means displays Null data in a display area corresponding to the transmission packet when the received transmission packet includes a flag indicating that transmission of display image data is not performed. . Imaging means for imaging real space and generating captured image data;
A position and orientation sensor for measuring the position and orientation of the display device;
Prediction means for predicting the position and orientation of the display device based on the measurement result of the position and orientation sensor;
A determination unit that determines a cut-out position of the captured image data based on a prediction result of the prediction unit;
When transmitting the captured image data in a transmission packet in units of lines, in the captured image data, a line to be transmitted and a line that may not be transmitted are determined based on the cut-out position, and are not transmitted. For the line of the captured image data determined to be acceptable, the header includes at least a flag indicating that transmission of the captured image data is not performed and a synchronization signal of the line, and excludes the captured image data. Packet generation means for generating a packet;
The display device according to claim 6, further comprising transmission means for transmitting the transmission packet to the control device. A control apparatus according to claim 1, a display device according to claim 6 or 7, and a display device in which the receiving means receives the transmission packet from the transmission means of the control device according to claim 1 A mixed reality system that displays the display image data obtained by synthesizing a real space image and a virtual image on the display device. A control method of a control device for controlling transmission of display image data for display on an external display device,
When the determination unit of the control device transmits the display image data to be transmitted in a transmission packet for each line, a line to which the display image data is to be transmitted and a line that may not be transmitted are determined in advance. A determination process for determining based on the set value,
A synchronization between the line indicating that the display image data is not transmitted at least for the line of the display image data determined by the packet generation means of the control device to be not transmitted in the determination step and the line A packet generation step of generating the transmission packet including a signal and the display image data excluding the display image data;
A control method for a control device, comprising: a transmission step for transmitting the transmission packet generated by the packet generation step to an external display device. A display device control method comprising:
A reception step of receiving the transmission packet from the transmission unit of the control device according to any one of claims 1 to 5, wherein the reception unit of the display device;
The display unit of the display device includes a display step of displaying the display image data included in the received transmission packet on the display unit;
In the display step, when the received transmission packet includes a flag indicating that the display image data is not transmitted, null data is displayed in a display area corresponding to the transmission packet. Control method of the device. The display device
Imaging means for imaging real space and generating captured image data;
A position and orientation sensor for measuring the position and orientation of the display device,
The control method is:
A predicting step in which the prediction unit of the display device predicts the position and orientation of the display device based on a measurement result in the position and orientation sensor;
A determination step in which the determination unit of the display device determines a cut-out position of the captured image data based on a prediction result in the prediction step; and a packet generation unit of the display device transmits the captured image data in line units. In the captured image data, the captured image that is determined not to be transmitted is determined based on the cut-out position in the captured image data. A packet generation step for generating a transmission packet excluding the captured image data, including a flag indicating that transmission of the captured image data is not performed and a synchronization signal of the line in the header for each data line;
The display device control method according to claim 10, further comprising: a transmission step of transmitting the transmission packet to the control device.   The computer program for functioning a computer as a control apparatus of any one of Claims 1 thru | or 5.   A computer program for causing a computer to function as the display device according to claim 6.

Images