CN103136793A - Live-action fusion method based on augmented reality and device using the same - Google Patents

Abstract

The present invention relates to a real-scene fusion method and device based on augmented reality. The device includes: an image processor, a camera and a display, wherein the image data output end of the camera is connected with the data input end of the image processor, and the output end of the image processor is connected to the There is a display; the method is: use the camera to collect the video data of the real scene, input it into the image processor for processing; convert the above video data into continuous single-frame images, and convert each frame of image into graphic texture data; use three-dimensional graphics The engine maps this texture data to a rectangular surface that exceeds the distance of all virtual scenes; in the 3D graphics engine, the above rectangular surface and the virtual scene generated by the image processor are simultaneously rendered to form an image video that combines virtual and real. The invention can greatly improve the sense of reality of the simulation training of the observation, aiming and tracking devices, and extend the simulation training of the viewing, aiming and tracking devices from indoor to outdoor.

Description

A real-scene fusion method and device based on augmented reality

technical field

The invention relates to an image augmented reality vision technology, in particular to a real scene fusion method and device based on augmented reality.

Background technique

At present, the simulation training equipment of various sighting, aiming and tracking devices used in large quantities is mainly based on virtual reality technology, and pure virtual two-dimensional or three-dimensional scenes are generated by computers. There are very obvious differences in the scenery, and the scene fidelity is poor. In addition, due to the use of pure virtual scenes, most of these equipment are used indoors and cannot be combined with other outdoor training, which limits the scope of use of training equipment.

Contents of the invention

Aiming at the deficiencies in the visual simulation effect of the simulation training equipment in the prior art, such as poor scene fidelity and inability to combine with other outdoor training, the technical problem to be solved by the present invention is to provide a real, easy-to-use, outdoor Application, a real-scene fusion method and device of an augmented reality closer to an actual combat environment.

In order to solve the problems of the technologies described above, the technical solution adopted in the present invention is:

The real-scene fusion device based on augmented reality of the present invention includes: an image processor, a camera and a display, wherein the image data output end of the camera is connected with the data input end of the image processor, and the output end of the image processor is connected with a display.

The camera adopts an industrial color digital camera, the scanning method is progressive scanning, the width and height ratio of the photosensitive element is 4:3, it has an IEEE-1934b output interface, and can output in RGB format with a resolution of not less than 640×480 pixels and a refresh rate of Video data not lower than 25fps; shutter and gain controllable, with automatic exposure control.

The display adopts a binocular glasses display with a resolution of not less than 640×480 pixels, a refresh rate of not less than 60Hz, a color depth of not less than 16 bits, and a VGA signal input interface.

The present invention is based on the augmented reality real scene fusion side and comprises the following steps:

Use the video camera to collect the video data of the real scene, and input it into the image processor for processing;

Convert the above video data into continuous single-frame images, and convert each frame of images into graphic texture data;

Use a 3D graphics engine to map this texture data onto a rectangular surface that exceeds the distance of all virtual scenes;

In the three-dimensional graphics engine, the above-mentioned rectangular surface and the virtual scene generated by the image processor are simultaneously rendered to form an image video combining virtual and real.

The process of using the 3D graphics engine to map this texture data to a rectangular surface that exceeds the distance of all virtual scenes is: use the 3D graphics engine to draw a rectangular surface with the same size as the FOV section of the 3D graphics engine in a space that exceeds the distance of all virtual scenes ; Use the 3D graphics engine to perform texture filtering on the texture data; bind the texture data to the above-mentioned rectangular surface.

The camera collects the video data of the real scene using a multi-threaded collection method, and opens up two asynchronously executed collection threads for each video that needs to be collected.

The beneficial effects of the present invention are:

1. It can greatly improve the sense of reality of the simulation training of observation, aiming and tracking devices.

At present, the vast majority of computer vision simulation training equipment for various types of sighting and tracking devices are purely virtual two-dimensional or three-dimensional scenes generated by computers. There are very obvious differences in the actual scenery, and the scene fidelity is poor. The invention combines the real outdoor scene with the virtual scene generated by the computer, and the main body of the scene is the real outdoor scene, which greatly improves the realism of the training scene.

2. Extend the simulation training of sight-seeing and tracking devices from indoor to outdoor

Since most of the current simulation training equipment for sight-seeing and tracking devices use purely virtual training scenes, most of them can only be used indoors. The device of the invention can be applied outdoors, which not only expands the use range of the simulation training device, but also enables the simulation training to be combined with other outdoor training subjects, which helps to improve the comprehensive skills of trainees.

Description of drawings

Fig. 1 is a structural block diagram of the composition of the device of the present invention;

Fig. 2 is a flowchart of the real scene fusion method of the present invention;

Fig. 3 is a schematic diagram of the CVid class framework in the method of the present invention;

Figure 4 is a schematic diagram of the CVid class model.

Detailed ways

The present invention will be described in further detail below in conjunction with the accompanying drawings.

As shown in Figure 1, it is the composition structure of the real-scene fusion device based on augmented reality of the present invention, including image processor, camera and display, wherein the image data output end of camera is connected with the data input end of image processor, and the image processor's The output terminal is connected with a display.

In this embodiment, the image processor is a high-performance computer, connected to a color digital camera through an IEEE-1394b interface, and connected to a binocular glasses display through a VGA interface. A graphics engine is a switching platform or software environment based on a certain graphics library (such as OpenGL, DirectX, etc.), running on a computer. The camera adopts industrial color digital camera, the scanning method is progressive scanning, the aspect ratio of the photosensitive element is 4:3, it has an IEEE-1934b output interface, and can output in RGB format with a resolution of not less than 640×480 pixels and a refresh rate of not low Video data at 25fps. Shutter and gain controllable, with automatic exposure control (automatic shutter method, automatic gain method, can control exposure by maximum gain method). Present embodiment adopts the Flea2 color digital video camera that Point Gray Research company produces. When installed, the optical axis of the digital color camera coincides with the aiming line of the analog sighting and tracking device. The binocular glasses display is an eye display with a resolution of not less than 640×480 pixels, a refresh rate of not less than 60Hz, a color depth of not less than 16 bits, and a VGA signal input interface. The binocular glasses display used in this embodiment has a resolution of 800×600, a 32-bit color depth, a refresh rate of 60 Hz, and a VGA signal input interface.

The real scene fusion method based on augmented reality of the present invention comprises the following steps:

Use the video camera to collect the video data of the real scene, and input it into the image processor for processing;

Convert the above video data into continuous single-frame images, and convert each frame of images into texture data;

Use a 3D graphics engine to map this texture data onto a rectangular surface that exceeds the distance of all virtual scenes;

In the three-dimensional graphics engine, the above-mentioned rectangular surface and the virtual scene generated by the image processor are simultaneously rendered to form an image video combining virtual and real.

The process of using the 3D graphics engine to map this texture data to a rectangular surface that exceeds the distance of all virtual scenes is: use the 3D graphics engine to draw a rectangular surface with the same size as the FOV section of the 3D graphics engine in a space that exceeds the distance of all virtual scenes ; Use the 3D graphics engine to perform texture filtering on the texture data; bind the texture data to the above-mentioned rectangular surface.

The camera collects the video data of the real scene using a multi-threaded collection method, and opens up two asynchronously executed collection threads for each video that needs to be collected.

As shown in Figure 2, the implementation process of the real scene fusion method in this embodiment is as follows:

The video data from the digital camera is processed by the CVid class object into a sequence of texture data.

As shown in Figure 3, it is a schematic diagram of the CVid class framework. The CVid class object is built on the ARToolkit library and DSVL library to realize the functions of video capture and converting single-frame video into texture data. The CVid class object mainly realizes the continuous acquisition of camera video and the acquisition of single frame image.

As shown in Figures 2 and 4, it is a model of the CVid class. The constructor CVid() gets the .xml file describing the camera settings. RunCapture() reads these settings and creates a capture buffer, creates a capture thread and starts capturing video. The high-performance acquisition method is an asynchronously executed multi-threaded acquisition method, which opens up two asynchronously executed acquisition threads for each video to be acquired to ensure continuous and uninterrupted video acquisition. Among them, DummyThreadProc (LPVOID lpParameter) is a collection thread specially developed. In order to ensure the performance of the collection process, two DummyThreadProc threads have been opened up in order to be able to collect video without interruption. SceneBlend() is responsible for blending the texture data of the real scene with the virtual scene. The specific processing process is as follows: call the VideoGetImage() method of the CVid class object to obtain the current frame of the video, and save it in the global variable; the obtained frame of video Become texture; Utilize the three-dimensional graphics engine to draw a rectangular surface just identical with the FOV section size of the three-dimensional graphics engine on the space (the present embodiment is set to the space at 10,000 meters) beyond the distance of all virtual scenery; Utilize the three-dimensional graphics engine Perform texture filtering on the texture data; bind the texture data on the above rectangular surface and mix it with the virtual scene. Call the VideoGetNext() method of the CVid object to get the next frame of image. After SceneBlend() is called and executed, the drawn rectangle and mapped texture enter the frame buffer, and then the 3D graphics engine performs subsequent operations such as depth testing and rendering, and finally generates a frame of virtual-real graphics. This process loop is executed cyclically in the main loop of the simulation software to form a virtual-real video, which is finally sent to the binocular glasses display.

Claims (6)

1. outdoor scene fusion method based on augmented reality is characterized in that comprising the following steps:

Utilize the video data of the true scenery of camera acquisition, be input in image processor and process;

Above-mentioned video data is changed into continuous single-frame images, each two field picture is converted to the graphical textures data;

Utilize the three-dimensional picture engine that this data texturing is mapped on rectangular surfaces over all virtual scene distances;

In the three-dimensional picture engine, the virtual scene of above-mentioned rectangular surfaces and image processor generation is played up simultaneously, formed the image/video of actual situation combination.

2. a kind of outdoor scene fusion method based on augmented reality according to claim 1 is characterized in that: utilize the three-dimensional picture engine that this data texturing is mapped to over the process on the rectangular surfaces of all virtual scene distances to be: utilize the three-dimensional picture engine to draw a rectangular surfaces identical with the FOV cross-sectional sizes of three-dimensional picture engine on the space of all virtual scene distances surpassing; Utilize the three-dimensional picture engine to carry out texture filtering to data texturing; This data texturing is bundled on above-mentioned rectangular surfaces.

3. a kind of outdoor scene fusion method based on augmented reality according to claim 1, it is characterized in that: the video data of the true scenery of camera acquisition adopts the multithreading acquisition method, and the video that need to gather for each road is opened up the collecting thread of two asynchronous executions.

4. outdoor scene fusing device based on augmented reality, it is characterized in that comprising: image processor, video camera and display, wherein the view data output terminal of video camera is connected with the data input pin of image processor, and the output terminal of image processor is connected to display.

5. by the outdoor scene fusing device based on augmented reality claimed in claim 4, it is characterized in that: described video camera adopts the industry color digital camera, scan mode is for lining by line scan, the photo-sensitive cell wide high proportion is 4: 3, have the IEEE-1934b output interface, can be not less than the video data that 640 * 480 pixels, refresh rate are not less than 25fps with the rgb format output resolution ratio; Shutter is controlled with gain, with auto-exposure control.

6. by the described outdoor scene fusing device based on augmented reality of claim 4, it is characterized in that: described display adopts the binocular-type glasses display, and resolution is not less than 640 * 480 pixels, and refresh rate is not less than 60Hz, color depth is not less than 16, has the VGA signal input interface.

Images