JP2008539614A - Integrated wireless multimedia transmission system - Google Patents

Abstract

The present invention generally relates to a method and system for wirelessly transmitting data from a source to a monitor in real time. The invention further generally relates to substantially automatic configuration of wireless devices. In an exemplary embodiment, the present invention is a method for capturing video and audio data from a source and wirelessly transmitting the data, reproducing the data, capturing the video data using a mirror display driver A step of capturing the audio data from an input source, a step of compressing the captured audio and video data, and a step of transmitting the compressed audio and video data using a transmitter.
[Selection] Figure 5

Description

The present invention relates generally to methods and systems for transmitting data from a source to a monitor wirelessly and in real time. The present invention also generally relates to substantially automatic configuration of wireless devices.

BACKGROUND OF THE INVENTION Individuals may include, but are not limited to, personal computers, storage devices, mobile phones, personal data assistants, for storage, recording, transmission, reception and playback of media including graphics, text, video, images, and audio. And a computing device including a server. Such media can be obtained from a number of sources including, but not limited to, the Internet, CDs, DVDs, other networks, or other recording devices.

  In particular, individuals can quickly and massively distribute and access media through open networks without always being subject to time, geography, price, content range, and other restrictions. However, an individual is always forced to display the acquired media on a small screen that is not suitable for one or more spectators.

  Despite the rapid growth and flexibility in the use of computing devices to store, record, transmit, receive, and play media, the vast majority of people around the world receive television and audio / video transmissions. Still using as a means. In particular, air, satellite, and cable media transmissions to television still represent the primary means of personal contact and audio / video media experienced by individuals. However, these transmissions are very limited by cost, content range, access time, and geography.

  Given the ubiquity of individual computing devices used to store, record, transmit, receive, and play media, it is important for individuals to acquire media quickly and flexibly and use their own television for the media experience. In order to be possible, it is preferable that the same computing device can be used in conjunction with a very widespread television platform.

  Prior attempts to enable computing devices to integrate with televisions include: a) transforming televisions into networked computing appliances that directly access the Internet and obtain media; b) receive media from computing devices, Focuses on creating a dedicated hardware device for storing and transferring to a television over a wireless connection and / or c) integrating means into the television to allow storage devices such as memory sticks. However, these conventional approaches suffer from substantial modifications to existing equipment, such as replacement of existing computing devices and / or televisions, or the purchase of expensive new hardware.

  In addition, both approaches generally require the use of multiple physical wiring connections to transmit graphics, text, audio and video. Such a physical connection limits the use of the device to a single television, limits the replacement of equipment to a specific area in the home, and results in an unsightly wiring network. Lastly, the requirement to physically store media in a storage element such as a memory stick and then input to a television is not only difficult and not flexible, but also greatly limits the amount of data that can be transferred.

  There is a need for methods, devices, and systems that allow individuals to receive, transmit, store, and play media using existing computing devices and experience media using existing television. It has been. There is also a need to easily and inexpensively transmit media from computing devices to televisions, thus changing the television to a remote monitor. It is further desirable that a number of different standards applicable to the transmission of text, graphics, video and audio be operated by a single, universal wireless media transmission system. Finally, there is a need to automate the configuration of wireless devices for ease of use.

SUMMARY OF THE INVENTION The present invention generally relates to a method and system for transmitting data from a source to a monitor wirelessly and in real time. The invention further generally relates to substantially automatic configuration of wireless devices.

In one embodiment, the present invention is a method for capturing media from a source and wirelessly transmitting the media, the method comprising:
Playing back the media comprising at least audio data and video data on a computing device;
Capturing the video data using a mirror display driver;
Capturing the audio data from an input source;
Compressing the captured audio and video data; and
And transmitting the compressed audio and video data using a transmitter.

  Optionally, the method further includes receiving the media at a receiver, decompressing the captured audio and video data, and extracting the decompressed audio and video data on a display remote from the source. It consists of steps to play. As an option, the transmitter and the receiver establish a connection using TCP, and the transmitter transmits video data using UDP.

  Optionally, the media further comprises graphics and text data, the graphics and text data being captured using a mirror display driver together with the video data described above. Optionally, the method further comprises the step of processing the aforementioned video data using CODEC. As an option, CODEC removes temporal redundancy from video data using motion estimation blocks.

  Optionally, CODEC uses a DCT transform block to convert a frame of video data into 8 * 8 or 4 * 4 block pixels. Optionally, CODEC encodes video content into short words using VLC coding circuitry. Optionally, CODEC uses the IDCT block to convert the spatial frequency of the video data back to the pixel domain. Optionally, CODEC consists of a rate control mechanism for speeding up media transmission.

  In another embodiment, the invention comprises a program stored on a computer readable board for capturing media from at least video data from a source and transmitting the media wirelessly, the program comprising: It comprises a mirror display driver operating in a kernel mode to capture the video data, a CODEC for processing the video data, and a transmitter for transmitting the processed video data.

  Optionally, the program further comprises a virtual display driver. Optionally, the transmitter establishes a connection with the receiver using TCP, and the transmitter transmits a packet of video data using UDP. Optionally, the media further comprises graphic and text data, and the aforementioned mirror display driver captures the graphic and text data together with the aforementioned video data. Optionally, the CODEC consists of motion estimation blocks to remove temporal redundancy from the video data.

  Optionally, the CODEC consists of a DCT block that converts a frame of video data into 8 * 8 or 4 * 4 block pixels. Optionally, CODEC consists of a VLC coding circuit for encoding video content into short words. Optionally, CODEC consists of an IDCT block for converting the spatial frequency of the video data back to the pixel domain. Optionally, CODEC consists of a rate control mechanism to increase the speed of media transmission.

  These and other embodiments are clearly described in the detailed description with reference to the brief description of the drawings.

DETAILED DESCRIPTION The present invention is an integrated wireless system for transmitting media wirelessly from one device to another in real time. The present invention will be described with reference to the following drawings. The embodiments described herein are not intended to limit or limit the invention to the precise form disclosed. They have been chosen to explain the invention and its applications and to enable those skilled in the art to utilize the invention.

  As shown in FIG. 1, a conventional personal computer, desktop, laptop, PDA, mobile phone, game station, set top box, sanitary receiver, DVD player, personal video recorder, or the new system of the present invention A computing device 101, such as another device that operates, communicates with the remote monitor 103 through the wireless network 102. The computing device 101 and the remote monitor 103 preferably further comprise a processing system on chip that can wirelessly transmit and receive graphics, audio, text, and video encoded in multiple standards.

  The remote monitor 103 can be a known television, plasma display device, flat panel LCD, HDD, projector, or any other known electronic display device capable of rendering graphics, audio, and video. The processing system on chip can be integrated into the remote monitor 103 and the computing device 101 or incorporated into a stand-alone device that is in wired communication with the remote monitor 103 or computing device 101. An exemplary processing system on chip is described in application PCT / US2006 / 00622 assigned to the owner of the present application and incorporated herein by reference.

  FIG. 2 illustrates a computing device 200 of the present invention. The computing device 200 includes an operating system 201 that can execute the new software system 202 of the present invention, and a transceiver 203. The operating system 201 is not limited to MS Windows 2000 (registered trademark), MS Windows NT (registered trademark), MS Windows XP (registered trademark), Linux (registered trademark), OS / 2 (registered trademark), and Palm (registered trademark). Can be any operating system, including a trademark-based operating system, a cellphone operating system, an iPOD® operating system, and a MAC OS®.

  Computing device 200 may be used for transmission of graphics, text, video, and audio signals, for example, in particular, IEEE 802.11a, 802.11g, Bluetooth 2.0, HomeRF 2.0, HiperLAN / 2, and Ultra Wideband, and these standards. The media is transmitted using an appropriate wireless standard of any original specification.

  FIG. 3 illustrates a module 300 of the new software system of the present invention. The software 300 includes a module 301 for real-time capture of media, a module 302 for managing a buffer for storing captured media, a CODEC 303 for compressing and decompressing media, and packaging processed media for transmission. The module 304 is made up of.

  In one embodiment, the computing device, whether downloaded from the Internet, streamed in real time from the Internet, transmitted from a cable or satellite station, transferred from a storage device, or other source, Receive media from the source. The media is played on the computing device via a suitable player installed on the computing device. When the media is played on the computing device, the software module 301 captures the data in real time and temporarily stores it in a buffer before sending it to the CODEC. The CODEC 303 compresses this and prepares it for transmission.

  FIG. 4 illustrates the receiver of the present invention. The receiver 404 includes a transceiver 401, a CODEC 402, a display device 403 for rendering video and graphic data, and an audio device 404 for rendering audio data. The transceiver 401 preferably receives the compressed media data through the new transmission protocol utilized in the present invention. In one example, the new transmission protocol is a TCP / UDP hybrid protocol.

  The TCP / UDP hybrid protocol for real-time transmission of packets combines TCP security services with UDP simple and low processing requirements. The content is received by the receiver and sent to the CODEC 402 for decompression. CODEC decompresses the media and prepares video and audio signals to be sent to the display device 403 and speaker 404 for rendering.

  As shown in FIG. 5, the flowchart illustrates exemplary operation of the integrated wireless system of the present invention. The personal computer plays the media using an appropriate media player on its console (step 501). Such media players may include players from Apple (iPod), RealNetworks (RealPlayer), Microsoft (Windows Media Player), or any other media player. it can.

  The software of the present invention captures real-time video directly from the video buffer (step 502). The captured video is then compressed using CODEC (step 503). Similarly, audio is captured using audio software running on the computing device (step 504) and compressed using CODEC.

  In one embodiment, the software of the present invention captures video through execution of a software module consisting of a mirror display driver and a virtual video driver. In one embodiment, the mirror display driver and virtual display driver run on a computer and are installed as kernel mode components of an operating system that hosts the software of the present invention.

  The mirror display driver for the virtual device mirrors the operation of the physical display device driver by mirroring the operation of the physical display device driver. In one embodiment, a virtual display driver is used to capture the content of an “extended desktop” or second display device associated with a computer, while the mirror display driver is a primary display associated with the computer. Used to capture content.

  In use, the operating system renders graphics and video content onto the video memory of the virtual display driver and mirror display driver. Thus, for example, any media being played by a computer using a media player is also rendered on one of these drivers. The software application component of the present invention maps the video memory of the virtual display driver and mirror display driver to the application space.

  Thus, the application of the present invention obtains a pointer to the video memory. The application of the present invention captures real-time images shown on the display (and thus the displayed real-time graphics or video content) by copying the memory from mapped video memory to locally allocated memory. To do.

  In one embodiment, the mirror display driver and the virtual display driver operate in the kernel space of a Microsoft operating system, such as a Windows 2000 / NT compatible operating system. As shown in FIG. 17, a Microsoft Windows framework 1700 for display driver development is illustrated. An application 1701 running on the computer issues a call to a graphic display interface called Win32 GDI (Graphics Display Interface) 1702.

  The GDI 1702 issues a graphic output request. These requests are routed to software running in kernel space, including kernel mode GDI 1705. The kernel mode GDI 1705 provides intermediate support between the kernel mode graphic driver 1706 and the application 1701. The kernel mode GDI 1705 sends these requests to an appropriate miniport 1709 or graphics driver, such as a display driver 1706 or a printer driver (not shown).

  There is a video miniport 1709 corresponding to each display driver (DDI). Miniport driver 1709 is written for one graphic adapter (or family of adapters). Display driver 1706 can be written for any number of adapters that share a common drawing interface.

  This is because the display driver performs drawing while the miniport driver performs operations such as mode setting and provides hardware information to the driver. One or more display drivers can also operate using a specific miniport driver. The active components of this architecture are the Win32-GDI process 1702 and the application 1701. The remaining components 1705 to 1710 are called from the Win32-GDI process 1702.

  The video miniport driver 1709 generally handles operations that interact with other kernel components 1703. For example, operations such as hardware initialization and memory mapping require action by the NT I / O subsystem. The scope of responsibility of the video miniport driver 1709 includes hardware management and source management such as physical device memory mapping. The video miniport driver 1709 is dedicated to video hardware.

  The display driver 1706 uses the video miniport driver 1709 for frequently requested operations such as resource management, physical device memory mapping, ensuring that register outputs do not occur in close intervals, or responding to interrupts. To do. The video miniport driver 1709 is a mode register in interaction with a graphic card of a plurality of hardware types (minimizing hardware type dependency in the display driver), and a video register to the address space of the display driver 1706 Also handles the mapping.

  There are certain functions that the driver writer must perform to write to the miniport. These functions are output to the video port with which the miniport interacts. At the miniport, the driver writer specifies the absolute addresses of the video memory and registers on the video card. These addresses are initially converted to bus-relative addresses and then to virtual addresses in the call process address space.

  The main task of the display driver 1706 is rendering. When an application calls a Win32 function with a device-independent graphics request, the graphics device interface (GDI) 1705 translates these instructions and calls the display driver 1706. The display driver 1706 then translates these requests into commands for the video hardware and illustrates the graphic on the screen.

  The display driver 1706 can directly access the hardware. By default, the GDI 1705 handles standard format bitmap rendering operations on hardware or the like including a frame buffer. Display driver 1706 can be hooked and implemented by hardware for any drawing function for a specific support offer. For less time critical and more complex operations not supported by the graphics adapter, the driver 1706 can push functions back to the GDI 1705, allowing the GDI 1705 to operate.

  In particular, for time critical operation, the display driver 1706 provides direct access to video hardware registers. For example, for some illustration and text operations, a VGA display driver for an x86 system utilizes optimized assembly code to perform direct access to hardware registers.

  Apart from rendering, the display driver 1706 performs other operations such as surface management and palette management. As shown in FIG. 18, a plurality of inputs and outputs between the GDI and the display driver are illustrated. In one embodiment, GDI 1801 issues a DrvEnableDriver command 1810 to display driver 1802. Then, the GDI 1801 issues a DrvEnablePDEV command 1811 to the display driver 1802.

  The GDI 1801 receives the EngCreatePalette command 1812 from the display driver 1802. Then, the GDI 1801 issues a DrvCompletePDEV command 1813 to the display driver 1802. Then, the GDI 1801 issues a DrvEnableSurface command 1814 to the display driver 1802. The GDI 1801 receives the EngCreateDeviceSurface command 1815 from the display driver 1802 and receives the EngModifySurface command 1816 from the display driver 1802.

As shown in FIG. 19, a software architecture for a graphics generation system is illustrated. Software architecture 1900 represents Microsoft's DirectDraw®, which includes the following components:
1. User mode DirectDraw. This is what is loaded and called from the DirectDraw application. This component provides hardware emulation to manage various DirectDraw objects, and provides display memory and display hardware management services.

  2. Kernel mode DirectDraw. This is a system-provided graphics engine that is loaded by the kernel mode display driver. This part of DirectDraw validates the parameters for the driver and makes it easier to implement a more robust driver. Kernel mode DirectDraw also handles synchronization with GDI and all cross-process states.

  3. DirectDraw part of the display driver. This is implemented by the graphics card hardware vendor along with the rest of the display driver. Other parts of the display driver handle GDI and other non-DirectDraw related calls.

  When the DirectDraw 1900 is read, it directly accesses the graphic card through the DirectDraw driver 1902. DirectDraw 1900 calls DirectDraw driver 1902 for supported hardware functions, or calls hardware emulation layer (HEL) 1903 for functions that must be emulated in software. A GDI 1905 call is sent to the driver.

  During initialization and mode conversion, the display driver returns capability bits to DirectDraw 1900. This allows DirectDraw 1900 to access information about available driver functions, their addresses, and display card and driver performance (stretching, transparent bits, display pitch, and other detailed features). .

  Once DirectDraw 1900 has this information, the DirectDraw driver can be used for direct access to the display card without making a GDI call or using the GDI specific portion of the display driver. In order to directly access the video buffer from the application, it is necessary to map the video memory to the call processing virtual address space.

  In one embodiment, the virtual display driver and the mirror display driver are derived from a normal display driver architecture and include a miniport driver and a corresponding display driver. Conventional display drivers include physical devices attached to a PCI bus or AGP slot. Video memory and registers are physically represented on the video card that is mapped to the address space of the GDI process or that captures the application using DirectDraw.

  However, in this embodiment, there is no physical video memory. The operating system represents the presence of a physical device (referred to as a virtual device) and video memory and registers, and is responsible for the allocated memory in main memory. When the miniport of the present invention is loaded, a chunk of memory, such as 2.5 MB, is reserved from nonpaged pool memory. This memory serves as a video memory.

  This memory is then mapped into the virtual address space of the GDI process (in the case of graphics, the application draws operations). The display driver of the present invention requests a pointer to memory, and the miniport returns a pointer to video memory reserved in RAM. Thus, whether the video memory is on RAM or on a video card is transparent to the GDI and Display Device Interface (DDI) (or application in the case of DirectDraw).

  DDI or GDI renders on this memory location. The miniport of the present invention also allocates a separate memory for overlay. Certain applications such as Power DVD, Win DVD, and video playback software use overlay memory for video rendering.

  In one conventional embodiment, rendering is performed by DDI and GDI. While DDI performs device specific operations, GDI provides general device independent rendering operations. The display architecture provides the convenience of making GDI a layer above DDI, and DDI transferring authority to GDI. In one embodiment of the invention, there is no physical device, so there is no device specific operation.

  Therefore, the display driver of the present invention shifts the rendering operation to GDI. The DDI provides a GDI with a video memory pointer, which renders based on the received request from the Win32 GDI process. Similarly, if the present invention is compatible with DirectDraw, the rendering operation is transferred to HEL (Hardware Emulation Layer) by DDI.

  In one embodiment, the present invention comprises a mirror driver that attaches itself to a primary display driver when loaded. Thus, all render calls to the primary display driver are routed to the mirror driver, and any data rendered on the primary display driver's video memory is rendered on the mirror driver's video memory. Thus, the mirror driver is used for duplicating a computer display.

  In one embodiment, the present invention comprises a virtual driver that, when loaded, operates as an extended virtual driver. When the virtual driver is installed, it appears as a second driver in the display properties of the computer and the user turns on the option to extend this display to this display driver.

  In one embodiment, the mirror driver and virtual driver support the following resolutions: 640 * 480, 800 * 600, 1024 * 768, and 1280 * 1024. For each of these resolutions, the driver supports 8, 16, 24, 32 bit color depth and 60 and 75 Hz refresh rates. Rendering on the overlay surface is done in the YUV420 format.

  In one embodiment, the software library is utilized to support capture of a computer display utilizing mirrors or virtual device drivers. When this library is initialized, it maps mirrors in the application space and video memory allocated in the virtual device driver. In the capture function, the library copies the mapped video buffer to the application buffer. Thus, the application has a copy of the computer display in its particular case.

  In order to capture the overlay surface, the library maps the video buffer to application space. In addition, a pointer with the address of the last rendered overlay surface is also mapped into the application space. This pointer is updated in the driver. When rendering on the overlay memory is initiated, the library gets a notification from the virtual display driver.

  The display driver informs the capture library of color key values. After copying the main video memory, the software module, CAPI, copies the last overlay surface rendered using a pointer mapped from the driver space. After extending to the required dimension on the rectangular area of the main video memory where the color key value exists, YUV is converted to RGB, and RGB data is pasted.

  The color key value is a special value pasted onto the main video memory by GDI and represents a region in which data to be copied with an overlay is rendered. For use on computers running the current Windows / NT operating system, overlay can only be applied to extended virtual device drivers, not mirror drivers. The reason is that DirectDraw is automatically disabled when the mirror driver is attached.

Although video and graphics capture methods and systems have been described in detail with respect to the Microsoft operating system, it is clear that similar mirror display driver and virtual display driver approaches are available for computers running on other operating systems.

  In one embodiment, audio is captured through an interface conventionally used by computer-based audio players to reproduce the audio data. In one embodiment, audio is captured using the Microsoft Windows Multimedia API, which is a software module compatible with Microsoft Windows and NT operating systems. The Microsoft Windows Multimedia Library provides an interface to the application and uses the waveOut call to play audio data on the audio device.

  Similarly, an interface for recording audio data from an audio device is provided. The source for the recording device can be a line-in, a microphone, or any other specified source. The application can indicate the format (sampling frequency, sample bits) when attempting to record data. Exemplary steps for audio capture within a Windows / NT compatible operating system computing environment are as follows.

1. The application opens the audio device using the wavelnOpen () function. This indicates the callback function to call when the audio format to record, the size of the audio data to capture at a time, and the specific size to the audio data are available.
2. The application passes the number of empty audio buffers to the windows audio subsystem using the wavelnAddBuffer () call.

3. The application calls wavelnStart () to instruct the start of capture.
4). When audio data of a specific size is available, the Windows audio subsystem passes the audio data passed by the application to one application in the audio buffer and calls the callback function.

5. The application copies the audio data to its local buffer, continues to capture if necessary, and passes an empty audio buffer to the Windows audio subsystem through wavelnAddBuffer ().
6). When the application needs to stop capturing, the application calls wavelnClose ().

  In one embodiment, the stereo mix option is selected for the media playback application and audio is captured on the fly. An audio device generally has the ability to route audio by returning the audio being played back to the output pin to the input pin. Although different systems have different names, they are commonly referred to as “stereo mixes”.

  When the stereo mix option is selected for the playback option, the sound is recorded from the default audio device using the waveln call and played back on the system, such as the audio being played on the system can be captured. You can record everything. Obviously, the particular approach will depend on the performance of the particular audio device being utilized, and those skilled in the art will know how to capture the audio stream in accordance with what has been taught above. It is also clear that the local audio (on the computer) is muted without muting the audio routed to the input pins to prevent parallel playback of audio from the computer and the remote device.

  In another embodiment, a virtual audio driver called a virtual audio cable (VAC) is installed as a normal audio driver that can be selected as the default playback and / or recording device. The VAC feature, by default, routes all audio going to its audio output pin to its input pin. Thus, if VAC is selected as the default playback device, all audio playing on the system goes to the output pin of the VAC and therefore goes to its input pin.

  When any application captures audio from a VAC input pin using a suitable interface, such as the waveln API, then it is possible to capture everything being played on the system. In order to capture audio using VAC, it must be selected as the default audio device. Once VAC is selected as the default audio device, then no audio on the local speaker is heard.

  As described above, the media is wirelessly transmitted simultaneously to the receiver (step 505). A receiver in data communication with the remote monitoring device receives the compressed media data (step 506). The media data is then decompressed using CODEC (step 507). The data is finally rendered on the display device (step 508).

  Any transmission protocol can be used to transmit the media. However, it is preferable to send the synchronized video and audio data streams utilizing a clock or counter separately according to the hybrid TCP / UDP protocol. In particular, a clock or counter column is transferred to indicate that each data stream has been time ordered.

  As shown in FIG. 6, an embodiment of the above-described TCP / UDP hybrid protocol is illustrated. The TCP / UDP hybrid protocol 600 includes a TCP packet header 601 equal to the size of the 20 TCP, 20 IP, and physical layer header, and a UDP packet header 602 equal to the size of the 8 UDP, 20 IP, and physical layer header.

  FIG. 7 is a flow diagram illustrating the functional steps of the TCP / UDP real-time (RT) transmission protocol implemented in the present invention. The transmitter and receiver described previously establish a connection using TCP (step 701), and the transmitter transmits all reference frames using TCP (step 702). Thereafter, to send the rest of the real-time packet, the transmitter uses the same TCP port that was used to establish the connection in step 701 (step 703), but switches to UDP as the transport protocol (step 703). 704).

  While transmitting real-time packets using UDP, the transmitter further checks for the presence of RT packets that have expired transmission. The transmitter discards the expired frame at the transmitter itself between the IP and the MAC (step 705). However, expired reference frames / packets are always transmitted. Thus, the TCP / UDP protocol substantially increases RT traffic and network throughput performance and significantly reduces collisions.

  Additionally, the TCP / UDP protocol is adapted to utilize ACK spoofing as a congestion signaling method for RT transmission over a wireless network. The transmission of RT traffic over the wireless network can be slow. One reason for this is that, conventionally, after the transmission of each data block, the TCP requests the destination / receiver to receive an ACK signal before resuming transmission of the next block or frame.

  IP networks, especially wireless networks, have a high probability of ACK signal loss due to network congestion, especially for RT traffic. Thus, since TCP performs both flow control and congestion control, this congestion control results in disconnection on the wireless network due to a scenario such as non-acceptance of an ACK signal from the receiver.

  To solve the disconnection, in one embodiment, the present invention utilizes ACK spoofing for RT traffic sent to the network. By implementing ACK spoofing, if the receiver does not see any ACK within a certain time interval, the transmitter generates a fake ACK for TCP so that the transmission process can resume.

  In an alternative embodiment, in the case of low quality transmission due to congestion and reduced network throughput, the connection between the transmitter and the receiver is broken and a new TCP connection is opened to the same receiver. This neatly solves the congestion problem associated with the previous connection. It is clear that this transmission method is just one of a plurality of transmission methods, will be used and will be described in the operation example.

  FIG. 9 is a block diagram illustrating CODEC components of an integrated wireless system as shown in FIG. CODEC 800 is a motion estimation block 801 that removes temporal redundancy from streaming content, a DCT block 802 that performs DCT by converting a frame into 8 * 8 block pixels, a VLC coding circuit 803 that encodes the content into shorter words, a spatial frequency Is converted to the pixel domain and converted into an IDCT block 804, and a rate control mechanism 805 for speeding up media transmission.

  By using temporal redundancy between adjacent frames of the video, the motion estimation block 801 is used to compress the video. The algorithm used for motion estimation is preferably a full search algorithm in which each block of the reference frame is compared with the current frame to obtain the best matching block. As its terminology, this full search algorithm takes each point of the search area as a checkpoint and compares all pixels between the block of the current frame with the block corresponding to all checkpoints in the reference frame. Then, the best checkpoint is determined and the motion vector value is acquired.

  For example, FIG. 9 illustrates the functional steps of one embodiment of a motion estimation block. Checkpoints A and A1 shown in the figure correspond to blocks 902 and 904 of the reference frame, respectively. If checkpoint A moves 1 pixel to the lower left, this becomes checkpoint A1. Thus, block 902 becomes block 904 when shifted one pixel to the lower left.

  The comparison technique is performed by calculating the differences in all corresponding pixel image information and summing the absolute values of the differences in the image information. Finally, a sum of absolute values (SAD) is performed. Then, the check point having the smallest SAD among all the check points is determined as the best check point. The block corresponding to the best checkpoint is the block of the reference frame that most closely matches the block of the current frame being encoded. These two blocks obtain a motion vector.

  Returning to FIG. 8, once motion estimation is performed, the picture is encoded using DCT block 802 using a discrete cosine transform (DCT). The DCT coding scheme converts pixels (or error terms) into coefficient sets that correspond to the strength of a particular cosine basis function. This discrete cosine transform (DCT) is generally regarded as the most efficient transform coding technique for video compression and is applied to sampled data such as digital image data compared to continuous waveforms. .

  The transformation transforms an N (point) highly correlated input space vector in the form of pixel rows and columns into an N-point DCT coefficient vector containing rows and columns of DCT coefficients, with high frequency coefficients typically being zero values. Therefore, it is convenient to use DCT for image compression. The energy of the space vector defined by the square value of each element of the vector is saved by the DCT transform. The low frequency and high correlation spatial images are compressed to the lowest frequency DCT coefficient and all symbolic energy is preserved as well.

  Moreover, the human psycho-visual system has low sensitivity to high frequency signals as follows. Accurate reduction of the high frequency DCT coefficient equation results in minimal reduction in received image quality. In one embodiment, the 8 * 8 block resulting from the DCT block is divided by the quantization matrix to reduce the magnitude of the DCT coefficients. In such cases, the information associated with the highest frequency that tends to be less visible to the human eye is removed. The result is rearranged and transmitted to the variable length coding block 803.

  A variable length coding (VLC) block 803 is a statistical coding block that assigns codewords to encoded values. High frequency values of occurrence are assigned to short codewords, and those low occurrences are assigned to long codewords. On average, more frequent short codewords are aligned so that the code string is shorter than the original data.

  VLC coding that generates a code of the finished DCT coefficient value level and the execution length of the number of pixels between the non-zero DCT coefficients generates a high compression code when the number of zero-value DCT coefficients is maximum. Data obtained from the VLC coding block is transferred to the transmitter at an appropriate bit rate. The amount of data transferred per second is known as the bit rate.

  FIG. 10 illustrates an exemplary digital signal waveform and data transfer. The vertical axis 1001 represents voltage and the horizontal axis 1002 represents time. When N represents the bit time of a pulse, the digital waveform has a pulse width N and a period (or repetition) of 2N (eg, the time during which information is transferred). The pulse width N can be any unit time such as nanoseconds, microseconds, picoseconds, and the like.

The maximum data rate that can be transmitted in this way is 1 / N transfer per second, or 1 bit of data (time label amount N) every half cycle. The fundamental frequency of the digital waveform is 1 / 2N hertz. In one embodiment, simplified rate control is employed, which increases the bit rate of the data by 50% compared to the method described above using MPEG2®.
As a result of the shorter time, large chunks of data being transferred to the transmitter are processing in real time.

  The compressed data is transmitted according to the transmission protocol described above and received wirelessly by the receiver. In order to provide motion video performance, compressed video information must be decoded quickly and efficiently. The aspect of the decoding process utilized in the preferred embodiment is the inverse discrete cosine transform (IDCT). Inverse discrete cosine transform (IDCT) transforms transform domain data back to a spatial domain format.

  The size of a commonly used two-timed data block that gives a balance between coding efficiency and hardware complexity is 8 * 8 pixels. The inverse DCT circuit performs an inverse discrete cosine transform of the decoded video signal based on a block-by-block and provides a decompressed video signal.

  As shown in FIG. A diagram of IDCT block processing and selective optimization is shown. The circuit 1100 includes a preprocessing DCT coefficient block (hereinafter referred to as PDCT) 1101, an evaluation coefficient block 1102, a selection IDCT block 1103, a calculation IDCT block 1104, a monitoring frame rate block 1105, and an adjustment IDCT parameter block 1106. In operation, the wirelessly transmitted media received from the transmitter includes various coded DCT coefficients routed to the PDCT block 1101.

  The PDCT block 1101 selectively sets various DCT coefficients to zero values to increase the processing speed of the inverse discrete cosine transform procedure with little or no degradation in video quality. The DCT coefficient evaluation block 1102 receives the preprocessed DCT coefficient from the PDCT 1101. The evaluation circuit 1102 examines the coefficients in the DCT coefficient block before calculating the inverse discrete cosine transform operation. Based on the number of non-zero coefficients, an inverse discrete cosine transform (IDCT) selection circuit 1103 selects an optimal IDCT procedure for coefficient processing.

  The coefficient calculation is performed by the calculation IDCT block 1104. In one embodiment, several inverse discrete cosine transform (IDCT) engines can be selectively enabled and utilized by the selection circuit 1103. In general, coefficients obtained by inverse discrete cosine transform are combined with other data before display. Thereafter, the monitor frame rate block 1105 determines an appropriate frame rate for the video system.

  This determination is made, for example, by reading a system block register (not shown) and comparing the elapsed time with a prestored frame interval corresponding to the desired frame rate. The adjustment IDCT parameter block 1106 then adjusts the parameters including the non-zero coefficient threshold, frequency, and magnitude according to the desired or appropriate frame rate.

ここで、ｘ（ｉ，ｊ）は、空間ドメインｉとｊの８＊８イメージブロックのピクセル値で、X(u,v)は、変換ドメインｕ，ｖ内に８＊８変換ブロックに変換された係数である。C(O)は、１／.ｓｑｒｏｏｔ．２であり、C(u)=C(v)=1である。 The IDCT block described above computes the inverse discrete cosine transform according to the appropriate selected IDCT method. For example, the 8 * 8 forward discrete cosine transform (DCT) is defined by the following equation.

Where x (i, j) is the pixel value of the 8 * 8 image block in spatial domains i and j, and X (u, v) is converted to an 8 * 8 transform block in transform domains u and v. Coefficient. C (O) is 1 / .sqroot. 2 and C (u) = C (v) = 1.

The inverse discrete cosine transform (IDCT) is defined by the following equation.

  8 * 8IDCT is regarded as a combination of 64 orthogonal DCT basis matrices of one basis matrix each having a two-dimensional frequency (v, u). Furthermore, each basis matrix is considered to be a two-dimensional IDCT transform with each single transform coefficient set to 1. Since there are 64 transform coefficients in 8 * 8 IDCT, there are 64 basis matrices.

The IDCT kernel K (v, u) is also called a DCT basis matrix that represents a transform coefficient at a frequency (v, u) according to the following equation.
[Equation 3]
K (v, u) =. Nu. (U) .nu. (V) cos ((2m + 1) .pi.u / 16) cos ((2n + 1) .pi.v / 16)
Where .nu. (U) and .nu. (V) are when u = 0.nu. (U) = 1 / .sqroot.8, when u> 0.nu. (u) = 1 A normalization factor defined as / 2. The IDCT is computed by transforming the coefficients at that location, scaling each kernel, and having the sum of the scaled kernels.

なお、４＊４変換ブロックが利用されることは明らかである。 The spatial domain matrix S is obtained using the following formula.

Obviously, 4 * 4 transform blocks are used.

  As previously discussed, the various media streams can be multiplexed and transmitted as a single stream, while preferably simultaneously transmitting the media data streams separately. As shown in FIG. 12, a block diagram illustrating components of a synchronization media data synchronization circuit of an integrated wireless system is illustrated.

  In the transmitter 1206, the synchronization circuit 1200 includes a buffer 1201 having video and audio media, a first socket 1202 for video transmission, a second socket 1203 for audio transmission, a first counter 1204, a second counter 1205, and The reception end 1213 includes a first receiver 1207 for video data, a second receiver 1208 for audio data, a first counter 1209, a second counter 1210, a mixer 1211, and a buffer 1212.

  In operation, after compression, the audio and video data 1201 buffered by the transmitter 1206 is transmitted separately to the first socket 1202 and the second socket 1203. Counters 1204 and 1205 add the same sequence number to both video and audio data before transmission. In one embodiment, audio data is preferably routed via User Datagram Protocol (UDP) and video data is routed via Transmission Control Protocol (TCP).

  At the receiving end 1213, the UDP protocol and the TCP protocol implemented by the audio receiver block 1208 and the video receiver block 1207 receive audio and video signals. Counters 1209, 1210 determine the sequence numbers from the audio and video signals and provide them to the mixer 1211 to allow accurate mixing of the signals. The mixed data is stored in buffer 1212 and rendered by the remote monitor.

  As shown in FIG. 13, the flow chart illustrates another embodiment of audio and video signal synchronization of the integrated wireless system of the present invention. Initially, the receiver wirelessly receives a stream of encoded video data and encoded audio data (step 1301). Then, to process the video and audio portions of the encoded stream, the receiver ascertains the requested time (step 1302).

  The receiver then compares the processing time of the video portion of the encoded stream with the processing time of the audio portion of the encoded stream to determine a time difference (step 1303). The receiver subsequently establishes which processing time is large (eg, video processing time or audio processing time) (step 1304).

  If the audio processing time is large, the video presentation is delayed by the difference time due to the decision (step 1305), thus synchronizing the decoded video data with the decoded audio data. However, if the video processing time is large, the audio presentation is played back at its fixed rate without being delayed (step 1306).

  By discarding the video frames at regular intervals, the video presentation attempts to catch up with the audio presentation. The data is finally rendered on the remote monitor (step 1307). Thus, the meaning of audio “leading” video is that the video synchronizes itself with the audio.

  In certain embodiments, the decoded video data is substantially synchronized with the decoded audio data. This substantially synchronized means that there can be a small, theoretically measurable difference between the presentation of video data and the corresponding presentation of audio data, but audio Such a small difference between the presentation of video data and video data is unlikely to be noticed by users viewing the video and audio data being performed.

  A typical transport stream is received at a substantially fixed rate. In this case, the delay applied to the video presentation or audio presentation is unlikely to change frequently. Thus, to ensure that the delay currently applied to the video presentation or audio presentation is within a certain threshold (eg, visually or audio not recognizable), the above procedure is performed periodically ( For example, every few seconds or every 30 video frames received. Alternatively, this procedure can be performed for each new frame of video data received from the transport stream.

As shown in FIG. 14, another embodiment of an audio and video synchronization circuit is shown. A synchronization circuit 1400 at the transmission end 1401 includes a buffer 1402 having media data, a multiplexer 1403 for synthesizing media data signals such as graphics, text, audio, and video signals, and media contents for synchronization. It consists of a clock 1404 that provides a time stamp.

  At the receiving end 1405, the separator 1406 uses the clock 1407 to separate the data stream into individual media data streams. The time stamp provided by the clock is useful for audio and video synchronization at the receiving end. The clock is set to the same frequency as the frequency at the receiver. The separated audio and video is routed to the speaker 1408 and the display device 1409 for rendering.

  In one embodiment, the present invention provides a system and method for automatically downloading, installing, and updating the new software of the present invention on a computing device or remote monitor. A software CD is not required to install the software program on the remote monitor, the receiver in the remote monitor, the computing device, or the transmitter in the computing device.

  As an example, a personal computer communicating with a wireless projector is provided and is a general model, but can be applied to any combination of a computing device and a remote monitor. Assume that both the personal computer and the wireless projector are in data communication with a processing system on chip as described above.

  At the start, the wireless projector (WP-AP) executes a script to configure itself as an access point. The WP-AP sets the SSID as QWPxxxxxx when “xxxxxxxx” is the lower 6 bytes of the MAC address of the AP. The WP-AP sets its IP address as 10.0.0.1. The WP-AP starts an HTTP server.

The WP-AP starts the DHCP server with the settings in the next configuration file.
Start address: 10.0.0.3
End address: 10.0.0.254
DNS: 10.0.0.1
Default gateway: 10.0.0.1
[The second and third octets of the address are configurable. ]

  The WP-AP starts a small DNS server that is configured to send back 10.0.0.1 (such as the address of the WP-AP) for any DNS query. When the IP address of the WP-AP changes, the IP address of the response is changed. The default page of the HTTP server includes a small software program such as Java Applet that performs automatic software updater.

In order to make sure that the default page is always accessed with any kind of HTTP request, the HTTP server error page redirects to the default page. This can happen when the default page on the browser has, for example, the following directory:
(Not published for public order and morals violation) isapi / redir.dll? Prd = ie & pver = 6 & = msnhome

  The WP-AP informs its existence as an access point through its system on chip and transmitter. The user computing device includes a transceiver capable of transmitting and receiving information wirelessly according to existing wireless communication protocols and standards. The user's computing device recognizes the presence of the wireless projector as an access point. The user instructs the computing device to join the access point through a graphical user interface known to those skilled in the art.

  After joining the wireless projector access point, the user opens a web browser application on the computing device and allows any URL to be typed into a dialog box or returned to the default URL. The opening of the web browser accesses the default page of the WP-AP HTTP server and causes initialization of the software program (such as Java Applet).

  In one embodiment, the software program checks whether the user's browser supports it and processes the software automatic updater. The remaining examples are described in relation to Java, but it is clear that any software programming language can be used.

  When the browser supports Java, applet checks to see if the software and drivers necessary to perform the media delivery method described here are already installed. When already present, Java Applet compares versions and automatically initializes the installation when the computing device software version is older than the version on the remote monitor.

  When Java is not supported in the browser, the user's web page is redirected to run the installation and represents a prompt for the user to save or execute it. This page also displays instructions on how to save and run the installation. The installation program also checks whether the user has already installed the software and whether the required version has been updated. In this case, it is recommended that the user install Java.

  In the first embodiment, the DNS start address of the WP-AP is 10.0.0.2. The WP-AP executes a DHCP client for Ethernet connection, and obtains IP, gateway, subnet, and DNS addresses from a DHCP server on the local area network. When DHCP is disabled, a static value is used. The installation program installs an application, uninstallation, and driver.

  The application is automatically lunched. At the time of connection, the application acquires the DNS address of the WP-AP Ethernet port and sets it on the local machine. After establishing the connection, the WP-AP enables IP forwarding and sets the firewall as follows. This forwards only packets from connected applications to Ethernet and vice versa. These settings allow the user to access the WP-AP's Ethernet local area network and the Internet.

  The firewall verifies that only users connected to the WP-AP can access the LAN / Ethernet. When disconnected, the WP-AP disables IP forwarding and restores the firewall settings. An application running on the user system sets the DNS setting to 10.0.0.1. At the end of the application, the DNS settings are set to DHCP.

  In other embodiments, during installation, the user is prompted to choose whether the computing device behaves as a gateway. Depending on the response, appropriate drivers, software and scripts are installed.

  Now, as shown in FIG. 15, another exemplary configuration for automatic downloading and updating of the software of the present invention is shown. The wireless projector access point has an IP address set to 10.A.B.1, and the gateway system has an IP address set to 10.A.B.2. Here, the octets of A and B can be changed by the user.

  The WP-AP is booted. The user's computing device scans for available wireless networks and selects QWPxxxxxx. The wireless configuration of the computing device must have a valid automatic TCP / IP configuration. For example, the options “Obtain IP address automatically” and “Obtain DNS server address automatically” must be checked. The computing device automatically takes an IP address from an address from 10.0.0.3 to 10.0.0.254. The default gateway and DNS are set as 10.0.0.1.

  The user opens a browser and the software updater is automatically started when Java is supported. When Java is not supported, the user is prompted to save the installation and runs this manually. If the computing device does not act as a gateway to a network such as the Internet, during installation, the user selects “No” for the gateway option.

  The installation executes the script and sets DNS to 10.0.0.2. The next DNS query should be redirected appropriately. Application links are created on the desktop. A user executes an application. The application starts sending the correct content of the user's screen to the projector. If requested, the user can change the WP-AP configuration (SSID, channel, IP address settings: second and third octets can be changed to 10.0.0.x).

  If the computing device acts as a gateway to a network such as the Internet, during installation, the user selects the gateway option “Yes” when prompted. Installation enables Internet sharing (IP forwarding) over Ethernet interface (sharing is an option within the network interface pro party for both Windows 2000 and Windows XP), and the system's wireless interface IP to 10.0. Set to 0.2, set system radio interface netmask to 255.255.255.0, and set system radio interface gateway to 10.0.0.1.

  Application links are created on the desktop. The user executes the application as follows. The application starts sending the correct content of the user's screen to the projector. If requested, the user can change the WP-AP configuration (SSID, channel, IP address settings: second and third octets can be changed to 10.0.0.x).

  Obviously, the present invention enables real-time transmission of media from a computing device to one or more remote monitoring devices or other computing devices. As shown in FIG. 16, another arrangement of the integrated wireless multimedia system of the present invention is shown. In this particular embodiment, communication between a transmitter 1601 and a plurality of receivers 1602, 1603, 1604 is illustrated.

  The transmitter 1601 wirelessly transmits media to a receiver that is integrated or in data communication with multiple devices 1601, 1602, and 1603 for real-time rendering. In an alternative embodiment, the software described above can be utilized for both mirror capture mode and extended mode. In the mirror capture mode, real-time streaming of content is performed on the same content displayed at both the transmission end and the reception end. However, in enhanced mode, the user can work on some other application on the sender side, and transmission can continue as a back-end process.

  The above examples are merely illustrative of the many applications of the system of the present invention. Although only a few embodiments of the present invention have been described herein, it can be implemented in other specific forms without departing from the spirit and scope of the present invention. For example, other configurations of transmitters, networks, and receivers can be used within the scope and spirit of the present invention. Accordingly, the examples and embodiments are to be regarded as illustrative rather than restrictive, and the invention is not limited to the details shown, but may be modified within the scope of the appended claims.

FIG. 1 is a block diagram of an integrated wireless media transmission system of the present invention. FIG. 2 is a diagram illustrating components of a transmitter according to an embodiment of the present invention. FIG. 3 is a diagram illustrating a plurality of software modules according to one embodiment of the software implementation of the present invention. FIG. 4 is a diagram illustrating the components of the receiver according to one embodiment of the present invention. FIG. 5 is a flowchart illustrating an exemplary operation of the present invention. FIG. 6 is a diagram illustrating one embodiment of the TCP / UDP RT hybrid protocol header structure of the present invention. FIG. 7 is a flowchart illustrating exemplary functional steps of the TCP / UDP RT transmission protocol of the present invention. FIG. 8 is a block diagram illustrating an exemplary CODEC utilized in the present invention. FIG. 9 is a functional diagram of an exemplary motion estimation block utilized in the present invention. FIG. 10 is a diagram illustrating one embodiment of a digital signal waveform and corresponding data transfer. FIG. 11 is a diagram illustrating a block diagram of exemplary video processing and selective optimization of an IDCT block of the present invention. FIG. 12 is a block diagram illustrating components of a synchronization circuit for synchronizing audio and video data according to the present invention. FIG. 13 is a flowchart illustrating another embodiment for synchronizing audio and video signals of the present invention. FIG. 14 is a diagram illustrating another embodiment of the audio and video signal synchronization circuit of the present invention. FIG. 15 is a diagram illustrating an enterprise configuration for automatic software download and updater according to the present invention. FIG. 16 is a schematic diagram illustrating communication between a transmitter and a plurality of receivers. FIG. 17 is a block diagram of a Microsoft Windows framework for display driver development. FIG. 18 is a block diagram illustrating the interaction between the GDI and the display driver. FIG. 19 is a diagram illustrating a block diagram of the DirectDraw architecture.

Claims (19)

In a method for capturing media from a source and transmitting the media wirelessly,
Playing the media comprising at least audio data and video data on a computing device;
Capturing the video data using a mirror display driver;
Capturing the audio data from an input source;
Compressing the captured audio and video data; and
A method for capturing media and transmitting wirelessly, comprising: transmitting the compressed audio and video data using a transmitter. The method of capturing and wirelessly transmitting media according to claim 1.
Receiving the media at a receiver, decompressing the captured audio and video data, and playing the decompressed audio and video data on a remote display from the source. A media capture method and a wireless transmission method. The media capture and wireless transmission method according to claim 2,
The method of capturing and wirelessly transmitting media, wherein the transmitter and the receiver establish a connection using TCP, and the transmitter transmits a packet of video data using UDP. The method of capturing and wirelessly transmitting media according to claim 1.
The media further comprises graphic and text data, and
The method of capturing and wirelessly transmitting media, wherein the graphic and text data are captured together with the video data using a mirror display driver. The method of capturing and wirelessly transmitting media according to claim 1.
A method of capturing media and transmitting wirelessly, comprising: processing the video data using CODEC. The method of capturing and wirelessly transmitting media according to claim 1.
The CODEC removes temporal redundancy from the video data using a motion estimation block. A media capture and wireless transmission method according to claim 1, wherein: The method of capturing and wirelessly transmitting media according to claim 1.
The CODEC uses a DCT block to convert a frame of video data into 8 * 8 block pixels. A method for capturing and wirelessly transmitting media. The method of capturing and wirelessly transmitting media according to claim 1.
The CODEC encodes video content into a short word using a VLC coding circuit. A method for capturing and wirelessly transmitting media. The method of capturing and wirelessly transmitting media according to claim 1.
The CODEC uses an IDCT block to convert the spatial frequency of the video data back to the pixel domain and convert it. A method for capturing and wirelessly transmitting media. The method of capturing and wirelessly transmitting media according to claim 1.
The CODEC comprises a rate control mechanism for improving the speed of media transmission. A program stored on a computer-readable board for capturing media consisting of at least video data from a source and transmitting the media wirelessly,
A mirror display driver operating in kernel mode to capture the video data;
A CODEC for processing the video data; and
A program for media capture and wireless transmission comprising a transmitter for transmitting the processed video data. In the program for media capture and wireless transmission according to claim 1,
A media capture and wireless transmission program characterized by comprising a virtual display driver. In the program for media capture and wireless transmission according to claim 1,
The transmitter establishes a connection with a receiver using TCP, and the transmitter transmits a packet of video data using UDP. A program for media capture and wireless transmission characterized in that . In the program for media capture and wireless transmission according to claim 1,
The media further comprises graphic and text data,
The mirror display driver captures graphics and text data together with the video data. A program for capturing media and transmitting wirelessly. In the program for media capture and wireless transmission according to claim 1,
The CODEC comprises a motion estimation block for removing temporal redundancy from the video data. A program for capturing media and transmitting wirelessly. In the program for media capture and wireless transmission according to claim 1,
The CODEC comprises a DCT block for converting a frame of video data into 8 * 8 block pixels. A program for capturing media and transmitting wirelessly. In the program for media capture and wireless transmission according to claim 1,
The CODEC comprises a VLC coding circuit for encoding video content into short words. A program for media capture and wireless transmission characterized in that the codec is a VLC coding circuit. In the program for media capture and wireless transmission according to claim 1,
The CODEC comprises an IDCT block for converting the spatial frequency of the video data back to the pixel domain, and a program for media capture and wireless transmission. In the program for media capture and wireless transmission according to claim 1,
The CODEC comprises a rate control mechanism for improving the speed of media transmission. A program for media capture and wireless transmission characterized by the above.

Images