CN115514949B - Data streaming method and system - Google Patents
Data streaming method and system Download PDFInfo
- Publication number
- CN115514949B CN115514949B CN202110695950.0A CN202110695950A CN115514949B CN 115514949 B CN115514949 B CN 115514949B CN 202110695950 A CN202110695950 A CN 202110695950A CN 115514949 B CN115514949 B CN 115514949B
- Authority
- CN
- China
- Prior art keywords
- data
- streaming
- packets
- time point
- receiving device
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 37
- 230000005540 biological transmission Effects 0.000 claims abstract description 22
- 238000012545 processing Methods 0.000 claims description 33
- 230000002159 abnormal effect Effects 0.000 claims description 9
- 230000004044 response Effects 0.000 claims description 8
- 230000003190 augmentative effect Effects 0.000 claims description 6
- 238000012546 transfer Methods 0.000 claims description 3
- 230000002547 anomalous effect Effects 0.000 claims 1
- 238000012163 sequencing technique Methods 0.000 claims 1
- 230000003993 interaction Effects 0.000 abstract description 18
- 238000010586 diagram Methods 0.000 description 10
- 230000000007 visual effect Effects 0.000 description 7
- 238000013507 mapping Methods 0.000 description 6
- 230000002452 interceptive effect Effects 0.000 description 5
- 230000008569 process Effects 0.000 description 5
- 230000005856 abnormality Effects 0.000 description 3
- 238000005259 measurement Methods 0.000 description 3
- 230000001360 synchronised effect Effects 0.000 description 3
- 238000004891 communication Methods 0.000 description 2
- 238000004590 computer program Methods 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 230000004807 localization Effects 0.000 description 2
- 230000002093 peripheral effect Effects 0.000 description 2
- 238000010276 construction Methods 0.000 description 1
- 230000001934 delay Effects 0.000 description 1
- 230000003111 delayed effect Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 238000003672 processing method Methods 0.000 description 1
- APTZNLHMIGJTEW-UHFFFAOYSA-N pyraflufen-ethyl Chemical compound C1=C(Cl)C(OCC(=O)OCC)=CC(C=2C(=C(OC(F)F)N(C)N=2)Cl)=C1F APTZNLHMIGJTEW-UHFFFAOYSA-N 0.000 description 1
- 230000036962 time dependent Effects 0.000 description 1
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/10—Processing, recording or transmission of stereoscopic or multi-view image signals
- H04N13/106—Processing image signals
- H04N13/139—Format conversion, e.g. of frame-rate or size
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/10—Processing, recording or transmission of stereoscopic or multi-view image signals
- H04N13/106—Processing image signals
- H04N13/167—Synchronising or controlling image signals
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/10—Processing, recording or transmission of stereoscopic or multi-view image signals
- H04N13/194—Transmission of image signals
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N7/00—Television systems
- H04N7/01—Conversion of standards, e.g. involving analogue television standards or digital television standards processed at pixel level
- H04N7/0135—Conversion of standards, e.g. involving analogue television standards or digital television standards processed at pixel level involving interpolation processes
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Two-Way Televisions, Distribution Of Moving Picture Or The Like (AREA)
Abstract
A data streaming method and system. The data streaming system comprises at least one transmitting device and at least one receiving device. The method comprises the following steps: each transmitting device responds to a transmission request of the receiving device for the streaming data, encapsulates the streaming data into a plurality of packets according to a predefined format, and transmits the packets to the receiving device; the receiving device stores the received packet data in the storage device and records the time point of receiving each packet; and when the receiving device runs the streaming data, reading the data of each packet stored in the storage device, and carrying out interpolation operation on the data of each packet according to the time point of each packet to obtain the corresponding data required by the receiving device when the receiving device runs the streaming data at the current time point. The data streaming method and the system can simultaneously consider real-time performance and interactivity of virtual-real interaction.
Description
[ Field of technology ]
The present invention relates to a data processing method and system, and more particularly, to a data streaming method and system.
[ Background Art ]
In Virtual Reality (VR), augmented Reality (Augmented Reality, AR) or Mixed Reality (MR), virtual-real interaction is one of the important application scenarios in Human-computer interaction interfaces (Human-MACHINE INTERFACE). The virtual-real interaction is mainly aimed at interacting with hardware devices such as a head-mounted display (Head Mounted Display, HMD) by a user, and confusing the sense of the user or making the user generate illusion in the interaction process, so as to present the information of the entity which does not exist in the real world such as abstract information, virtual images, virtual objects and the like in front of the eyes of the user.
In the current mainstream algorithm modules or systems, the virtual-real interaction system mostly comprises five main functions of an inertial navigation system (Inertial Navigation System, INS) "," a visual inertial odometer (Visual Inertial Odometry, VIO) "," instant positioning and mapping (Simultaneous localization AND MAPPING, SLAM) "," a gesture tracking module ", and" virtual-real interaction module ", and the purpose is to find out the real scenes corresponding to each other in the virtual space through the inertial navigation system, the visual inertial odometer, and the instant positioning and mapping module, and trigger the events of the field under the virtual space through gesture actions, so as to present the virtual information, the virtual images, the virtual objects, etc. conforming to the field in front of the eyes of the user.
However, because the operation requirements of the modules or systems are different, delays generated in the operation process are not consistent, and dependencies or sequences of processing flows exist among the modules or systems, so that the problem of information asynchronization among the modules or systems can be generated when the operation results of the modules or systems are transferred. Because the modules or the systems have the operation delay, if the modules or the systems are required to realize information synchronization in the process of being input into a user from a sensor (such as a camera, an inertia measurement unit and the like), the operation delay of each module or each system is accumulated, and the whole virtual-real interaction system flow is greatly delayed as a result, so that the experience of the user is reduced.
In detail, since the processing speed and delay time between each input source and each module or system are not equal, it is difficult for the virtual-real interactive module in the last step to synchronize the information. Fig. 1 is a schematic diagram of a conventional virtual-real interactive system. Taking the virtual-real interactive system 10 of fig. 1 as an example, the camera (or video camera) 11 processes a photograph for about 1/30 seconds or about 1/60 seconds, and the inertial measurement unit 12 transmits a value of about 1/200 seconds. With the same hardware and the same computing power, the processing speed of the gesture tracking module 13 is about 1/15 seconds, the processing speeds of the Visual Inertial Odometer (VIO) 14 and the real-time positioning and mapping (SLAM) 16 are about 1/25 seconds to about 1/20 seconds, and the processing speed of the Inertial Navigation System (INS) 15 is about 1/100 seconds. Therefore, the virtual-real interaction module 17 is synchronized based on the processing speed of the slowest gesture tracking module 13 of 1/15 second at the time point of synchronization of each module or system. However, the slowest module has too slow processing speed and the time points of data transmission of each module or system are not synchronous, so that the real-time performance and the interactivity of the virtual-real interaction are poor.
[ Invention ]
The invention provides a data streaming method and a data streaming system, which can simultaneously consider real-time performance and interactivity of virtual-real interaction.
The invention provides a data streaming method, which is suitable for a data streaming system comprising at least one transmitting device and at least one receiving device. The method comprises the following steps: each transmitting device responds to a transmission request of the receiving device for the streaming data, encapsulates the streaming data into a plurality of packets according to a predefined format, and transmits the packets to the receiving device; the receiving device stores the received packet data in the storage device and records the time point of receiving each packet, wherein the plurality of packets comprise a first packet and a second packet, the first packet corresponds to a first time point, and the second packet corresponds to a second time point; and when the receiving device runs the streaming data, reading the data of each stored packet in the storage device, and carrying out interpolation (operation) operation on the data of each packet according to the time point of each packet to obtain corresponding data required by the receiving device when the receiving device runs the streaming data at the current time point, wherein interpolation operation is carried out according to the first packet, the first time point, the second packet and the second time point to obtain corresponding data and a third time point corresponding to the corresponding data, the third time point corresponds to the current time point, and the third time point is later than the first time point and the second time point.
The invention provides a data streaming system, which comprises at least one transmitting device and at least one receiving device. Each transmitting device responds to a transmission request of the streaming data, encapsulates the streaming data into a plurality of packets according to a predefined format, and transmits the packets. The receiving device is connected with each transmitting device and is used for sending a transmission request to each transmitting device, receiving a packet sent by each transmitting device in response to the transmission request, storing the received packet in the storage device and recording the time point of receiving each packet, wherein the plurality of packets comprise a first packet and a second packet, the first packet corresponds to a first time point, and the second packet corresponds to a second time point. When the receiving device runs the streaming data, the data of each stored packet in the storage device is read, interpolation operation is performed on the data of each packet according to the time point of each packet, corresponding data required by the receiving device when the receiving device runs the streaming data at the current time point is obtained, interpolation operation is performed according to the first packet, the first time point, the second packet and the second time point, so as to obtain corresponding data and a third time point corresponding to the corresponding data, the third time point corresponds to the current time point, and the third time point is later than the first time point and the second time point.
In order to make the above features and advantages of the present invention more comprehensible, embodiments accompanied with figures are described in detail below.
[ Description of the drawings ]
Fig. 1 is a schematic diagram of a conventional virtual-real interactive system.
Fig. 2 is a schematic diagram of a data streaming system according to an embodiment of the invention.
FIG. 3 is a schematic diagram illustrating interpolation of one-to-one data streams according to an embodiment of the present invention.
Fig. 4 is a schematic diagram of a data streaming system according to an embodiment of the present invention.
Fig. 5 is a flow chart illustrating a data streaming method according to an embodiment of the invention.
Fig. 6 is a flow chart illustrating a data streaming method according to an embodiment of the invention.
Fig. 7 is a flow chart illustrating a data streaming method according to an embodiment of the invention.
Fig. 8 is a flow chart illustrating a data streaming method according to an embodiment of the invention.
FIG. 9 is a schematic diagram illustrating interpolation of many-to-one data streams according to an embodiment of the present invention.
[ Symbolic description ]
10: Virtual-real interaction system
11: Camera with camera body
12: Inertial measurement unit
13: Gesture tracking module
14: Visual inertial odometer
15: Inertial navigation system
16: Instant positioning and map construction
17: Virtual-real interaction module
20. 40: Data streaming system
21: First module
22: Second module
23: Storage device
42: Conveying device
422. 442: Time counter
424. 444, 92: Storage device
44. 90: Receiving device
A0 to a8, b0 to b5, c0 to c7, a1', a3', a4', a6", a7', a8', b3', b4", b5', c6", c7', T1 to T4: time point
S502 to S506, S602 to S606, S702 to S714, S802 to S806: and (3) step (c).
[ Detailed description ] of the invention
The embodiment of the invention provides that the data processed by each input source and each module are packaged into a streaming mode to communicate information among the modules. The definition of "streaming" in the embodiments of the present invention is: data of a predefined format is transmitted or received in a stable continuous and time dependent manner via an application programming interface (Application Programming Interface, API), universal serial bus (Universal Serial Bus, USB) or network, etc., and the remaining data is allowed to be transmitted while still received.
Fig. 2 is a schematic diagram of a data streaming system according to an embodiment of the invention. Referring to fig. 2, the data streaming system 20 of the embodiment of the invention includes a first module 21 and a second module 22. The direction indicated by the unidirectional arrow in fig. 2 is the direction in which the source (i.e., the first module 21) sends information to the sink (i.e., the second module 22). The present invention adopts a streaming mode to communicate between the first module 21 and the second module 22, and aims to achieve real-time response effect by using an asynchronous mode for streaming data with time dependency.
In detail, in the data streaming system 20, the second module 22 has reserved a buffer space in the storage device 23 to store the packets streamed in by the first module 21. Therefore, when the second module 22 is running, it can run asynchronously, and the unknown data corresponding points are obtained through interpolation (interpolation) operation, so as to improve the performance of the virtual-real interactive system.
FIG. 3 is a schematic diagram illustrating interpolation of one-to-one data streams according to an embodiment of the present invention. Referring to fig. 2 and 3, it is assumed that the processing data of the first module 21 is a 0-a 8 of the stream one, and the processing data of the second module 22 is b 0-b 5 of the stream two. When the second module 22 processes the data at the time point b1, the second module 22 can take out the data at the time points a0 and a1 of the streaming one from the buffer space reserved by the data streaming system 20, and calculate the data at the approximate unknown corresponding point a1' by using interpolation algorithms such as linear interpolation (linear interpolation), polynomial interpolation (polynomial interpolation), spline interpolation (Spline interpolation) or exponential moving average (exponential moving average), so that the second module 22 can obtain the result of the current operation (time point b 1) in real time without waiting for the time point a2 of the first module 21.
Fig. 4 is a schematic diagram of a data streaming system according to an embodiment of the present invention. Referring to fig. 4, a data streaming system 40 according to an embodiment of the present invention includes one or more transmitting devices 42 and one or more receiving devices 44. In some embodiments, the transmitting device 42 and the receiving device 44 are, for example, computer devices such as file servers, database servers, application servers, workstations or personal computers with computing capabilities, or mobile devices such as smart phones, tablet computers, but not limited thereto.
In some embodiments, the data streaming system 40 is, for example, a virtual reality system, an augmented reality system or a mixed reality system, and the transmitting device 42 and the receiving device 44 are, for example, but not limited to, an "inertial navigation system (Inertial Navigation System, INS)", "visual odometer (Visual Inertial Odometry, VIO)", "real-time positioning and mapping (Simultaneous localization AND MAPPING, SLAM)", "gesture tracking module" or "virtual-real interaction module".
In some embodiments, the transmitting device 42 and the receiving device 44 include, for example, any wired or wireless interface devices corresponding to each other that can be used to transfer data between the transmitting device 42 and the receiving device 44. For wired mode, the interface device may be, but not limited to, a universal serial bus (universal serial bus,USB)、EIA-RS-232(Electronic Industry Association-Recommended Standard-232,RS232)、 universal asynchronous receiver/transmitter (universal asynchronous receiver/transmitter, UART), an internal integrated circuit (Inter-INTEGRATED CIRCUIT, I 2 C), a serial peripheral interface (SERIAL PERIPHERAL INTERFACE, SPI), a display port (displayport), or a lightning port (thunderbolt). For wireless manner, the interface device may be a device supporting communication protocols such as wireless fidelity (WIRELESS FIDELITY, wi-Fi), wireless radio frequency identification (Radio Frequency Identification, RFID), bluetooth, infrared, near Field Communication (NFC) or device-to-device (D2D), but is not limited thereto. In some embodiments, the interface device may also be a network card supporting Ethernet (Ethernet) or supporting wireless network standards such as 802.11g, 802.11n, 802.11ac, etc., but is not limited thereto.
In some embodiments, the transmitting device 42 and the receiving device 44 each include a time counter 422 and a time counter 442. The transmitting device 42 uses the time counter 422 to record the corresponding time point of each packet of data in the streaming data. The receiving device 44 records the point in time at which it received each packet using a time counter 442.
In some embodiments, the transmitting device 42 and the receiving device 44 each include a storage device 424 and a storage device 444 to store computer programs and data, such as any type of fixed or removable random access Memory (Random Access Memory, RAM), read-Only Memory (ROM), flash Memory (Flash Memory), hard disk, or the like, or a combination thereof.
In some embodiments, the transmitting device 42 and the receiving device 44 each include a Processor (not shown), such as a central processing unit (Central Processing Unit, CPU), other programmable general purpose or special purpose Microprocessor (Microprocessor), microcontroller (Microcontroller), digital signal Processor (DIGITAL SIGNAL Processor, DSP), programmable controller, application SPECIFIC INTEGRATED Circuits (ASIC), programmable logic device (Programmable Logic Device, PLD), or other similar devices or combinations of these devices, to load computer programs from the storage device 424 and the storage device 444 to perform the data streaming methods of embodiments of the present invention, but the embodiments are not limited thereto.
Fig. 5 is a flow chart illustrating a data streaming method according to an embodiment of the invention. Referring to fig. 4 and fig. 5, the method of the present embodiment is applicable to the data streaming system 40 of fig. 4. The detailed steps of the data streaming method of the present embodiment are described below with respect to each device of the data streaming system 40.
In step S502, each transmitting device 42 encapsulates the streaming data into a plurality of packets according to a predefined format in response to the transmission request of the receiving device 44 for the streaming data, and transmits the packets to the receiving device 44. The predefined format includes the destination transmission device of the packet, the size of the data, and the timestamp of the corresponding time point of the data of the packet in the streaming data. Each transmitting device 42 sets a transmission path of the packet according to the transmission request, and the receiving device 44 sets a receiving path of the received packet. The transmission path and the receiving path may be, for example, the data of the packet transmitted and received by the wired manner and the wireless manner in the above embodiments.
In step S504, the receiving device 44 stores the data of the received packets in the storage device and records the time point when each packet is received.
In step S506, when the receiving device 44 runs the streaming data, the receiving device 44 reads the data of each packet stored in the storage device 444, and performs interpolation (interpolation) operation on the data of each packet according to the time point of each packet, so as to obtain the corresponding data required by the receiving device 44 when the receiving device runs the streaming data at the current time point.
In some embodiments, the receiving device 44 receives packets from only a single transmitting device 42, while in other embodiments, the receiving device 44 receives packets from multiple transmitting devices 42. The receiving device 44 sorts the packets based on the time points when it receives the packets and performs interpolation operation to obtain the corresponding data required for running the streaming data. The following examples are provided in detail.
Fig. 6 is a flow chart illustrating a data streaming method according to an embodiment of the invention. Referring to fig. 4 and 6, in the present embodiment, the receiving device 44 receives packets from only a single transmitting device 42. The transmitting device 42 encapsulates the streaming data into N packets according to a predefined format in response to a transmission request of the receiving device 44 for the streaming data.
In step S602, the receiving device 44 estimates M corresponding data required for running the streaming data according to the average processing frequency of the streaming data.
In step S604, the receiving device 44 reads the received N packets of data from the storage device 444, and sorts the N packets of data according to the time points recorded by the time stamps, wherein M, N is a positive integer.
In step S606, the receiving device 44 performs interpolation operation according to the data of the N packets after the sorting to obtain M corresponding data required for running the streaming data.
Fig. 7 is a flow chart illustrating a data streaming method according to an embodiment of the invention. Referring to fig. 4 and fig. 7, in the present embodiment, after each transmitting device 42 encapsulates the streaming data into each packet, for example, a flag indicating whether it is abnormal is recorded on each packet, which is used as a basis for the subsequent receiving device 44 to perform interpolation operation using the data of the packet. In the present embodiment, the number of the conveying devices 42 may be one or more.
In step S702, each transmitting device 42 encapsulates the streaming data into a plurality of packets according to a predefined format in response to the transmission request of the receiving device 44 for the streaming data.
In step S704, each transmitting apparatus 42 determines whether the standard deviation of the processing frequency of the processed packet is greater than a threshold value or whether the data of the processed packet is NULL (NULL).
In step S706, each transmitting apparatus 42 marks the flag of the packet as abnormal when the standard deviation of the processing frequency of the packet is greater than a threshold value or when the data of the processed packet is empty. In contrast, in step S708, when the data of the processed packet is content and the standard deviation of the processing frequency is less than or equal to the threshold, each transmitting device 42 marks the flag of the packet as normal. The flag is, for example, normally T and the abnormality is F. In detail, if an abnormality occurs in processing data, the transmission device 42 will make the time spent processing the data each time different (the standard deviation of the processing frequency becomes larger), and the transmission device 42 may set the flag of the abnormal packet to F. Or the transmitting device 42 sets the flag of the empty packet to F when an output abnormality occurs, for example, when a certain number of data are empty.
In step S710, each transmitting apparatus 42 transmits the processed packet to the receiving apparatus 44.
Correspondingly, in step S712, the receiving device 44 stores the data of the received packets in the storage device 444 and records the time point when each packet is received.
In step S714, the receiving device 44 reads the flag of each packet when running the streaming data, and records the packet as abnormal without using the flag when performing interpolation operation using the data of the packet.
Fig. 8 is a flow chart illustrating a data streaming method according to an embodiment of the invention. Referring to fig. 4 and 8, in the present embodiment, the receiving device 44 receives packets from a plurality of transmitting devices 42. The plurality of transmitting devices 42 encapsulate the streaming data into a plurality of packets according to a predefined format in response to a transmission request of the receiving device 44 for the streaming data.
In step S802, the receiving device 44 predicts M corresponding data needed for running the streaming data according to the average processing frequency of the streaming data, and divides the timing of processing the M corresponding data into a plurality of sub-timings, wherein M is a positive integer.
In step S804, the receiving device 44 reads the data of the packets received from the plurality of transmitting devices 42 from the storage device 444, and sorts the data of the packets according to the time points recorded by the time stamps.
In step S806, the receiving device 44 performs interpolation operation according to each sub-time by using the data of the plurality of packets after the sorting to obtain M corresponding data required for running the streaming data at each sub-time.
In detail, in Virtual Reality (VR), augmented Reality (AR) or Mixed Reality (MR) applications, many-to-one streaming situations (i.e., multiple modules streaming to the same target module, such as multiple modules or systems streaming to the virtual-real interaction module 17 in fig. 1) are easy to occur. At this time, the receiving device 44 needs to use the packets received from the plurality of different transmitting devices 42 to calculate the corresponding data required for running the streaming data.
For example, fig. 9 is a schematic diagram illustrating interpolation operation of many-to-one data streams according to an embodiment of the present invention. Referring to fig. 9, in the present embodiment, when the situation of many-to-one data streams occurs, the receiving device 90 first arranges the packets stored in the storage device 92 according to their time stamps (e.g. stream one, stream two, …, stream N shown in fig. 9). The receiving device 90 calculates the time point T4 based on the average processing frequency between the previous time points T1 to T3, for example. Since the receiving device 90 needs synchronized stream one, stream two, …, and M data points to stream N, the timing between T3 and T4 is divided into M sub-timings, such as T3 1、T32 and T3 3, on the stream timing. For example, the data at time points a6, a7, a8 of stream one is interpolated to the data at time points a6', a6", a7', a8', the data at time points b3, b4, b5 of stream two is interpolated to the data at time points b3', b4", b5', and the data at time points c5, c6, c7 of stream N is interpolated to the data at time points c5', c6", c7 '. The time points of interpolation of the first stream, the second stream and the third stream N correspond to the sub-timing and the time sequence, for example, the time points a6', b3' and c5' correspond to the sub-timing T3 1, the time points a6", b4' and c6' correspond to the sub-timing T3 2, the time points a7', b4" and c6 "correspond to the sub-timing T3 3, and the time points a8', b5' and c7' correspond to the time sequence T4. By the method, the data corresponding to the sub-time sequence and the time sequence are obtained, so that the effect of synchronizing multiple streams of data in real time can be achieved.
In summary, according to the data streaming method and system of the embodiment of the present invention, the transmission path and the receiving path between the modules are defined in advance, and the unknown data points are obtained by effectively using interpolation (interpolation), so that the devices or modules can operate independently without data synchronization. By applying the data streaming system of the embodiment of the invention to the virtual-real interaction system, the delay of the virtual-real interaction can be improved without affecting the accuracy of the virtual-real interaction system, thereby improving the experience of users.
Although the present invention has been described with reference to the above embodiments, it should be understood that the invention is not limited thereto, but rather may be modified or altered somewhat by persons skilled in the art without departing from the spirit and scope of the invention.
Claims (20)
1. A data streaming method for a data streaming system including at least one transmitting device and one receiving device, the method comprising the steps of:
Each of the at least one transmitting device responds to a transmission request of the receiving device for streaming data, encapsulates the streaming data into a plurality of packets according to a predefined format, and transmits the plurality of packets to the receiving device;
The receiving device stores the received data of the plurality of packets in the storage device and records the time point of receiving each of the plurality of packets, wherein the plurality of packets comprise a first packet and a second packet, the first packet corresponds to a first time point, and the second packet corresponds to a second time point; and
When the receiving device runs the streaming data, the receiving device reads the data of each of the plurality of packets stored in the storage device, and performs interpolation operation on the data of each of the plurality of packets according to the time point of each of the plurality of packets, so as to obtain corresponding data required by the receiving device when the receiving device runs the streaming data at a current time point, wherein the interpolation operation is performed according to the first packet, the first time point, the second packet and the second time point so as to obtain the corresponding data and a third time point corresponding to the corresponding data, the third time point corresponds to the current time point, and the third time point is later than the first time point and the second time point.
2. The data streaming method of claim 1 wherein the predefined format includes a destination transfer device for each of the plurality of packets, a size of data, and a timestamp of a corresponding point in time in the streaming data for the data for each of the plurality of packets.
3. The data streaming method of claim 2, wherein the step of the receiving device running the streaming data further comprises:
Estimating M corresponding data required by running the streaming data according to the average processing frequency of processing the streaming data;
reading the received data of the N packets from the storage device, and sequencing the data of the N packets according to the time points recorded by the time stamp, wherein M, N is a positive integer; and
And performing interpolation operation according to the sequenced data of the N packets to obtain the M corresponding data required by running the streaming data.
4. The data streaming method of claim 2, wherein the step of the receiving device running the streaming data further comprises:
Estimating M corresponding data required by running the streaming data according to the average processing frequency of processing the streaming data, and segmenting and processing the time sequence of the M corresponding data into a plurality of sub-sequences, wherein M is a positive integer;
reading the data of the plurality of packets received from each of the at least one transmitting device from the storage device, and sorting the data of the plurality of packets according to the time points recorded by the time stamps; and
And performing interpolation operation according to each of the plurality of sub-sequences by using the data of the plurality of sequenced packets to obtain the M corresponding data required by running the streaming data in each of the plurality of sub-sequences.
5. The method of claim 1 wherein the step of each of the at least one transmitting device encapsulating the streaming data into each of the plurality of packets further comprises marking each of the plurality of packets with a flag indicating whether it is abnormal.
6. The data streaming method according to claim 5, wherein the flag of the packet is abnormal when a standard deviation of a processing frequency of the transmitting device is greater than a threshold or when data of the packet is empty.
7. The data streaming method of claim 5, wherein the step of the receiving device running the streaming data further comprises:
And reading the mark of each of the plurality of packets, and recording the packets which are abnormal without using the mark when the interpolation operation is performed by using the data of each of the plurality of packets.
8. The data streaming method of claim 1, wherein the interpolation operation comprises a linear interpolation, a polynomial interpolation, a spline interpolation, or an exponential moving average interpolation.
9. The data streaming method of claim 1, wherein the data streaming system comprises a virtual reality system, an augmented reality system, or a mixed reality system.
10. A data streaming system, comprising:
at least one transmitting device, wherein each transmitting device responds to a transmission request of streaming data, encapsulates the streaming data into a plurality of packets according to a predefined format, and sends the plurality of packets;
The receiving device is connected with each at least one transmitting device, sends the transmission request to each at least one transmitting device, receives the plurality of packets sent by each at least one transmitting device in response to the transmission request, stores the received plurality of packets in the storage device and records the time point of receiving each plurality of packets, wherein the plurality of packets comprise a first packet and a second packet, the first packet corresponds to a first time point, and the second packet corresponds to a second time point, and the receiving device is used for receiving the plurality of packets sent by each at least one transmitting device in response to the transmission request, wherein
When the receiving device runs the streaming data, the receiving device reads the data of each of the plurality of packets stored in the storage device, and performs interpolation operation on the data of each of the plurality of packets according to the time point of each of the plurality of packets, so as to obtain corresponding data required by the receiving device when the receiving device runs the streaming data at a current time point, wherein the interpolation operation is performed according to the first packet, the first time point, the second packet and the second time point so as to obtain the corresponding data and a third time point corresponding to the corresponding data, the third time point corresponds to the current time point, and the third time point is later than the first time point and the second time point.
11. The data streaming system of claim 10 wherein the receiving means includes a time counter to record the point in time at which each of the plurality of packets is received.
12. The data streaming system of claim 10 wherein the predefined format includes a destination transfer device for each of the plurality of packets, a size of data, and a timestamp of a corresponding point in time in the streaming data for the data for each of the plurality of packets.
13. The data streaming system according to claim 12 wherein the transmitting device includes another time counter for recording a corresponding point in time of the data of each of the plurality of packets in the streaming data.
14. The data streaming system according to claim 12, wherein the receiving device further comprises means for estimating M corresponding data required for running the streaming data according to an average processing frequency of processing the streaming data, means for reading the received N packets of data from the storage device, means for ordering the N packets of data according to the time points recorded by the time stamps, wherein M, N is a positive integer, and means for performing the interpolation operation according to the ordered N packets of data to obtain the M corresponding data required for running the streaming data.
15. The data streaming system according to claim 12, wherein the receiving device further predicts M corresponding data needed to run the streaming data according to an average processing frequency of processing the streaming data, and segments a timing sequence of processing the M corresponding data into a plurality of sub-sequences, wherein M is a positive integer, reads the data of the plurality of packets received from each of the at least one transmitting device respectively from the storage device, sorts the data of the plurality of packets according to the time points recorded by the time stamps, and performs the interpolation operation according to each of the plurality of sub-sequences by using the sorted data of the plurality of packets to obtain the M corresponding data needed to run the streaming data in each of the plurality of sub-sequences.
16. The data streaming system according to claim 10, wherein each of the at least one transmitting device marks a flag on each of the plurality of packets as to whether it is abnormal.
17. The data streaming system of claim 16 wherein the means for transmitting marks the flag of the packet as abnormal when a standard deviation of a processing frequency of each of the at least one means for transmitting is greater than a threshold or when data of the packet is empty.
18. The data streaming system according to claim 16, wherein the receiving device reads the flag of each of the plurality of packets while running the streaming data and records the packet as anomalous without using the flag when performing the interpolation operation using the data of each of the plurality of packets.
19. The data streaming system of claim 10, wherein the interpolation operation comprises a linear interpolation, a polynomial interpolation, a spline interpolation, or an exponential moving average interpolation.
20. The data streaming system of claim 10, wherein the data streaming system comprises a virtual reality system, an augmented reality system, or a mixed reality system.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110695950.0A CN115514949B (en) | 2021-06-23 | 2021-06-23 | Data streaming method and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110695950.0A CN115514949B (en) | 2021-06-23 | 2021-06-23 | Data streaming method and system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN115514949A CN115514949A (en) | 2022-12-23 |
| CN115514949B true CN115514949B (en) | 2024-07-23 |
Family
ID=84499859
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110695950.0A Active CN115514949B (en) | 2021-06-23 | 2021-06-23 | Data streaming method and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115514949B (en) |
Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JPH06113256A (en) * | 1992-09-25 | 1994-04-22 | Sony Corp | Receiver/reproducing device for digital picture signal |
Family Cites Families (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101917635B (en) * | 2010-08-09 | 2012-06-06 | 杭州爱威芯科技有限公司 | Quick time base correction device for digital composite video signal |
| KR101344459B1 (en) * | 2013-02-05 | 2013-12-26 | 주식회사 지오멕스소프트 | Spatial information system connected to moving picture recording device and operating method thereof |
| KR102044508B1 (en) * | 2018-03-14 | 2019-11-13 | 국방과학연구소 | Method and apparatus for providing navigation data in inertial navigation system |
| KR101948728B1 (en) * | 2018-09-28 | 2019-02-15 | 네이버랩스 주식회사 | Method and system for collecting data |
| US11159645B2 (en) * | 2019-06-21 | 2021-10-26 | Dell Products, L.P. | Adaptive backchannel synchronization for virtual, augmented, or mixed reality (xR) applications in edge cloud architectures |
| CN112855410B (en) * | 2021-01-20 | 2021-11-30 | 昆明理工大学 | Hydraulic generator temperature field visual monitoring system and construction method thereof |
-
2021
- 2021-06-23 CN CN202110695950.0A patent/CN115514949B/en active Active
Patent Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JPH06113256A (en) * | 1992-09-25 | 1994-04-22 | Sony Corp | Receiver/reproducing device for digital picture signal |
Also Published As
| Publication number | Publication date |
|---|---|
| CN115514949A (en) | 2022-12-23 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109949422B (en) | Data processing method and equipment for virtual scene | |
| US20210056998A1 (en) | Cinematic space-time view synthesis for enhanced viewing experiences in computing environments | |
| CN112154669B (en) | Correlation of video stream frame timestamps based on system clock | |
| CN114185829B (en) | Shared resources for multiple communication services | |
| CN111182226B (en) | Method, device, medium and electronic equipment for synchronous working of multiple cameras | |
| US9877064B2 (en) | Systems and methods for efficient event-based synchronization in media file transfer and real-time display rendering between a peripheral system and a host device | |
| CN112835484B (en) | Dynamic display method and device based on operation body, storage medium and electronic equipment | |
| US20190045248A1 (en) | Super resolution identifier mechanism | |
| US20130278728A1 (en) | Collaborative cross-platform video capture | |
| US20170026684A1 (en) | Communications between a peripheral system and a host device in efficient event-based synchronization of media transfer for real-time display rendering | |
| US11375244B2 (en) | Dynamic video encoding and view adaptation in wireless computing environments | |
| US9752892B2 (en) | Methods and systems for acquiring sensor data on a device using multiple acquisition modes | |
| US11816269B1 (en) | Gesture recognition for wearable multimedia device using real-time data streams | |
| CN109074152A (en) | A kind of virtual reality image sending method and device | |
| CN115699096A (en) | Tracking augmented reality device | |
| WO2021083031A1 (en) | Time delay error correction method, terminal device, server, and storage medium | |
| CN115209215A (en) | Video processing method, device and equipment | |
| CN115514949B (en) | Data streaming method and system | |
| CN107407944B (en) | Reference sensor discovery and utilization | |
| TWI786694B (en) | Data streaming method and data streaming system | |
| US20230328306A1 (en) | Wireless projection method, source end device, and destination end device | |
| CN113589926A (en) | Virtual interface operation method, head-mounted display device and computer readable medium | |
| CN113556366A (en) | Multi-sensor data synchronization method and system and electronic equipment | |
| US20170026694A1 (en) | Adaptive selection amongst alternative framebuffering algorithms in efficient event-based synchronization of media transfer for real-time display rendering | |
| CN116033199A (en) | Multi-device audio and video synchronization method and device, electronic device and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |