[go: up one dir, main page]
More Web Proxy on the site http://driver.im/

CN114584598A - Image processing method, device and equipment and readable storage medium - Google Patents

Image processing method, device and equipment and readable storage medium Download PDF

Info

Publication number
CN114584598A
CN114584598A CN202011277952.XA CN202011277952A CN114584598A CN 114584598 A CN114584598 A CN 114584598A CN 202011277952 A CN202011277952 A CN 202011277952A CN 114584598 A CN114584598 A CN 114584598A
Authority
CN
China
Prior art keywords
image
cache
rendered
frame image
frame
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202011277952.XA
Other languages
Chinese (zh)
Inventor
滕瑞
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
China Mobile Communications Group Co Ltd
China Mobile Communications Ltd Research Institute
Original Assignee
China Mobile Communications Group Co Ltd
China Mobile Communications Ltd Research Institute
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by China Mobile Communications Group Co Ltd, China Mobile Communications Ltd Research Institute filed Critical China Mobile Communications Group Co Ltd
Priority to CN202011277952.XA priority Critical patent/CN114584598A/en
Publication of CN114584598A publication Critical patent/CN114584598A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/01—Protocols
    • H04L67/06—Protocols specially adapted for file transfer, e.g. file transfer protocol [FTP]
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/01—Protocols
    • H04L67/10—Protocols in which an application is distributed across nodes in the network
    • H04L67/1097—Protocols in which an application is distributed across nodes in the network for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/10—Processing, recording or transmission of stereoscopic or multi-view image signals
    • H04N13/106—Processing image signals
    • H04N13/161—Encoding, multiplexing or demultiplexing different image signal components
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/10—Processing, recording or transmission of stereoscopic or multi-view image signals
    • H04N13/194—Transmission of image signals
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/50—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding
    • H04N19/597—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding specially adapted for multi-view video sequence encoding

Landscapes

  • Engineering & Computer Science (AREA)
  • Signal Processing (AREA)
  • Multimedia (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Image Analysis (AREA)

Abstract

The invention discloses an image processing method, an image processing device, image processing equipment and a readable storage medium, relates to the technical field of image processing, and aims to solve the problem of resource waste. The method comprises the following steps: acquiring a VR frame image to be rendered; searching a first-level cache, and determining whether a first cache image exists or not, wherein the similarity between the first cache image and a VR frame image to be rendered meets a first preset requirement; under the condition that the first-level cache does not have the first cache image, acquiring a second cache image from a second-level cache, wherein the similarity between the second cache image and the VR frame image to be rendered meets a second preset requirement; obtaining a difference image according to the VR frame image to be rendered and the second cache image; sending the difference image to a terminal; wherein the first-level cache and the second-level cache are respectively used for storing rendered VR frame images. The embodiment of the invention can save resources of the edge cloud.

Description

Image processing method, device and equipment and readable storage medium
Technical Field
The present invention relates to the field of image processing technologies, and in particular, to an image processing method, an image processing apparatus, an image processing device, and a readable storage medium.
Background
When the mobile terminal load cloud technology is applied to VR (Virtual Reality) application, a service is independently provided for each user, that is, each VR frame of a user is independently rendered by an edge cloud according to a display requirement or a fixed frame rate, for example, a screen refresh rate is 60FPS, and the edge cloud must ensure that one frame of rendering is completed in a 16.67ms process and is completely transmitted to the mobile terminal.
However, due to the limited computing capacity of the edge cloud, when the number of users increases, the existing scheme causes the load of the edge cloud to be excessive, and causes the resource waste.
Disclosure of Invention
The embodiment of the invention provides an image processing method, an image processing device, image processing equipment and a readable storage medium, and aims to solve the problem of resource waste.
In a first aspect, an embodiment of the present invention provides an image processing method, applied to an edge cloud, including:
acquiring a VR frame image to be rendered;
searching a first-level cache, and determining whether a first cache image exists or not, wherein the similarity between the first cache image and a VR frame image to be rendered meets a first preset requirement;
under the condition that the first-level cache does not have the first cache image, acquiring a second cache image from a second-level cache, wherein the similarity between the second cache image and the VR frame image to be rendered meets a second preset requirement;
obtaining a difference image according to the VR frame image to be rendered and the second cache image;
sending the difference image to a terminal;
wherein the first-level cache and the second-level cache are respectively used for storing rendered VR frame images.
Wherein the obtaining the second cache image from the second level cache comprises:
acquiring the pose of a target camera of the VR frame image to be rendered;
searching the second-level cache, and acquiring a third cache image from the second-level cache;
and obtaining the second cache image according to the third cache image.
Wherein obtaining the second cache image according to the third cache image comprises:
if the distortion distance between the camera pose of the third cache image and the target camera pose is 0, taking the third cache image as the second cache image;
and if the distortion distance between the camera pose of the third cache image and the target camera pose is greater than 0 and less than or equal to a preset threshold value, performing image distortion on the third cache image to obtain a second cache image.
Wherein the obtaining a difference image according to the VR frame image to be rendered and the second cache image comprises:
obtaining the pixel value of the difference image in the ith channel according to the following mode:
Figure BDA0002779737380000021
wherein, Idelta(i) I-channel value, I, representing the difference imagel(i) Representing I-channel value, I, of the VR frame image to be renderedc(i) And i-channel values representing the second buffer image, i ═ Y, U, V.
Wherein the method further comprises:
and sending the first cache image to the terminal under the condition that the first-level cache has the first cache image.
Wherein the method further comprises:
rendering the VR frame image to be rendered under the condition that the second-level cache does not have the second cache image, and sending the rendered VR frame image to the terminal;
and storing the rendered VR frame image to the second-level cache.
In a second aspect, an embodiment of the present invention further provides an image processing method, applied to a terminal, including:
acquiring a difference image from an edge cloud, wherein the difference image is acquired by the edge cloud according to a VR frame image to be rendered and a rendered VR frame image cached by the edge cloud;
acquiring a VR frame image at the current moment;
and obtaining a VR frame image to be displayed according to the VR frame image at the current moment and the difference image.
Obtaining a VR frame image to be displayed according to the VR frame image at the current moment and the difference image, wherein the obtaining of the VR frame image to be displayed comprises:
determining whether a distortion distance between the camera pose of the VR frame image at the current moment and the camera pose of the VR frame image to be rendered is greater than 0;
if the difference image is larger than 0, performing image distortion on the difference image;
and obtaining a VR frame image to be displayed according to the VR frame image at the current moment and the distorted difference image.
Obtaining a VR frame image to be displayed according to the VR frame image at the current moment and the distorted difference image, wherein the obtaining of the VR frame image to be displayed comprises:
obtaining the pixel value of the VR frame image to be displayed in the ith channel according to the following mode:
Inew(i)＝min[2×(Idelta(i)-127)+Id(i)，255)
wherein, Inew(i) Representing I-channel value, I, of the VR frame image to be displayeddelta(i) I-channel value, I, representing the difference imaged(i) And i-channel values, i ═ Y, U, and V, representing the VR frame image at the current time.
In a third aspect, an embodiment of the present invention further provides an image processing apparatus, applied to an edge cloud, including:
the first acquisition module is used for acquiring a VR frame image to be rendered;
the first determining module is used for searching a first-level cache and determining whether a first cache image exists or not, wherein the similarity between the first cache image and the VR frame image to be rendered meets a first preset requirement;
a second obtaining module, configured to obtain a second cache image from a second-level cache under a condition that the first-level cache does not have the first cache image, where a similarity between the second cache image and the VR frame image to be rendered meets a second preset requirement;
a third obtaining module, configured to obtain a difference image according to the VR frame image to be rendered and the second cache image;
the first sending module is used for sending the difference image to a terminal;
wherein the first-level cache and the second-level cache are respectively used for storing rendered VR frame images.
Wherein the second obtaining module includes:
the first obtaining submodule is used for obtaining the target camera pose of the VR frame image to be rendered;
the first searching submodule is used for searching the second-level cache and acquiring a third cache image from the second-level cache;
and the second obtaining submodule is used for obtaining the second cache image according to the third cache image.
Wherein the second obtaining sub-module includes:
a first obtaining unit, configured to take the third cached image as the second cached image if a distortion distance between a camera pose of the third cached image and the target camera pose is 0;
and the second obtaining unit is used for performing image distortion on the third cache image to obtain a second cache image if the distortion distance between the camera pose of the third cache image and the target camera pose is greater than 0 and less than or equal to a preset threshold value.
Wherein the third obtaining module is configured to:
obtaining the pixel value of the difference image in the ith channel according to the following mode:
Figure BDA0002779737380000041
wherein, Idelta(i) I-channel value, I, representing the difference imagel(i) Representing I-channel value, I, of the VR frame image to be renderedc(i) And i-channel values, i ═ Y, U, V, representing the second buffer image.
Wherein the apparatus further comprises:
and the second sending module is used for sending the first cache image to the terminal under the condition that the first-level cache has the first cache image.
In a fourth aspect, an embodiment of the present invention further provides an image processing apparatus, applied to a terminal, including:
a first obtaining module, configured to obtain a difference image from an edge cloud, where the difference image is obtained by the edge cloud according to a VR frame image to be rendered and a rendered VR frame image cached by the edge cloud;
the second acquisition module is used for acquiring a VR frame image at the current moment;
and the third acquisition module is used for acquiring a VR frame image to be displayed according to the VR frame image at the current moment and the difference image.
Wherein the third obtaining module comprises:
a first determining submodule, configured to determine whether a distortion distance between a camera pose of the VR frame image at the current time and a camera pose of the VR frame image to be rendered is greater than 0;
a first processing sub-module for performing image warping on the difference image if the difference image is greater than 0;
and the first obtaining submodule is used for obtaining a VR frame image to be displayed according to the VR frame image at the current moment and the distorted difference image.
The third obtaining module is configured to obtain a pixel value of the VR frame image to be displayed in an ith channel according to the following manner:
Inew(i)＝min[2×(Idelta(i)-127)+Id(i)，255)
wherein, Inew(i) Representing I-channel value, I, of the VR frame image to be displayeddelta(i) I-channel value, I, representing the difference imaged(i) And i-channel values, i ═ Y, U, and V, representing the VR frame image at the current time.
In a fifth aspect, an embodiment of the present invention further provides an image processing apparatus, applied to an edge cloud, including: a processor and a transceiver;
wherein the processor is configured to:
acquiring a VR frame image to be rendered;
searching a first-level cache, and determining whether a first cache image exists or not, wherein the similarity between the first cache image and a VR frame image to be rendered meets a first preset requirement;
under the condition that the first-level cache does not have the first cache image, acquiring a second cache image from a second-level cache, wherein the similarity between the second cache image and the VR frame image to be rendered meets a second preset requirement;
obtaining a difference image according to the VR frame image to be rendered and the second cache image;
sending the difference image to a terminal;
wherein the first-level cache and the second-level cache are respectively used for storing rendered VR frame images.
Wherein the processor is further configured to:
acquiring the pose of a target camera of the VR frame image to be rendered;
searching the second-level cache, and acquiring a third cache image from the second-level cache;
and obtaining the second cache image according to the third cache image.
Wherein the processor is further configured to:
if the distortion distance between the camera pose of the third cache image and the target camera pose is 0, taking the third cache image as the second cache image;
and if the distortion distance between the camera pose of the third cache image and the target camera pose is greater than 0 and less than or equal to a preset threshold value, performing image distortion on the third cache image to obtain a second cache image.
Wherein the processor is further configured to:
obtaining the pixel value of the difference image in the ith channel according to the following mode:
Figure BDA0002779737380000061
wherein, Idelta(i) I-channel value, I, representing the difference imagel(i) Representing I-channel value, I, of the VR frame image to be renderedc(i) And i-channel values representing the second buffer image, i ═ Y, U, V.
Wherein the processor is further configured to:
and sending the first cache image to the terminal under the condition that the first-level cache has the first cache image.
Wherein the processor is further configured to:
rendering the VR frame image to be rendered under the condition that the second-level cache does not have the second cache image, and sending the rendered VR frame image to the terminal;
and storing the rendered VR frame image to the second-level cache.
In a fifth aspect, an embodiment of the present invention further provides an image processing apparatus, which is applied to a terminal, and includes: a processor and a transceiver;
wherein the processor is configured to:
acquiring a difference image from an edge cloud, wherein the difference image is acquired by the edge cloud according to a VR frame image to be rendered and a rendered VR frame image cached by the edge cloud;
acquiring a VR frame image at the current moment;
and obtaining a VR frame image to be displayed according to the VR frame image at the current moment and the difference image.
Wherein the processor is further configured to:
obtaining a VR frame image to be displayed according to the VR frame image at the current moment and the difference image, wherein the VR frame image to be displayed comprises:
determining whether a distortion distance between the camera pose of the VR frame image at the current moment and the camera pose of the VR frame image to be rendered is greater than 0;
if the difference image is larger than 0, performing image distortion on the difference image;
and obtaining a VR frame image to be displayed according to the VR frame image at the current moment and the distorted difference image.
Wherein the processor is further configured to: obtaining a pixel value of the VR frame image to be displayed in an ith channel according to the following mode:
Inew(i)＝min[2×(Idelta(i)-127)+Id(i)，255)
wherein, Inew(i) Representing I-channel value, I, of the VR frame image to be displayeddelta(i) I-channel value, I, representing the difference imaged(i) And i-channel values, i ═ Y, U, and V, representing the VR frame image at the current time.
In a seventh aspect, an embodiment of the present invention further provides a readable storage medium, where a program is stored on the readable storage medium, and when the program is executed by a processor, the program implements the steps in the image processing method described above.
In an eighth aspect, an embodiment of the present invention further provides a communication device, including: a transceiver, a memory, a processor, and a program stored on the memory and executable on the processor; the processor is used for reading the program in the memory to realize the steps of the image processing method.
In the embodiment of the invention, when the VR frame image needs to be rendered, the VR frame image is searched in the first-level cache and the second-level storage of the edge cloud to obtain the rendered VR frame image of which the similarity with the VR frame image meets the preset requirement, so that unnecessary computing waste of VR frame rendering on the edge cloud is avoided, and resources of the edge cloud are saved.
Drawings
FIG. 1 is a flow chart of an image processing method provided by an embodiment of the invention;
FIG. 2 is a schematic diagram of a memory structure provided by an embodiment of the invention;
FIG. 3 is a second flowchart of an image processing method according to an embodiment of the present invention;
FIG. 4 is a schematic illustration of a process provided by an embodiment of the present invention;
FIG. 5 is a block diagram of an image processing apparatus according to an embodiment of the present invention;
FIG. 6 is a second block diagram of an image processing apparatus according to an embodiment of the present invention;
FIG. 7 is a third block diagram of an image processing apparatus according to an embodiment of the present invention;
FIG. 8 is a fourth block diagram of an image processing apparatus according to an embodiment of the present invention.
Detailed Description
The term "and/or" in the embodiments of the present invention describes an association relationship of associated objects, and indicates that three relationships may exist, for example, a and/or B may indicate: a exists alone, A and B exist simultaneously, and B exists alone. The character "/" generally indicates that the former and latter associated objects are in an "or" relationship.
In the embodiments of the present application, the term "plurality" means two or more, and other terms are similar thereto.
The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only a part of the embodiments of the present application, and not all of the embodiments. All other embodiments obtained by a person of ordinary skill in the art based on the embodiments in the present application without making any creative effort belong to the protection scope of the present application.
Referring to fig. 1, fig. 1 is a flowchart of an image processing method provided by an embodiment of the present invention, and is applied to an edge cloud, as shown in fig. 1, including the following steps:
step 101, obtaining a VR frame image to be rendered.
When VR frame image rendering needs to be carried out, the edge cloud can acquire the VR frame image to be rendered from the terminal.
And 102, searching a first-level cache, and determining whether a first cache image exists or not, wherein the similarity between the first cache image and the VR frame image to be rendered meets a first preset requirement.
As shown in fig. 2, in an embodiment of the present invention, two levels of storage are established at the edge cloud. The system comprises a first-level cache and a second-level storage. Wherein the first-level cache and the second-level cache are respectively used for storing rendered VR frame images.
Specifically, the first level cache may be configured to store rendered VR frame images corresponding to a certain terminal, and the second level cache is configured to store rendered VR frame images of all terminals. One or more terminals may correspond to the same or multiple first level caches.
The delivery of the second-level cache rendered image to each terminal inevitably results in Inter-Process Communication (IPC). To minimize the operating overhead of such systems, a local small cache (i.e., a first level cache) is also created and maintained for each terminal, which stores the partially rendered images for faster multiplexing. In general, the VR frame rate is 60fps, and the buffer in the second-level cache library is designed to store images in 5s, that is, 300 frames of images.
When a request of a certain terminal is received, the request can be firstly searched in the first-level cache corresponding to the terminal. For example, the corresponding first level cache is looked up according to the representation of the terminal. Specifically, the first-level cache can be searched according to the pose of the target camera of the VR frame image to be rendered, so as to determine whether an image meeting a first preset requirement with the similarity between the first-level cache image and the VR frame image to be rendered exists in the first-level cache. For example, the first preset requirement may be that the distance between camera poses of the image is smaller than a certain preset value (which may be set as required), or that the similarity of the background is greater than a certain preset value (which may be set as required), and the like. If so, the first cache image can be directly sent to the terminal. If not, then the second level cache needs to be looked up.
Step 103, under the condition that the first-level cache does not have the first cache image, obtaining a second cache image from a second-level cache, wherein the similarity between the second cache image and the VR frame image to be rendered meets a second preset requirement.
For example, the second preset requirement may be that the distance between camera poses of the image is less than a certain preset value (which may be set as needed), and the like.
Specifically, in this step, the pose of the target camera of the VR frame image to be rendered may be obtained, then the second-level cache is searched, a third cache image is obtained from the second-level cache, and finally the second cache image is obtained according to the third cache image.
And if the distortion distance between the camera pose of the third cached image and the target camera pose is 0, the third cached image and the second cached image can be considered to be completely matched or the similarity is 100%, and then the third cached image can be taken as the second cached image. And if the distortion distance between the camera pose of the third cache image and the target camera pose is greater than 0 and less than or equal to a preset threshold (wherein the threshold can be set as required), performing image distortion on the third cache image to obtain the second cache image.
In a particular application, selecting a larger threshold may warp the image to a greater distance and save more computation. Typically, the threshold D is presetthreshold5.0, namely 5m distance of the actual environment, the picture before and after distortion can be maintained to be more than 40% repeated; preset threshold DthresholdThe picture before and after distortion can be maintained to be more than 60% repeated as 2.0. In the embodiment of the present invention, a preset threshold D may be setthreshold＝2.0。
And 104, obtaining a difference image according to the VR frame image to be rendered and the second cache image.
In order to avoid transmitting a complete VR frame image, reduce rendering and coding and decoding time delays, and save edge cloud transmission bandwidth, in the embodiment of the present invention, delta images (or referred to as difference images) of two reference frames are transmitted. Wherein, the two reference frames are respectively: the VR frame image to be rendered and the second buffer image. A difference image refers to the difference between two reference frames. The difference image is transmitted by compression coding such as H.264, H.265 technology and the like, and by adjusting the compression rate of the coding, the network transmission bandwidth is saved while the picture quality is ensured. Specifically, if the VR frame image is 8bit per channel color depth transmission, the pixel value of the difference image in the ith channel:
Figure BDA0002779737380000101
wherein, Idelta(i) I-channel value, I, representing the difference imagel(i) Representing I-channel value, I, of the VR frame image to be renderedc(i) And i-channel values representing the second buffer image, i ═ Y, U, V.
That is, the channel values of the difference image in the three channels Y, U, and V are calculated according to the above formula, and the calculation result maps the positive value and the negative value of the difference image into lighter color and darker color, respectively.
And 105, sending the difference image to a terminal.
In the transmission process, operations such as encoding of the difference image can be performed.
In the embodiment of the invention, when the VR frame image needs to be rendered, the VR frame image is searched in the first-level cache and the second-level storage of the edge cloud to obtain the rendered VR frame image of which the similarity with the VR frame image meets the preset requirement, so that unnecessary computing waste of VR frame rendering on the edge cloud is avoided, and resources of the edge cloud are saved.
Referring to fig. 3, fig. 3 is a flowchart of an image processing method according to an embodiment of the present invention, applied to a terminal, as shown in fig. 3, including the following steps:
step 301, obtaining a difference image from an edge cloud, where the difference image is obtained by the edge cloud according to a VR frame image to be rendered and a rendered VR frame image cached by the edge cloud.
The manner in which the edge cloud obtains the difference image may refer to the description of the foregoing embodiments. After the difference image is acquired, it may be subjected to operations such as decoding.
And step 302, obtaining a VR frame image at the current moment.
The current time may refer to a VR frame image obtained when the terminal receives the difference image.
And 303, obtaining a VR frame image to be displayed according to the VR frame image at the current moment and the difference image.
In order to enable the obtained image to better meet the requirements of users, in the embodiment of the invention, whether the distortion distance between the camera pose of the VR frame image at the current moment and the camera pose of the VR frame image to be rendered is greater than 0 or not is determined, and if the distortion distance is greater than 0, the difference image is subjected to image distortion, so that the difference image is consistent with the camera pose of the VR frame image at the current moment. If equal to 0, no twisting operation is performed. And then, obtaining a VR frame image to be displayed according to the VR frame image at the current moment and the distorted difference image. Specific warping methods may include ASW (Asynchronous space warping), ATW (Asynchronous time warping) algorithms, and the like.
The pixel value of the VR frame image to be displayed in the ith channel can be obtained as follows:
Inew(i)＝min[2×(Idelta(i)-127)+Id(i)，255)
wherein, Inew(i) Representing I-channel value, I, of the VR frame image to be displayeddelta(i) I-channel value, I, representing the difference imaged(i) And i-channel values, i ═ Y, U, and V, representing the VR frame image at the current time.
In the embodiment of the invention, when the VR frame image needs to be rendered, the VR frame image is searched in the first-level cache and the second-level storage of the edge cloud to obtain the rendered VR frame image of which the similarity with the VR frame image meets the preset requirement, so that unnecessary computing waste of VR frame rendering on the edge cloud is avoided, and resources of the edge cloud are saved.
As shown in fig. 4, in order to maximally reduce the latency, in the embodiment of the present invention, a parallel streaming processing mechanism is adopted. Moreover, a left-eye and right-eye separation type method is adopted in the process of rendering and image searching. The difference image for the left eye of frame 0 (L0) is encoded while rendering and image finding for the right eye of frame 0 (denoted as R0). In the execution process, in order to avoid processing pause or resource idle caused by heterogeneous computational complexity of different stages, a request queue is maintained for each stage, and then the next task can be immediately carried out without waiting.
It should be noted that image distortion does not impose any limitation on the pose of the target camera, so that perspective re-projection can be performed in the virtual world. However, the visual quality of image warping depends on the accuracy of measuring the pose of the user in the virtual world, i.e. the user's actions should be accurately traced and reflected. The accuracy of this pose measurement is inherently determined by the hardware of the mobile HMD (Head Mounted Device) that enables real-time acquisition tracking of the user's motion.
Meanwhile, in the embodiment of the invention, the cache performance is optimized through a second-level cache mechanism, and the method has the advantages of a centralized cache scheme and a distributed cache scheme. Therefore, IPC inter-process communication in the data transmission process can be reduced in this way, and the transmission bandwidth and the transmission delay can be reduced by only transmitting the difference image.
The embodiment of the invention also provides an image processing device which is applied to the edge cloud. Referring to fig. 5, fig. 5 is a structural diagram of an image processing apparatus according to an embodiment of the present invention. Because the principle of the image processing apparatus for solving the problem is similar to the image processing method in the embodiment of the present invention, the implementation of the image processing apparatus can refer to the implementation of the method, and repeated details are not repeated.
As shown in fig. 5, the image processing apparatus 500 includes:
a first obtaining module 501, configured to obtain a VR frame image to be rendered; a first determining module 502, configured to search a first-level cache, and determine whether a first cache image exists, where a similarity between the first cache image and a VR frame image to be rendered meets a first preset requirement; a second obtaining module 503, configured to obtain a second cache image from a second-level cache when the first-level cache does not have the first cache image, where a similarity between the second cache image and the VR frame image to be rendered meets a second preset requirement; a third obtaining module 504, configured to obtain a difference image according to the VR frame image to be rendered and the second cache image; a first sending module 505, configured to send the difference image to a terminal; wherein the first-level cache and the second-level cache are respectively used for storing rendered VR frame images.
Wherein the second obtaining module includes:
the first obtaining submodule is used for obtaining the target camera pose of the VR frame image to be rendered;
the first searching submodule is used for searching the second-level cache and acquiring a third cache image from the second-level cache;
and the second obtaining submodule is used for obtaining the second cache image according to the third cache image.
Wherein the second obtaining sub-module includes:
a first obtaining unit, configured to take the third cached image as the second cached image if a distortion distance between a camera pose of the third cached image and the target camera pose is 0;
and the second acquisition unit is used for carrying out image distortion on the third cache image to obtain a second cache image if the distortion distance between the camera pose of the third cache image and the target camera pose is greater than 0 and less than or equal to a preset threshold value.
Wherein the third obtaining module is configured to:
obtaining the pixel value of the difference image in the ith channel according to the following mode:
Figure BDA0002779737380000121
wherein, Idelta(i) I-channel value, I, representing the difference imagel(i) Representing I-channel value, I, of the VR frame image to be renderedc(i) And i-channel values, i ═ Y, U, V, representing the second buffer image.
Wherein the apparatus further comprises:
and the second sending module is used for sending the first cache image to the terminal under the condition that the first-level cache has the first cache image.
The apparatus provided in the embodiment of the present invention may implement the method embodiments, and the implementation principle and the technical effect are similar, which are not described herein again.
The embodiment of the invention also provides an image processing device which is applied to the terminal. Referring to fig. 6, fig. 6 is a structural diagram of an image processing apparatus according to an embodiment of the present invention. Because the principle of the image processing apparatus for solving the problem is similar to the image processing method in the embodiment of the present invention, the implementation of the image processing apparatus can refer to the implementation of the method, and repeated details are not repeated.
As shown in fig. 6, the image processing apparatus 600 includes:
a first obtaining module 601, configured to obtain a difference image from an edge cloud, where the difference image is obtained by the edge cloud according to a VR frame image to be rendered and a rendered VR frame image cached by the edge cloud; a second obtaining module 602, configured to obtain a VR frame image at a current time; a third obtaining module 603, configured to obtain a VR frame image to be displayed according to the VR frame image at the current time and the difference image.
Wherein the third obtaining module comprises:
a first determining submodule, configured to determine whether a distortion distance between a camera pose of the VR frame image at the current time and a camera pose of the VR frame image to be rendered is greater than 0;
a first processing sub-module for performing image warping on the difference image if the difference image is greater than 0;
and the first obtaining submodule is used for obtaining a VR frame image to be displayed according to the VR frame image at the current moment and the distorted difference image.
The third obtaining module is configured to obtain a pixel value of the VR frame image to be displayed in an ith channel according to the following manner:
Inew(i)＝min[2×(Idelta(i)-127)+Id(i)，255)
wherein, Inew(i) Representing I-channel value, I, of the VR frame image to be displayeddelta(i) I-channel value, I, representing the difference imaged(i) And i-channel values, i ═ Y, U, and V, representing the VR frame image at the current time.
The apparatus provided in the embodiment of the present invention may implement the method embodiments, and the implementation principle and the technical effect are similar, which are not described herein again.
The embodiment of the invention also provides an image processing device which is applied to the edge cloud. Referring to fig. 7, fig. 7 is a structural diagram of an image processing apparatus according to an embodiment of the present invention. Because the principle of the image processing apparatus for solving the problem is similar to the image processing method in the embodiment of the present invention, the implementation of the image processing apparatus can refer to the implementation of the method, and repeated details are not repeated.
As shown in fig. 7, the image processing apparatus 700 includes: a processor 701 and a transceiver 702.
Wherein the processor 701 is configured to:
acquiring a VR frame image to be rendered;
searching a first-level cache, and determining whether a first cache image exists or not, wherein the similarity between the first cache image and a VR frame image to be rendered meets a first preset requirement;
under the condition that the first-level cache does not have the first cache image, acquiring a second cache image from a second-level cache, wherein the similarity between the second cache image and the VR frame image to be rendered meets a second preset requirement;
obtaining a difference image according to the VR frame image to be rendered and the second cache image;
sending the difference image to a terminal;
wherein the first-level cache and the second-level cache are respectively used for storing rendered VR frame images.
Wherein the processor 701 is further configured to:
acquiring the pose of a target camera of the VR frame image to be rendered;
searching the second-level cache, and acquiring a third cache image from the second-level cache;
and obtaining the second cache image according to the third cache image.
Wherein the processor 701 is further configured to:
if the distortion distance between the camera pose of the third cache image and the target camera pose is 0, taking the third cache image as the second cache image;
and if the distortion distance between the camera pose of the third cache image and the target camera pose is greater than 0 and less than or equal to a preset threshold value, performing image distortion on the third cache image to obtain a second cache image.
Wherein the processor 701 is further configured to:
obtaining the pixel value of the difference image in the ith channel according to the following mode:
Figure BDA0002779737380000141
wherein, Idelta(i) I-channel value, I, representing the difference imagel(i) Representing I-channel value, I, of the VR frame image to be renderedc(i) And i-channel values, i ═ Y, U, V, representing the second buffer image.
Wherein the processor 701 is further configured to:
and sending the first cache image to the terminal under the condition that the first-level cache has the first cache image.
Wherein the processor 701 is further configured to:
rendering the VR frame image to be rendered under the condition that the second-level cache does not have the second cache image, and sending the rendered VR frame image to the terminal;
and storing the rendered VR frame image to the second-level cache.
The apparatus provided in the embodiment of the present invention may implement the method embodiments, and the implementation principle and the technical effect are similar, which are not described herein again.
The embodiment of the invention also provides an image processing device which is applied to the edge cloud. Referring to fig. 8, fig. 8 is a structural diagram of an image processing apparatus according to an embodiment of the present invention. Because the principle of the image processing apparatus for solving the problem is similar to the image processing method in the embodiment of the present invention, the implementation of the image processing apparatus can refer to the implementation of the method, and repeated details are not repeated.
As shown in fig. 8, the image processing apparatus 800 includes: a processor 801 and a transceiver 802.
Wherein the processor 801 is configured to:
acquiring a difference image from an edge cloud, wherein the difference image is acquired by the edge cloud according to a VR frame image to be rendered and a rendered VR frame image cached by the edge cloud;
acquiring a VR frame image at the current moment;
and obtaining a VR frame image to be displayed according to the VR frame image at the current moment and the difference image.
Wherein the processor 801 is further configured to:
obtaining a VR frame image to be displayed according to the VR frame image at the current moment and the difference image, wherein the VR frame image to be displayed comprises:
determining whether a distortion distance between the camera pose of the VR frame image at the current moment and the camera pose of the VR frame image to be rendered is greater than 0;
if the difference image is larger than 0, performing image distortion on the difference image;
and obtaining a VR frame image to be displayed according to the VR frame image at the current moment and the distorted difference image.
Wherein the processor 801 is further configured to: obtaining the pixel value of the VR frame image to be displayed in the ith channel according to the following mode:
Inew(i)＝min[2×(Idelta(i)-127)+Id(i)，255)
wherein, Inew(i) Representing I-channel value, I, of the VR frame image to be displayeddelta(i) I-channel value, I, representing the difference imaged(i) And i-channel values, i ═ Y, U, and V, representing the VR frame image at the current time.
The apparatus provided in the embodiment of the present invention may implement the method embodiments, and the implementation principle and technical effects are similar, which are not described herein again.
An embodiment of the present invention further provides a communication device, including: a transceiver, a memory, a processor, and a program stored on the memory and executable on the processor; the processor is used for reading the program in the memory to realize the steps of the image processing method.
The embodiment of the present invention further provides a readable storage medium, where a program is stored on the readable storage medium, and when the program is executed by a processor, the program implements each process of the above-mentioned embodiment of the image processing method, and can achieve the same technical effect, and in order to avoid repetition, the detailed description is omitted here. The readable storage medium may be any available medium or data storage device that can be accessed by a processor, including but not limited to magnetic memory (e.g., floppy disk, hard disk, magnetic tape, magneto-optical disk (MO), etc.), optical memory (e.g., CD, DVD, BD, HVD, etc.), and semiconductor memory (e.g., ROM, EPROM, EEPROM, nonvolatile memory (NAND FLASH), Solid State Disk (SSD)), etc.
It should be noted that, in this document, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises the element.
Through the above description of the embodiments, those skilled in the art will clearly understand that the method of the above embodiments can be implemented by software plus a necessary general hardware platform, and certainly can also be implemented by hardware, but in many cases, the former is a better implementation manner. With such an understanding, the technical solutions of the present invention may be embodied in the form of a software product, which is stored in a storage medium (such as ROM/RAM, magnetic disk, optical disk) and includes instructions for enabling a terminal (such as a mobile phone, a computer, a server, an air conditioner, or a network device) to execute the methods according to the embodiments of the present invention.
While the present invention has been described with reference to the embodiments shown in the drawings, the present invention is not limited to the embodiments, which are illustrative and not restrictive, and it will be apparent to those skilled in the art that various changes and modifications can be made therein without departing from the spirit and scope of the invention as defined in the appended claims.

Claims (15)

1. An image processing method is applied to edge cloud, and is characterized by comprising the following steps:
acquiring a Virtual Reality (VR) frame image to be rendered;
searching a first-level cache, and determining whether a first cache image exists or not, wherein the similarity between the first cache image and a VR frame image to be rendered meets a first preset requirement;
under the condition that the first-level cache does not have the first cache image, acquiring a second cache image from a second-level cache, wherein the similarity between the second cache image and the VR frame image to be rendered meets a second preset requirement;
obtaining a difference image according to the VR frame image to be rendered and the second cache image;
sending the difference image to a terminal;
wherein the first-level cache and the second-level cache are respectively used for storing rendered VR frame images.
2. The method of claim 1, wherein said retrieving the second cached image from the second level cache comprises:
acquiring the pose of a target camera of the VR frame image to be rendered;
searching the second-level cache, and acquiring a third cache image from the second-level cache;
and obtaining the second cache image according to the third cache image.
3. The method of claim 2, wherein obtaining the second cached image from the third cached image comprises:
if the distortion distance between the camera pose of the third cache image and the target camera pose is 0, taking the third cache image as the second cache image;
and if the distortion distance between the camera pose of the third cache image and the target camera pose is greater than 0 and less than or equal to a preset threshold value, performing image distortion on the third cache image to obtain a second cache image.
4. The method of claim 1, wherein obtaining a difference image from the VR frame image to be rendered and the second buffered image comprises:
obtaining the pixel value of the difference image in the ith channel according to the following mode:
Figure FDA0002779737370000021
wherein, Idelta(i) I-channel value, I, representing the difference imagel(i) Representing I-channel value, I, of the VR frame image to be renderedc(i) And i-channel values, i ═ Y, U, V, representing the second buffer image.
5. The method of claim 1, further comprising:
and sending the first cache image to the terminal under the condition that the first-level cache has the first cache image.
6. The method of claim 1, further comprising:
rendering the VR frame image to be rendered under the condition that the second-level cache does not have the second cache image, and sending the rendered VR frame image to the terminal;
and storing the rendered VR frame image to the second-level cache.
7. An image processing method applied to a terminal is characterized by comprising the following steps:
acquiring a difference image from an edge cloud, wherein the difference image is acquired by the edge cloud according to a VR frame image to be rendered and a rendered VR frame image cached by the edge cloud;
acquiring a VR frame image at the current moment;
and obtaining a VR frame image to be displayed according to the VR frame image at the current moment and the difference image.
8. The method of claim 7, wherein obtaining the VR frame image to be displayed according to the VR frame image at the current time and the difference image comprises:
determining whether a distortion distance between the camera pose of the VR frame image at the current moment and the camera pose of the VR frame image to be rendered is greater than 0;
if the difference image is larger than 0, performing image distortion on the difference image;
and obtaining a VR frame image to be displayed according to the VR frame image at the current moment and the distorted difference image.
9. The method of claim 7, wherein obtaining the VR frame image to be displayed according to the VR frame image at the current time and the warped difference image comprises:
obtaining the pixel value of the VR frame image to be displayed in the ith channel according to the following mode:
Inew(i)＝min[2×(Idelta(i)-127)+Id(i)，255)
wherein, Inew(i) Representing I-channel value, I, of the VR frame image to be displayeddelta(i) I-channel value, I, representing the difference imaged(i) And i-channel values, i ═ Y, U, and V, representing the VR frame image at the current time.
10. An image processing apparatus applied to an edge cloud, comprising:
the first acquisition module is used for acquiring a VR frame image to be rendered;
the first determining module is used for searching a first-level cache and determining whether a first cache image exists or not, wherein the similarity between the first cache image and the VR frame image to be rendered meets a first preset requirement;
a second obtaining module, configured to obtain a second cache image from a second-level cache under a condition that the first-level cache does not have the first cache image, where a similarity between the second cache image and the VR frame image to be rendered meets a second preset requirement;
a third obtaining module, configured to obtain a difference image according to the VR frame image to be rendered and the second cache image;
the first sending module is used for sending the difference image to a terminal;
wherein the first-level cache and the second-level cache are respectively used for storing rendered VR frame images.
11. An image processing apparatus applied to a terminal, comprising:
a first obtaining module, configured to obtain a difference image from an edge cloud, where the difference image is obtained by the edge cloud according to a VR frame image to be rendered and a rendered VR frame image cached by the edge cloud;
the second acquisition module is used for acquiring a VR frame image at the current moment;
and the third acquisition module is used for acquiring a VR frame image to be displayed according to the VR frame image at the current moment and the difference image.
12. An image processing apparatus applied to an edge cloud, comprising: a processor and a transceiver;
wherein the processor is configured to:
acquiring a VR frame image to be rendered;
searching a first-level cache, and determining whether a first cache image exists or not, wherein the similarity between the first cache image and a VR frame image to be rendered meets a first preset requirement;
under the condition that the first-level cache does not have the first cache image, acquiring a second cache image from a second-level cache, wherein the similarity between the second cache image and the VR frame image to be rendered meets a second preset requirement;
obtaining a difference image according to the VR frame image to be rendered and the second cache image;
sending the difference image to a terminal;
wherein the first-level cache and the second-level cache are respectively used for storing rendered VR frame images.
13. An image processing apparatus applied to a terminal, comprising: a processor and a transceiver;
wherein the processor is configured to:
acquiring a difference image from an edge cloud, wherein the difference image is acquired by the edge cloud according to a VR frame image to be rendered and a rendered VR frame image cached by the edge cloud;
acquiring a VR frame image at the current moment;
and obtaining a VR frame image to be displayed according to the VR frame image at the current moment and the difference image.
14. A readable storage medium storing a program, wherein the program, when executed by a processor, implements the steps in the image processing method according to any one of claims 1 to 9.
15. A communication device, comprising: a transceiver, a memory, a processor, and a program stored on the memory and executable on the processor; characterized by a processor for reading a program in a memory implementing the steps of the image processing method according to any one of claims 1 to 9.
CN202011277952.XA 2020-11-16 2020-11-16 Image processing method, device and equipment and readable storage medium Pending CN114584598A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202011277952.XA CN114584598A (en) 2020-11-16 2020-11-16 Image processing method, device and equipment and readable storage medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202011277952.XA CN114584598A (en) 2020-11-16 2020-11-16 Image processing method, device and equipment and readable storage medium

Publications (1)

Publication Number Publication Date
CN114584598A true CN114584598A (en) 2022-06-03

Family

ID=81766780

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202011277952.XA Pending CN114584598A (en) 2020-11-16 2020-11-16 Image processing method, device and equipment and readable storage medium

Country Status (1)

Country Link
CN (1) CN114584598A (en)

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN104852959A (en) * 2015-04-09 2015-08-19 福建星网视易信息系统有限公司 Data transmission method and system for remote service
CN107534775A (en) * 2015-05-04 2018-01-02 谷歌公司 The coding prefetched for application stream
CN108122266A (en) * 2017-12-20 2018-06-05 成都卓杭网络科技股份有限公司 A kind of method, apparatus and storage medium for caching skeleton cartoon and rendering texture
US20180266843A1 (en) * 2015-09-17 2018-09-20 Baidu Online Network Technology (Beijing) Co., Ltd Method and apparatus for rendering navigation image and storage medium
CN110351480A (en) * 2019-06-13 2019-10-18 歌尔科技有限公司 Image processing method, device and electronic equipment for electronic equipment

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN104852959A (en) * 2015-04-09 2015-08-19 福建星网视易信息系统有限公司 Data transmission method and system for remote service
CN107534775A (en) * 2015-05-04 2018-01-02 谷歌公司 The coding prefetched for application stream
US20180266843A1 (en) * 2015-09-17 2018-09-20 Baidu Online Network Technology (Beijing) Co., Ltd Method and apparatus for rendering navigation image and storage medium
CN108122266A (en) * 2017-12-20 2018-06-05 成都卓杭网络科技股份有限公司 A kind of method, apparatus and storage medium for caching skeleton cartoon and rendering texture
CN110351480A (en) * 2019-06-13 2019-10-18 歌尔科技有限公司 Image processing method, device and electronic equipment for electronic equipment

Similar Documents

Publication Publication Date Title
US11694316B2 (en) Method and apparatus for determining experience quality of VR multimedia
US10834391B2 (en) Hologram streaming machine
CN111614967B (en) Live virtual image broadcasting method and device, electronic equipment and storage medium
EP2724343A1 (en) Video remixing system
WO2022000298A1 (en) Reinforcement learning based rate control
CN115205925A (en) Expression coefficient determining method and device, electronic equipment and storage medium
CN112929728A (en) Video rendering method, device and system, electronic equipment and storage medium
CN110211017A (en) Image processing method, device and electronic equipment
JP7559156B2 (en) Method, system, and medium for streaming video content using an adaptive buffer
CN111405293B (en) Video transmission method and device
CN113473172B (en) VR video caching method and device, caching service device and storage medium
US20240098316A1 (en) Video encoding method and apparatus, real-time communication method and apparatus, device, and storage medium
CN114584598A (en) Image processing method, device and equipment and readable storage medium
JP7578378B2 (en) Data encoding method, device, and computer program
EP4152180A2 (en) Method of detecting action, electronic device, and storage medium
CN117692704A (en) Virtual scene display method and system
CN113794887A (en) Method and related equipment for video coding in game engine
CN117424936B (en) Video edge gateway autonomous scheduling monitoring method, device, equipment and medium
CN115278290B (en) Virtual reality video caching method and device based on edge nodes
CN116112740B (en) Video playing method and device, electronic equipment and storage medium
WO2023125970A1 (en) Code rate allocation method and apparatus, storage method and apparatus, device, and storage medium
CN110087145B (en) Method and apparatus for processing video
CN118331414A (en) Image processing method, device, equipment and computer readable storage medium
CN115776447A (en) Streaming media quality evaluation method, device, client and storage medium
CN113673365A (en) Target detection method and device, electronic equipment and storage medium

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination