[go: up one dir, main page]
More Web Proxy on the site http://driver.im/

WO2021057097A1 - 图像渲染和编码方法及相关装置 - Google Patents

图像渲染和编码方法及相关装置 Download PDF

Info

Publication number
WO2021057097A1
WO2021057097A1 PCT/CN2020/095529 CN2020095529W WO2021057097A1 WO 2021057097 A1 WO2021057097 A1 WO 2021057097A1 CN 2020095529 W CN2020095529 W CN 2020095529W WO 2021057097 A1 WO2021057097 A1 WO 2021057097A1
Authority
WO
WIPO (PCT)
Prior art keywords
data
processor
format
interface
format data
Prior art date
Application number
PCT/CN2020/095529
Other languages
English (en)
French (fr)
Inventor
孙南勇
王凤娟
谢星华
Original Assignee
华为技术有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 华为技术有限公司 filed Critical 华为技术有限公司
Priority to EP20868794.7A priority Critical patent/EP3972253A4/en
Publication of WO2021057097A1 publication Critical patent/WO2021057097A1/zh
Priority to US17/572,067 priority patent/US11882297B2/en

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/47—End-user applications
    • H04N21/478—Supplemental services, e.g. displaying phone caller identification, shopping application
    • H04N21/4781—Games
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
    • H04N19/169—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding
    • H04N19/186—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being a colour or a chrominance component
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T1/00—General purpose image data processing
    • G06T1/20—Processor architectures; Processor configuration, e.g. pipelining
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/40—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using video transcoding, i.e. partial or full decoding of a coded input stream followed by re-encoding of the decoded output stream
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/42—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals characterised by implementation details or hardware specially adapted for video compression or decompression, e.g. dedicated software implementation
    • H04N19/423—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals characterised by implementation details or hardware specially adapted for video compression or decompression, e.g. dedicated software implementation characterised by memory arrangements
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/21—Server components or server architectures
    • H04N21/226—Characteristics of the server or Internal components of the server
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
    • H04N21/234—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
    • H04N21/234—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs
    • H04N21/23412—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs for generating or manipulating the scene composition of objects, e.g. MPEG-4 objects
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
    • H04N21/234—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs
    • H04N21/2343—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements
    • H04N21/234309—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements by transcoding between formats or standards, e.g. from MPEG-2 to MPEG-4 or from Quicktime to Realvideo
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
    • H04N21/443—OS processes, e.g. booting an STB, implementing a Java virtual machine in an STB or power management in an STB
    • A—HUMAN NECESSITIES
    • A63—SPORTS; GAMES; AMUSEMENTS
    • A63F—CARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR
    • A63F13/00—Video games, i.e. games using an electronically generated display having two or more dimensions
    • A63F13/30—Interconnection arrangements between game servers and game devices; Interconnection arrangements between game devices; Interconnection arrangements between game servers
    • A63F13/35—Details of game servers
    • A63F13/355—Performing operations on behalf of clients with restricted processing capabilities, e.g. servers transform changing game scene into an encoded video stream for transmitting to a mobile phone or a thin client
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2210/00—Indexing scheme for image generation or computer graphics
    • G06T2210/08—Bandwidth reduction
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2210/00—Indexing scheme for image generation or computer graphics
    • G06T2210/32—Image data format

Definitions

  • the present invention relates to the technical field of image data processing, in particular to an image rendering and coding method and related devices.
  • the server renders and encodes each frame of image and transmits it to the remote client through the network.
  • the remote client receives the frame data from the cloud server and then decodes and displays it.
  • the server side uses graphics processing unit (GPU) to render the game screen, and the rendered image is image data in RGBA format (RGBA represents the colors of Red, Green, Blue and Alpha) Space), and the data size of a typical 720P image is about 28M bits.
  • RGBBA represents the colors of Red, Green, Blue and Alpha
  • FIG. 1 The specific implementation of the data flow of rendering and encoding in the existing solution can be shown in Figure 1.
  • the rendered image data in RGBA format needs to be copied to the central processing unit (central processing unit, CPU), the CPU converts the image data in the RGBA format into the frame data in the YUV format, and then the CPU copies the frame data in the YUV format to the video memory space of the GPU, and then performs H264 or H265 encoding in the GPU.
  • the data after the rendering operation needs to be transferred twice between the CPU and the GPU, namely steps 3 and 5 in Figure 1, which consumes additional CPU and GPU resources, reduces performance, and reduces the efficiency of encoding and rendering.
  • the embodiment of the present application discloses an image rendering and encoding method and related devices, which can save processor resources and improve the efficiency of rendering and encoding.
  • an embodiment of the present application discloses an image rendering and encoding method, and the method includes:
  • the first processor sends the data to be rendered to the second processor
  • the first processor instructs the second processor to render data in a first format according to the data to be rendered.
  • the data in the first format includes the color characteristics of the three primary colors of the image; the data in the first format is stored in the second processor. In the first storage space;
  • the first processor instructs the second processor to convert the data in the first format into data in a second format, where the data in the second format includes the luminance value and the chrominance value of the image;
  • the first processor instructs the second processor to encode the second format data into third format data, and the data capacity of the third format data is smaller than the data capacity of the second format data;
  • the first processor sends the third format data to the client.
  • the above-mentioned first processor may be a central processing unit (CPU), a microprocessor without interlocking pipeline stage architecture, an advanced simplified instruction set machine, a field programmable gate array (FPGA), and the like.
  • the foregoing second processor may be a graphics processor GPU, an artificial intelligence accelerator card, or other devices or devices that can complete graphics processing.
  • the data in the first format rendered in the second processor needs to be copied to the memory of the first processor. After the data in the second format is converted by the first processor, it is copied to the first processor. Encoding is performed in the memory of the second processor.
  • the embodiment of the present application can complete the process of rendering, format conversion and encoding in the second processor, and does not require two additional data transmissions. Therefore, the embodiment of the present application can save processor resources , Improve the efficiency of rendering and coding.
  • the first processor instructs the second processor to render data in the first format according to the data to be rendered
  • the first processor instructs the second processor to render the first format data Before the data in the first format is converted into the data in the second format, it also includes:
  • the first processor invokes a first interface to obtain the storage address of the data in the first format in the first storage space;
  • the first processor calls the first interface to send the storage address to the second interface
  • the first processor calls the second interface to instruct the second processor to mark the storage address in the area to be encoded, and the area to be encoded includes storage for storing the data to be encoded and/or the storage address of the data to be encoded space.
  • the rendered image data can be obtained through a special application program interface, so that the image data can be formatted on the second processor side, which avoids copying the rendered data to the first processing.
  • the operation of copying to the second processor after converting the format by the processor saves the resources of the first and second processors.
  • the above-mentioned first interface is an application program interface configured in the graphics library for obtaining the above-mentioned storage address.
  • the second interface is configured in the video acceleration library for receiving the storage address sent by the first interface and instructing the second processor to mark the storage address to the to-be-coded Application program interface in the zone.
  • the first processor invokes a third interface to apply for a second storage space from the second processor, and the second storage space is an address space for storing the data in the second format;
  • the third interface is an application program interface in the video acceleration library; the first processor calls the third interface to instruct the second processor to convert the data in the first format into data in the second format.
  • the above-mentioned first processor instructs the above-mentioned second processor to encode the above-mentioned second format data into the third format data, including:
  • the first processor calls the fourth interface to apply to the second processor for a third storage space, the third storage space is an address space for storing the third format data; the fourth interface is an application in the video acceleration library Program interface
  • the first processor calls the fourth interface to instruct the second processor to encode the second format data into third format data.
  • an embodiment of the present application discloses an image rendering and encoding method, which is applied to a processor, and the method includes:
  • an embodiment of the present application discloses a processor, which performs the following operations:
  • the second processor to render data in the first format according to the data to be rendered, the data in the first format includes the three primary color characteristics of the image; the data in the first format is stored in the first storage space of the second processor ;
  • the processor instructs the second processor to render data in the first format according to the data to be rendered
  • the processor instructs the second processor to convert the data in the first format Before the data in the second format, it also includes:
  • the processor calls the first interface to obtain the storage address of the data in the first format in the first storage space;
  • the processor calls the first interface to send the storage address to the second interface
  • the processor calls the second interface to instruct the second processor to mark the storage address in the area to be encoded, and the area to be encoded includes a storage space for storing the data to be encoded and/or the storage address of the data to be encoded.
  • the above-mentioned first interface is an application program interface configured in the graphics library for obtaining the above-mentioned storage address.
  • the second interface is configured in the video acceleration library for receiving the storage address sent by the first interface and instructing the second processor to mark the storage address to the to-be-coded Application program interface in the zone.
  • the processor instructing the second processor to convert the data in the first format into data in the second format includes:
  • the processor calls the third interface to apply for a second storage space from the second processor, the second storage space is an address space for storing the data in the second format; the third interface is an application program interface in the video acceleration library ;
  • the processor calls the third interface to instruct the second processor to convert the data in the first format into data in the second format.
  • the processor instructing the second processor to encode the data in the second format into data in the third format includes:
  • the processor calls the fourth interface to apply to the second processor for a third storage space, the third storage space is an address space for storing the third format data; the fourth interface is an application program interface in the video acceleration library ;
  • the processor calls the fourth interface to instruct the second processor to encode the data in the second format into data in the third format.
  • an embodiment of the present application discloses a processor, which performs the following operations:
  • an embodiment of the present application discloses a processor, and the processor includes:
  • a sending unit configured to send the data to be rendered to the second processor
  • the instruction unit is configured to: instruct the second processor to render data in the first format according to the data to be rendered, the data in the first format includes three primary color characteristics of the image; the data in the first format is stored in the second processor In the first storage space;
  • the sending unit is further configured to send the third format data to the client.
  • the foregoing processor further includes:
  • the calling unit is configured to: after the instructing unit instructs the second processor to render data in the first format according to the data to be rendered, instruct the second processor to convert the data in the first format into a Before the second format data, call the first interface to obtain the storage address of the first format data in the first storage space;
  • Calling the second interface instructs the second processor to mark the storage address in a to-be-encoded area, and the to-be-encoded area includes a storage space for storing the data to be encoded and/or the storage address of the data to be encoded .
  • the above-mentioned first interface is an application program interface configured in the graphics library for obtaining the above-mentioned storage address.
  • the second interface is configured in the video acceleration library for receiving the storage address sent by the first interface and instructing the second processor to mark the storage address to the to-be-coded Application program interface in the zone.
  • the above-mentioned indication unit is specifically used for:
  • a third interface to apply to the second processor for a second storage space, where the second storage space is an address space for storing the data in the second format; the third interface is an application program interface in the video acceleration library;
  • the above-mentioned indicating unit is specifically further used for:
  • the fourth interface to apply to the second processor for a third storage space, where the third storage space is an address space for storing the third format data; the fourth interface is an application program interface in the video acceleration library;
  • the fourth interface is invoked to instruct the second processor to encode the data in the second format into data in the third format.
  • an embodiment of the present application discloses a processor, and the processor includes:
  • a receiving unit configured to receive data to be rendered sent by the first processor
  • a rendering unit configured to render data in a first format according to the data to be rendered, where the data in the first format includes three primary colors of the image; the data in the first format is stored in the first storage space of the processor;
  • a conversion unit configured to convert the data in the first format in the first storage space into data in a second format, the data in the second format including the luminance value and the chrominance value of the image;
  • An encoding unit configured to encode the second format data into third format data, and the data capacity of the third format data is smaller than the data capacity of the second format data;
  • the sending unit is used to send the above-mentioned third format data.
  • an embodiment of the present application discloses a server.
  • the server includes a first processor, a second processor, and a memory.
  • the memory stores a computer program.
  • the first processor is configured to call the computer program to execute the first processor. The method of any one of the aspects.
  • an embodiment of the present application discloses a computer-readable storage medium that stores a computer program in the computer-readable storage medium, and when the program is executed by a processor, the program described in any one of the above-mentioned first aspects is implemented. method.
  • the data in the first format rendered in the second processor needs to be copied to the memory of the first processor. After the first processor converts the data in the second format, , And then copied to the memory of the second processor for encoding.
  • the embodiment of the present application can complete the process of rendering, format conversion, and encoding in the second processor. There is no need for two additional data transmissions. Therefore, the embodiment of the present application can Save processor resources, improve rendering and coding efficiency.
  • FIG. 1 is a schematic diagram of interaction of a rendering and encoding method in the prior art
  • Figure 2 is a schematic diagram of a system architecture provided by an embodiment of the present invention.
  • FIG. 3 is a schematic diagram of interaction of a rendering and encoding method provided by an embodiment of the present invention.
  • FIG. 4 is a schematic diagram of a scenario of interaction between a client and a server according to an embodiment of the present invention
  • FIG. 5 is a schematic structural diagram of a processor provided by an embodiment of the present invention.
  • FIG. 6 is a schematic structural diagram of another processor provided by an embodiment of the present invention.
  • Fig. 7 is a schematic structural diagram of a server provided by an embodiment of the present invention.
  • the system architecture shown in FIG. 2 may include a server 201 and a client 202.
  • the server 201 is mainly used to render and encode the image or video required by the client 202 according to the request of the client 202, and then send it to the client 202 for display.
  • the server 201 may include, but is not limited to, a background server, a component server, an image rendering and encoding server, and so on.
  • the server 201 needs to run corresponding server-side programs to provide corresponding image rendering and encoding services, such as programs for data calculation, decision execution, and so on.
  • the client 202 may include handheld devices (for example, mobile phones, tablets, palmtop computers, etc.), vehicle-mounted devices, wearable devices (for example, smart watches, smart bracelets, pedometers, etc.), smart home devices (for example, refrigerators, TVs, etc.) , Air conditioners, electricity meters, etc.), smart robots, workshop equipment, and various forms of user equipment (User Equipment, UE), mobile stations (Mobile station, MS), terminal equipment (Terminal Equipment), etc.
  • handheld devices for example, mobile phones, tablets, palmtop computers, etc.
  • vehicle-mounted devices for example, wearable devices (for example, smart watches, smart bracelets, pedometers, etc.), smart home devices (for example, refrigerators, TVs, etc.) , Air conditioners, electricity meters, etc.), smart robots, workshop equipment, and various forms of user equipment (User Equipment, UE), mobile stations (Mobile station, MS), terminal equipment (Terminal Equipment), etc.
  • wearable devices for example, smart watches, smart bracelets, pedometers
  • system architecture involved in the image rendering and encoding method is not limited to the system architecture shown in FIG. 2.
  • FIG. 3 is an image rendering and encoding method provided by an embodiment of the present invention. The method includes but is not limited to the following steps:
  • Step 301 The first processor sends the data to be rendered to the second processor.
  • Step 302 The second processor renders data in a first format according to the data to be rendered, and stores the data in a first storage space.
  • the above-mentioned first processor may be a CPU, a microprocessor with no interlocking pipeline stage architecture, an advanced simplified instruction set machine, a field programmable gate array (FPGA), and the like.
  • the above-mentioned second processor may be a GPU, an artificial intelligence accelerator card, or other devices or equipment that can accelerate the completion of graphics processing.
  • the above-mentioned first processor sends the data to be rendered to the second processor, and after the second processor receives the data to be rendered, the first processor instructs the second processor according to The data to be rendered is rendered to obtain first format data, and the first format data is stored in the first storage space of the second processor.
  • the above-mentioned data to be rendered includes position information and texture information of the mark points of the image.
  • the position information of the mark points may be coordinate information of four vertices of a rectangle in the image to be rendered.
  • the texture information may include the color information of the filling in the image that needs to be rendered, that is, it includes the color layout information in the image that needs to be rendered, and the like.
  • the above-mentioned first processor may instruct the second processor to perform rendering according to the above-mentioned data to be rendered by calling an application program interface for data rendering in a graphics library.
  • the image library may be an open graphics library (open graphics library, OpenGL) or the like.
  • OpenGL can be implemented through Mesa3D, which is a 3D computer graphics library with open source code under the license of the Massachusetts Institute of Technology (massachusetts institute of technology, MIT), which can implement OpenGL in open source form interface.
  • Mesa3D is a 3D computer graphics library with open source code under the license of the Massachusetts Institute of Technology (massachusetts institute of technology, MIT), which can implement OpenGL in open source form interface.
  • the above-mentioned first format data may include the color characteristics of the three primary colors of the image.
  • the above-mentioned first format data may be image data in RGBA format.
  • RGBA is a color space representing red (Red), green (Green), blue (Blue) and Alpha.
  • the alpha channel can be used as an opacity parameter. If the alpha channel value of a pixel is 0%, it is completely transparent (that is, invisible), and a value of 100% means a completely opaque pixel (traditional digital image). A value between 0% and 100% allows pixels to be displayed through the background, just like through glass (translucency). This effect is simply not possible with binary transparency (transparency or opacity) of. It makes digital synthesis easy.
  • the alpha channel value can be expressed as a percentage, an integer, or a real number from 0 to 1 like the RGB parameter.
  • the above-mentioned first storage space may be that the above-mentioned first processor applies to the second processor for storing the above-mentioned first format data by calling an application program interface for applying for storage space in the graphics library. Storage space.
  • the first storage space may be a video memory space for storing the data in the first format.
  • the first processor involved in this solution may be a processor in a server (including a cloud server), or a processor of a device that can complete the rendering and coding operation and interact with the client.
  • the device where the processor is located is not specifically limited, as long as it is a device that can realize the functions of this solution, it is the device protected by this solution.
  • this solution does not impose specific restrictions on the device where the second processor is located. Any device that can implement the functions of this solution is the device protected by this solution.
  • Step 303 The second processor converts the first format data in the first storage space into a second format data.
  • the first processor may instruct the second processor to convert the data in the first format into data in the second format.
  • the foregoing first processor may instruct the second processor to convert the foregoing first format data by calling an application program interface for format conversion in the video acceleration library.
  • the video acceleration library can be used to provide video hardware encoding and decoding functions, and can also be used to convert data in the first format into data in the second format.
  • the video acceleration library may include a video acceleration application program interface (video acceleration API, VAAPI), etc.
  • video acceleration API video acceleration API
  • VAAPI video acceleration application program interface
  • VAAPI may be implemented through Mesa3D, which may implement a video acceleration (VA) application program interface in an open source form.
  • VA video acceleration
  • the above-mentioned second format data may include the luminance value and the chrominance value of the image.
  • the above-mentioned second format data may be frame data in YUV format.
  • YUV is a color coding method, which is often used in various video processing components. When YUV encodes photos or videos, it takes into account human perception and allows the bandwidth of chroma to be reduced.
  • YUV is a format used to compile image data. Proper nouns such as Y'UV, YUV, YCbCr, YPbPr, etc. can all be called YUV, which overlap with each other.
  • Y represents brightness (Luminance or Luma), which is the grayscale value
  • U and V represent chrominance (Chrominance or Chroma), which are used to describe the color and saturation of the image, and are used to specify pixels s color.
  • the above-mentioned graphics library includes an application program interface (which may be referred to as a first interface) configured to obtain storage resources for obtaining the above-mentioned first format data.
  • the above-mentioned first processor may directly call the first interface to obtain the storage resource of the above-mentioned first format data.
  • the storage resource may include the storage address information of the first format data, that is, the specific address information of the above-mentioned first storage space, and may also include information such as the capacity of the first format data.
  • the storage resource may be a video memory resource
  • the video memory resource may include the storage address information of the video memory space where the data in the first format is located, and may also include data in the first format. Capacity and other information.
  • the aforementioned video acceleration library includes an application program interface (may be referred to as a second interface) configured to receive the storage resource sent by the aforementioned first processor calling the aforementioned first interface, that is, the aforementioned first interface obtains the first After the storage resource of the format data, the storage resource is sent to the second interface.
  • the second interface may instruct the second processor to mark the storage resource in a region to be encoded, such as an encoding buffer, so as to wait for format conversion to perform encoding.
  • the area to be encoded includes a storage space for storing the data to be encoded and/or the storage address of the data to be encoded.
  • the above-mentioned first interface may be a newly added application in the graphics library that cannot realize the function of acquiring the storage resources of the above-mentioned first format data and sending the storage resources to the above-mentioned second interface.
  • a program interface which is specifically used to obtain the storage resource of the above-mentioned first format data and send the storage resource to the above-mentioned second interface.
  • the second interface may be an application program interface added to the video acceleration library that cannot realize the function of receiving the storage resource from the first interface and marking the storage resource in the area to be encoded. The interface is specifically used to receive the storage resource from the first interface and mark the storage resource in the area to be coded.
  • a computer program can be added to the OpenGL state tracker of Mesa3D to implement the first interface, and the OpenGL state tracker manages the storage resources of image data.
  • a computer program can be added to the Mesa3D VA status tracker to implement the above second interface.
  • the VA status tracker manages the storage resources of the encoding buffer, which is used to manage the input data and output of the image/video encoding. data.
  • the first processor may call a third interface to apply for a second storage space from the second processor, and the second storage space is an address space for storing the data in the second format; Then, the third interface is called to instruct the second processor to convert the data in the first format into data in the second format.
  • the foregoing third interface may be an application program interface in the foregoing video acceleration library, and specifically may be a format conversion interface, which is used to convert the foregoing first format data into a second format data.
  • the first processor invokes the third interface to obtain the storage resource of the data in the first format through the second interface, it applies to the second processor for a storage space for storing the data in the first format.
  • the converted data that is, the second format data
  • call the third interface to obtain the above-mentioned first format data according to the above-mentioned storage resources, and then instruct the second processor to convert the image data into the second format data, and transfer the second format
  • the data is stored in the requested storage space.
  • the second storage space may be a video memory space for storing the data in the second format.
  • the rendered image data can be obtained through a special application program interface, so that the image data can be formatted on the second processor side, which avoids copying the rendered data to the first processing.
  • the operation of copying to the second processor after converting the format by the processor saves the resources of the first and second processors.
  • Step 304 The second processor encodes the second format data into third format data.
  • the first processor instructs the second processor to convert the data in the first format into data in the second format
  • it continues to instruct the second processor to encode the data in the second format into the third format. data.
  • the data capacity of the data in the third format is smaller than the data capacity of the data in the second format. Therefore, the bandwidth occupied by the data in the third format during transmission is less than that of the data in the second format. Bandwidth occupied during transmission.
  • the aforementioned third format data may be frame data in H.264 or H.265 format.
  • H.264 is a highly compressed digital video codec standard.
  • the main parts of the H.264 standard include access unit delimiter, additional enhancement information (SEI), and primary coded picture.
  • SEI additional enhancement information
  • redundant coded picture it may also include instantaneous decoding refresh (IDR), hypothetical reference decoder (HRD), and hypothetical stream scheduler (HSS).
  • H.265 is high efficiency video coding (HEVC), which is a new video coding standard developed after H.264.
  • the first processor invokes a fourth interface to apply to the second processor for a third storage space, and the third storage space is an address space for storing the third format data; , And then call the fourth interface to instruct the second processor to encode the data in the second format into data in the third format.
  • the foregoing fourth interface may be an application program interface in the foregoing video acceleration library, and specifically may be an encoding interface, which is used to encode the foregoing second format data into third format data.
  • the first processor calls the fourth interface to apply to the second processor for a storage space for storing the data in the second format.
  • Data encoded by the data that is, data in the third format
  • the fourth interface to instruct the second processor to encode the data in the second format to obtain data in the third format, and store the data in the third format in the requested storage space .
  • the third storage space may be a video memory space for storing data in the third format.
  • Step 305 The second processor copies the third format data from the storage space corresponding to the second processor to the storage space corresponding to the first processor.
  • the first processor may instruct the second processor to copy the data in the third format from the storage space corresponding to the second processor to the storage space corresponding to the first processor, that is, the first processor instructs the The second processor sends the third format data from the storage space corresponding to the second processor to the storage space corresponding to the first processor.
  • the storage space corresponding to the first processor may be a memory space.
  • the memory is the memory of the CPU, which is used to store programs and data when the process is running, and is also called executable memory.
  • the above-mentioned memory space may be an address space allocated in the memory by the above-mentioned CPU for storing the above-mentioned third format data.
  • Step 306 The first processor sends the third format data to the client.
  • the first processor instructs the second processor to copy the encoded third format data from the storage space of the second processor to the storage space of the first processor, and then the first processor The device sends the third format data to the corresponding client for display.
  • the implementation of this solution can be based on the second processor open source driving the Mesa3D state tracker layer, without relying on the second processor device related driver layer, that is, not relying on a specific second processor Model, as long as it is a second processor that implements OpenGL interface and VAAPI interface based on the open source driver Mesa3D, this solution can be used.
  • the open source driver Mesa3D of the second processor can be implemented based on Linux system, of course, can also be implemented based on other systems such as Berkeley Software Distribution (BSD), BSD is Unix Derivative system.
  • BSD Berkeley Software Distribution
  • This solution does not impose restrictions on the system or platform to which the application is implemented, as long as the system or platform can realize the functions of this solution.
  • the above-mentioned first interface may also be an application program interface independent of the graphics library, which can also realize the functions that the above-mentioned first interface can implement, that is, obtain the storage resources of the above-mentioned first format data, And send the storage resource to the above-mentioned second interface.
  • the above-mentioned second interface may also be an application program interface independent of the video acceleration library, and the second interface can perform data interaction with the application program interface in the graphics library, so the above-mentioned first interface can also be implemented.
  • the function that can be implemented by the second interface is to receive the above-mentioned storage resource from the above-mentioned first interface, and then send the storage resource to the video acceleration library, and then mark the storage resource in the encoding buffer.
  • the operations described in step 301 to step 306 may be operations implemented by the first processor executing a main program. Specifically, the first processor downloads the main program to the memory space, and then runs the main program, thereby implementing the operations described in steps 301 to 306 above.
  • the image data in the RGBA format of the rendering result is transmitted back to the first processor, and the first processor performs transcoding from the RGBA format to the YUV format, and then sends the YUV format data to the second processor.
  • the second processor transcodes YUV data to H.264 or H265, and transmits H.264 or H265 data back to the first processor.
  • This entire process has many steps and a long processing time. The entire process takes about 80ms on average. .
  • one frame of image is directly completed in the second processor after rendering, the process of RGBA to YUV format conversion, and YUV format to H264 or H265 encoding process, the process is small, the processing time is short, the average needs about 8ms to complete, and the performance
  • the existing technology is increased by 10 times, the time delay is low, the client can receive the picture update faster, and the picture update is smoother.
  • the prior art requires the first processor to perform the conversion from RGBA to YUV format, which will consume the resources of the first processor.
  • the second processor performs the conversion, which saves the resources of the first processor.
  • the processor performs this type of conversion faster and more efficiently than the first processor.
  • the scenario shown in FIG. 4 includes a cloud mobile phone client and a server side, where a cloud mobile phone server runs on the server, and the cloud mobile phone service may be an Android virtual machine.
  • the cloud mobile phone service may include a control module, a picture capture service module, and a video streaming service module; the cloud mobile phone service may also include an image display module and an encoding library.
  • the server may also include a GPU.
  • the GPU may also be a graphics processor that is independent of the server but can perform data interaction with the server.
  • the process may include but is not limited to the following steps:
  • the cloud mobile phone client obtains the touch information in response to the touch operation of the hardware.
  • the touch information may include the position information of the touch, and the information that needs to be obtained according to the touch matching to obtain the displayed image data, etc. .
  • the cloud mobile phone client sends the touch information to the server via the network.
  • the server After receiving the touch information, the server triggers the control module of the cloud phone service to process the touch information to obtain processed data.
  • the server triggers the image display module of the cloud phone service to request the GPU to render the processed data to obtain the rendered image data in RGBA format.
  • the GPU continues to convert the rendered image data in RGBA format into frame data in YUV format.
  • GPU continues to encode the converted YUV format frames into H.264 or H.265 format data.
  • the GPU returns the encoded H.264 or H.265 frame data to the cloud mobile phone service video streaming service module.
  • the cloud mobile phone service video streaming service module sends H.264 or H.265 frame data to the cloud mobile phone client through the network.
  • the cloud mobile phone client After the cloud mobile phone client receives the H.264 or H.265 frame data, it decodes the video.
  • the cloud mobile phone client displays the decoded data on the display screen.
  • each network element such as a first processor, a second processor, etc.
  • each network element includes hardware structures and/or software modules corresponding to each function.
  • the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a certain function is executed by hardware or computer software-driven hardware depends on the specific application and design constraint conditions of the technical solution. Professionals and technicians can use different methods for each specific application to implement the described functions, but such implementation should not be considered beyond the scope of this application.
  • the embodiment of the present application may divide the first processor, the second processor, etc. into functional modules according to the above method examples.
  • each functional module may be divided corresponding to each function, or two or more functions may be integrated in In a processing module.
  • the above-mentioned integrated modules can be implemented in the form of hardware or software function modules. It should be noted that the division of modules in the embodiments of the present application is illustrative, and is only a logical function division, and there may be other division methods in actual implementation.
  • FIG. 5 shows a schematic diagram of a possible logical structure of the first processor involved in the foregoing method embodiment, and the first processor 500 includes: a sending unit 501, Instructing unit 502 and calling unit 503.
  • the sending unit 501 is configured to execute the steps of sending information by the first processor shown in steps 301 and 306 in the method embodiment shown in FIG. 3;
  • the instructing unit 502 is configured to execute the method implementation shown in FIG.
  • the first processor shown in step 302, step 303, step 304, and step 305 instructs the second processor to complete the steps of image rendering, format conversion, encoding, and copying;
  • the calling unit 503 is used to execute the method shown in FIG. 3 above
  • the steps in which the first processor calls the interface shown in possible implementations of step 302, step 303, and step 304 are shown.
  • the sending unit 501 is configured to send data to be rendered to the second processor; the instructing unit 502 is configured to instruct the second processor to render data in the first format according to the data to be rendered,
  • the first format data includes the color characteristics of the three primary colors of the image; the first format data is stored in the first storage space of the second processor; the second processor is instructed to store all the data in the first storage space. Converting the first format data into second format data, the second format data including the luminance value and chrominance value of the image; and instructing the second processor to encode the second format data into third format data,
  • the data capacity of the third format data is smaller than the data capacity of the second format data; the sending unit 501 is further configured to send the third format data to the client.
  • FIG. 6 shows a schematic diagram of a possible logical structure of the second processor involved in the foregoing method embodiment
  • the second processor 600 includes: a receiving unit 601, The rendering unit 602, the conversion unit 603, the encoding unit 604, and the sending unit 605.
  • the receiving unit 601 is configured to execute the step of receiving information by the second processor shown in step 302 in the foregoing method embodiment shown in FIG. 3
  • the rendering unit 602 is configured to execute the steps in the foregoing method embodiment shown in FIG. 3
  • the conversion unit 603 is used to perform the step of converting the format of the second processor shown in step 303 in the method embodiment shown in FIG.
  • the encoding unit 604 is used to execute the aforementioned image
  • the sending unit 605 is configured to execute the step of sending information by the second processor shown in step 305 in the method embodiment shown in FIG. 3.
  • the receiving unit 601 is configured to receive data to be rendered sent by the first processor; the rendering unit 602 is configured to render data in a first format according to the data to be rendered, and the first format data includes three parts of the image. Primary color color characteristics; the first format data is stored in the first storage space of the processor; the conversion unit 603 is configured to convert the first format data in the first storage space into second format data The second format data includes the luminance value and the chrominance value of the image; the encoding unit 604 is configured to encode the second format data into third format data, and the data capacity of the third format data is smaller than that of the first format data. The data capacity of the second format data; the sending unit 605 is used to send the third format data.
  • the embodiment of the present invention also provides a corresponding processor.
  • the processor may be the first processor described in FIG. 3, and the processor performs the following operations:
  • the second processor encodes the data in the second format into data in the third format, and the data capacity of the data in the third format is smaller than the data capacity of the data in the second format; and sends the data in the third format to the client.
  • the method further includes: the processor calls the first interface to obtain the storage address of the data in the first format in the first storage space; the processor calls the first interface to send the storage address to the second Interface; the processor calls the second interface and instructs the second processor to mark the storage address in the area to be encoded, and the area to be encoded includes storage for storing the data to be encoded and/or the storage address of the data to be encoded space.
  • the above-mentioned first interface is an application program interface configured in the graphics library for obtaining the above-mentioned storage address.
  • the second interface is configured in the video acceleration library for receiving the storage address sent by the first interface and instructing the second processor to mark the storage address to the to-be-coded Application program interface in the zone.
  • the processor instructing the second processor to convert the data in the first format into the data in the second format includes: the processor invoking a third interface to apply to the second processor for a second Storage space, the second storage space is an address space for storing the data in the second format; the third interface is an application program interface in the video acceleration library; the processor calls the third interface to instruct the second processor to The above-mentioned first format data is converted into second format data.
  • the processor instructing the second processor to encode the data in the second format into the data in the third format includes: the processor invoking a fourth interface to apply to the second processor for third Storage space, the third storage space is an address space for storing the third format data; the fourth interface is an application program interface in the video acceleration library; the processor calls the fourth interface to instruct the second processor to The above-mentioned second format data is encoded into third format data.
  • the embodiment of the present invention also correspondingly provides a processor.
  • the processor may be the second processor described in FIG. 3, and the processor performs the following operations:
  • Receiving data to be rendered rendering data in a first format according to the data to be rendered, the data in the first format including the three primary color characteristics of the image; the data in the first format is stored in the first storage space of the processor; The data in the first format is converted into data in the second format, and the data in the second format includes the luminance value and the chrominance value of the image; the data in the second format is encoded into data in the third format, and the data capacity of the data in the third format is less than that of the above The data capacity of the data in the second format; sending the data in the third format mentioned above.
  • FIG. 7 is a server 700 provided by an embodiment of the present invention.
  • the server 700 includes a first processor 701 (which may be the first processor described in FIG. 3), a memory 702, a communication interface 703, and a second processor 705 (It may be the second processor described in FIG. 3).
  • the first processor 701, the memory 702, the communication interface 703, and the second processor 705 are connected to each other through a bus 704.
  • the memory 702 includes, but is not limited to, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), or Portable read-only memory (compact disc read-only memory, CD-ROM), the memory 702 is used for storage of related instructions and data.
  • the communication interface 703 is used to receive and send data.
  • the first processor 701 in the server 700 is configured to read the computer program stored in the memory 702, so as to execute the method embodiment in the foregoing FIG. 3 and its possible implementation manners.
  • the embodiment of the present invention also provides a computer-readable storage medium, the computer-readable storage medium stores a computer program, and the computer program is executed by a processor to implement the method embodiment described in FIG. 3 and its possible implementation manners.
  • the embodiment of the present invention also provides a computer program product.
  • the computer program product is read and executed by a computer, the method embodiment described in FIG. 3 and its possible implementation manners are realized.
  • the embodiment of the present invention also provides a computer program.
  • the computer program When the computer program is executed on a computer, the computer will enable the computer to implement the method embodiment described in FIG. 3 and its possible implementation manners.
  • the data in the first format rendered in the second processor needs to be copied to the memory of the first processor, and the data in the second format is converted into the data in the first processor. After that, it is copied to the memory of the second processor for encoding.
  • the embodiment of the present application can complete the process of rendering, format conversion, and encoding in the second processor without requiring two additional data transmissions. Therefore, the embodiment of the present application It can save the resources of the first processor and the second processor, and improve the efficiency of rendering and encoding.

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Software Systems (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Compression Or Coding Systems Of Tv Signals (AREA)

Abstract

本发明实施例提供一种图像渲染和编码方法及相关装置，该方法包括：首先，第一处理器将待渲染的数据发送给第二处理器；然后，指示上述第二处理器根据上述待渲染的数据渲染得到第一格式数据；上述第一格式数据存储在上述第二处理器的第一存储空间中；上述第一处理器指示上述第二处理器将上述第一格式数据转换为第二格式数据；并指示上述第二处理器将上述第二格式数据编码为第三格式数据，上述第三格式数据的数据容量小于上述第二格式数据的数据容量；然后，向客户端发送上述第三格式数据。采用本发明实施例，可以提高图像渲染和编码的效率。

Description

图像渲染和编码方法及相关装置 技术领域
本发明涉及图像数据处理技术领域，尤其涉及一种图像渲染和编码方法及相关装置。
背景技术
在云服务环境下，服务或者程序运行在服务器端做关键或复杂的运算，并将运算的结果传送给远端客户端，远端客户端负责接收服务器端的结果。在手机云游戏场景中，服务器对每帧图像进行渲染、编码后通过网络传送给远端的客户端，远端客户端接收云服务器端的帧数据后再解码显示。服务器端采用图形处理器(graphics processing unit，GPU)渲染游戏画面，渲染出来的图像是RGBA格式的图像数据(RGBA是代表红色(Red)、绿色(Green)、蓝色(Blue)和Alpha的色彩空间)，而一幅典型的720P图像的数据大小约为28M比特,这么大数据量在网络带宽有限的情况下很难处理，因此需要对图像流进行H.264或H.265视频编码压缩后再进行网络传输，以此降低网络带宽。
现有方案中渲染和编码的数据流程的具体实现可以如图1所示，在图1中GPU完成渲染操作后需要将渲染后得到的RGBA格式的图像数据复制到中央处理器(central processing unit，CPU)，由CPU将RGBA格式的图像数据转换成YUV格式的帧数据，然后CPU再将YUV格式帧数据复制到GPU的显存空间，然后在GPU中进行H264或H265的编码。在这个过程中，渲染操作之后的数据在CPU和GPU间需传输2次，即图1中步骤3和步骤5，耗费了额外的CPU和GPU资源，降低性能，降低了编码和渲染的效率。
综上所述，在图像渲染和编码的过程中如何节省CPU和GPU资源、提高效率是本领域技术人员需要解决的技术问题。
发明内容
本申请实施例公开了一种图像渲染和编码方法及相关装置，能够节省处理器的资源、提高渲染和编码的效率。
第一方面，本申请实施例公开了一种图像渲染和编码方法，该方法包括：
第一处理器将待渲染的数据发送给第二处理器；
上述第一处理器指示上述第二处理器根据上述待渲染的数据渲染得到第一格式数据，上述第一格式数据包括图像的三基色颜色特征；上述第一格式数据存储在上述第二处理器的第一存储空间中；
上述第一处理器指示上述第二处理器将上述第一格式数据转换为第二格式数据，上述第二格式数据包括图像的亮度值和色度值；
上述第一处理器指示上述第二处理器将上述第二格式数据编码为第三格式数据，上述第三格式数据的数据容量小于上述第二格式数据的数据容量；
上述第一处理器向客户端发送上述第三格式数据。
上述第一处理器可以是中央处理器CPU、无互锁管道阶段架构的微处理器、进阶精 简指令集机器以及现场可编程逻辑门阵列(field programmable gate array，FPGA)等。上述第二处理器可以是图形处理器GPU、人工智能加速卡等其它可以完成图形处理的装置或设备。
相比于现有技术中需要将在第二处理器中渲染好的第一格式的数据复制到第一处理器内存中，在第一处理器转换为第二格式的数据之后，再复制到第二处理器内存中进行编码，本申请实施例可以在第二处理器中完成渲染、格式转换和编码的过程，不需要额外两次的数据传输，因此通过本申请实施例能够节省处理器的资源、提高渲染和编码的效率。
在其中一种可能的实施方式中，在上述第一处理器指示上述第二处理器根据上述待渲染的数据渲染得到第一格式数据之后，上述第一处理器指示上述第二处理器将上述第一格式数据转换为第二格式数据之前，还包括：
上述第一处理器调用第一接口获取上述第一格式数据在上述第一存储空间中的存储地址；
上述第一处理器调用上述第一接口将上述存储地址发送给第二接口；
上述第一处理器调用上述第二接口指示上述第二处理器将上述存储地址标记到待编码区中，上述待编码区包括用于存放待编码的数据和/或待编码数据的存储地址的存储空间。
在本申请实施例中，可以通过专门的应用程序接口获取渲染之后的图像数据，以使得可以在第二处理器侧对该图像数据进行格式转换，避免了将渲染后的数据复制到第一处理器转换格式后再复制到第二处理器的操作，节省了第一处理器和第二处理器的资源。
在其中一种可能的实施方式中，上述第一接口为图形库中配置的用于获取上述存储地址的应用程序接口。
在其中一种可能的实施方式中，上述第二接口为视频加速库中配置的用于接收上述第一接口发送过来的上述存储地址以及指示上述第二处理器将上述存储地址标记到上述待编码区中的应用程序接口。
在其中一种可能的实施方式中，上述第一处理器调用第三接口向上述第二处理器申请第二存储空间，上述第二存储空间为用于存放上述第二格式数据的地址空间；上述第三接口为视频加速库中的应用程序接口；上述第一处理器调用上述第三接口指示上述第二处理器将上述第一格式数据转换为第二格式数据。
在其中一种可能的实施方式中，上述第一处理器指示上述第二处理器将上述第二格式数据编码为第三格式数据，包括：
上述第一处理器调用第四接口向上述第二处理器申请第三存储空间，上述第三存储空间为用于存放上述第三格式数据的地址空间；上述第四接口为视频加速库中的应用程序接口；
上述第一处理器调用上述第四接口指示上述第二处理器将上述第二格式数据编码为第三格式数据。
第二方面，本申请实施例公开了一种图像渲染和编码方法，该方法应用于处理器，该方法包括：
接收待渲染的数据；
根据上述待渲染的数据渲染得到第一格式数据，上述第一格式数据包括图像的三基色颜色特征；上述第一格式数据存储在上述处理器的第一存储空间中；
将上述第一格式数据转换为第二格式数据，上述第二格式数据包括图像的亮度值和色度值；
将上述第二格式数据编码为第三格式数据，上述第三格式数据的数据容量小于上述第二格式数据的数据容量；
发送上述第三格式数据。
第三方面，本申请实施例公开了一种处理器，该处理器执行如下操作：
将待渲染的数据发送给第二处理器；
指示上述第二处理器根据上述待渲染的数据渲染得到第一格式数据，上述第一格式数据包括图像的三基色颜色特征；上述第一格式数据存储在上述第二处理器的第一存储空间中；
指示上述第二处理器将上述第一格式数据转换为第二格式数据，上述第二格式数据包括图像的亮度值和色度值；
指示上述第二处理器将上述第二格式数据编码为第三格式数据，上述第三格式数据的数据容量小于上述第二格式数据的数据容量；
向客户端发送上述第三格式数据。
在其中一种可能的实施方式中，在上述处理器指示上述第二处理器根据上述待渲染的数据渲染得到第一格式数据之后，上述处理器指示上述第二处理器将上述第一格式数据转换为第二格式数据之前，还包括：
上述处理器调用第一接口获取上述第一格式数据在上述第一存储空间中的存储地址；
上述处理器调用上述第一接口将上述存储地址发送给第二接口；
上述处理器调用上述第二接口指示上述第二处理器将上述存储地址标记到待编码区中，上述待编码区包括用于存放待编码的数据和/或待编码数据的存储地址的存储空间。
在其中一种可能的实施方式中，上述第一接口为图形库中配置的用于获取上述存储地址的应用程序接口。
在其中一种可能的实施方式中，上述第二接口为视频加速库中配置的用于接收上述第一接口发送过来的上述存储地址以及指示上述第二处理器将上述存储地址标记到上述待编码区中的应用程序接口。
在其中一种可能的实施方式中，上述处理器指示上述第二处理器将上述第一格式数据转换为第二格式数据，包括：
上述处理器调用第三接口向上述第二处理器申请第二存储空间，上述第二存储空间为用于存放上述第二格式数据的地址空间；上述第三接口为视频加速库中的应用程序接口；
上述处理器调用上述第三接口指示上述第二处理器将上述第一格式数据转换为第二格式数据。
在其中一种可能的实施方式中，上述处理器指示上述第二处理器将上述第二格式数据编码为第三格式数据，包括：
上述处理器调用第四接口向上述第二处理器申请第三存储空间，上述第三存储空间 为用于存放上述第三格式数据的地址空间；上述第四接口为视频加速库中的应用程序接口；
上述处理器调用上述第四接口指示上述第二处理器将上述第二格式数据编码为第三格式数据。
第四方面，本申请实施例公开了一种处理器，该处理器执行如下操作：
接收待渲染的数据；
根据上述待渲染的数据渲染得到第一格式数据，上述第一格式数据包括图像的三基色颜色特征；上述第一格式数据存储在上述处理器的第一存储空间中；
将上述第一格式数据转换为第二格式数据，上述第二格式数据包括图像的亮度值和色度值；
将上述第二格式数据编码为第三格式数据，上述第三格式数据的数据容量小于上述第二格式数据的数据容量；
发送上述第三格式数据。
第五方面，本申请实施例公开了一种处理器，该处理器包括：
发送单元，用于将待渲染的数据发送给第二处理器；
指示单元，用于：指示上述第二处理器根据上述待渲染的数据渲染得到第一格式数据，上述第一格式数据包括图像的三基色颜色特征；上述第一格式数据存储在上述第二处理器的第一存储空间中；
指示上述第二处理器将上述第一存储空间中的上述第一格式数据转换为第二格式数据，上述第二格式数据包括图像的亮度值和色度值；以及
指示上述第二处理器将上述第二格式数据编码为第三格式数据，上述第三格式数据的数据容量小于上述第二格式数据的数据容量；
上述发送单元，还用于向客户端发送上述第三格式数据。
在其中一种可能的实施方式中，上述处理器还包括；
调用单元，用于：在所述指示单元指示所述第二处理器根据所述待渲染的数据渲染得到第一格式数据之后，指示所述第二处理器将所述第一格式数据转换为第二格式数据之前，调用第一接口获取所述第一格式数据在所述第一存储空间中的存储地址；
调用所述第一接口将所述存储地址发送给第二接口；以及
调用所述第二接口指示所述第二处理器将所述存储地址标记到待编码区中，所述待编码区包括用于存放待编码的数据和/或待编码数据的存储地址的存储空间。
在其中一种可能的实施方式中，上述第一接口为图形库中配置的用于获取上述存储地址的应用程序接口。
在其中一种可能的实施方式中，上述第二接口为视频加速库中配置的用于接收上述第一接口发送过来的上述存储地址以及指示上述第二处理器将上述存储地址标记到上述待编码区中的应用程序接口。
在其中一种可能的实施方式中，上述指示单元具体用于：
调用第三接口向上述第二处理器申请第二存储空间，上述第二存储空间为用于存放上述第二格式数据的地址空间；上述第三接口为视频加速库中的应用程序接口；
调用上述第三接口指示上述第二处理器将上述第一格式数据转换为第二格式数据。
在其中一种可能的实施方式中，上述指示单元具体还用于：
调用第四接口向上述第二处理器申请第三存储空间，上述第三存储空间为用于存放上述第三格式数据的地址空间；上述第四接口为视频加速库中的应用程序接口；
调用上述第四接口指示上述第二处理器将上述第二格式数据编码为第三格式数据。
第六方面，本申请实施例公开了一种处理器，该处理器包括：
接收单元，用于接收第一处理器发送的待渲染的数据；
渲染单元，用于根据上述待渲染的数据渲染得到第一格式数据，上述第一格式数据包括图像的三基色颜色特征；上述第一格式数据存储在上述处理器的第一存储空间中；
转换单元，用于将上述第一存储空间中的上述第一格式数据转换为第二格式数据，上述第二格式数据包括图像的亮度值和色度值；
编码单元，用于将上述第二格式数据编码为第三格式数据，上述第三格式数据的数据容量小于上述第二格式数据的数据容量；
发送单元，用于发送上述第三格式数据。
第七方面，本申请实施例公开了一种服务器，该服务器包括第一处理器、第二处理器和存储器，该存储器存储有计算机程序，上述第一处理器用于调用该计算机程序来执行上述第一方面任一项所述的方法。
第八方面，本申请实施例公开了一种计算机可读存储介质，该计算机可读存储介质中存储有计算机程序，当该程序被处理器执行时，实现上述第一方面任一项所述的方法。
综上所述，相比于现有技术中需要将在第二处理器中渲染好的第一格式的数据复制到第一处理器内存中，在第一处理器转换为第二格式的数据之后，再复制到第二处理器内存中进行编码，本申请实施例可以在第二处理器中完成渲染、格式转换和编码的过程，不需要额外两次的数据传输，因此通过本申请实施例能够节省处理器的资源、提高渲染和编码的效率。
附图说明
下面将对本申请实施例中所需要使用的附图作介绍。
图1是现有技术中的一种渲染和编码方法的交互示意图；
图2是本发明实施例提供的一种系统架构示意图；
图3是本发明实施例提供的一种渲染和编码方法的交互示意图；
图4是本发明实施例提供的一种客户端与服务器交互的场景示意图；
图5是本发明实施例提供的一种处理器的结构示意图；
图6是本发明实施例提供的另一种处理器的结构示意图；
图7是本发明实施例提供的一种服务器的结构示意图。
具体实施方式
下面结合本发明实施例中的附图对本发明实施例进行描述。
为了更好的理解本发明实施例提供的一种图像渲染和编码方法及相关装置，下面先对本发明实施例涉及的系统架构进行描述。如图2所示的系统架构，可以包括服务器201和客户端202。其中：服务器201主要用于根据客户端202的请求对客户端202需要的图 像或视频进行渲染、编码后发送给客户端202显示。
服务器201可以包括但不限于后台服务器、组件服务器、图像渲染和编码服务器等。服务器201上需要运行有相应的服务器端程序来提供相应的图像渲染和编码服务，如用于数据计算、决策执行的程序等等。
客户端202可以包括手持设备(例如，手机、平板电脑、掌上电脑等)、车载设备、可穿戴设备(例如智能手表、智能手环、计步器等)、智能家居设备(例如，冰箱、电视、空调、电表等)、智能机器人、车间设备，以及各种形式的用户设备(User Equipment，UE)、移动台(Mobile station，MS)、终端设备(Terminal Equipment)，等等。
需要说明的是，本申请实施例提供的图像渲染和编码方法所涉及的系统构架不限于图2所示系统架构。
请参见图3，图3是本发明实施例提供的一种图像渲染和编码方法，该方法包括但不限于如下步骤：
步骤301、第一处理器将待渲染的数据发送给第二处理器。
步骤302、所述第二处理器根据所述待渲染的数据渲染得到第一格式数据，并存储在第一存储空间。
可选的，上述第一处理器可以是CPU、无互锁管道阶段架构的微处理器、进阶精简指令集机器以及现场可编程逻辑门阵列(field programmable gate array，FPGA)等。上述第二处理器可以是GPU、人工智能加速卡等其它可以加速完成图形处理的装置或设备。
在具体的实施例中，上述第一处理器将待渲染的数据发送给第二处理器，该第二处理器接收到该待渲染的数据之后，该第一处理器指示该第二处理器根据该待渲染的数据渲染得到第一格式数据，并将该第一格式数据存储在第二处理器的第一存储空间中。
可选的，上述待渲染的数据包括图像的标志点的位置信息和纹理信息等，例如，该标志点的位置信息可以是需要渲染得到的图像中的矩形的四个顶点的坐标信息等等。该纹理信息可以包括需要渲染得到的图像中的填充的颜色信息，即包括需要渲染得到的图像中的颜色布局信息等。
在其中一种可能的实施方式中，上述第一处理器可以通过调用图形库中用于数据渲染的应用程序接口来指示第二处理器根据上述待渲染的数据进行渲染。
可选的，该图像库可以为开放图形库(open graphics library，OpenGL)等。
可选的，上述OpenGL可以通过Mesa3D来实现，该Mesa3D是一个在麻省理工学院(massachusetts institute of technology,MIT)许可证下开放源代码的三维计算机图形库，可以以开源形式实现OpenGL的应用程序接口。
在其中一种可能的实现方式中，上述第一格式数据可以包括图像的三基色颜色特征。
可选的，上述第一格式数据可以是RGBA格式的图像数据。其中，RGBA是代表红色(Red)、绿色(Green)、蓝色(Blue)和Alpha的色彩空间。该alpha通道可以用作不透明度参数。如果一个像素的alpha通道数值为0％，那它就是完全透明的(也就是看不见的)，而数值为100％则意味着一个完全不透明的像素(传统的数字图像)。在0％和100％之间的值，则使得像素可以透过背景显示出来，就像透过玻璃(半透明性)，这种效果是简单的二元透明性(透明或不透明)做不到的。它使数码合成变得容易。alpha通道值可以用百分比、整数或者像RGB参数那样用0到1的实数表示。
在其中一种可能的实施方式中，上述第一存储空间可以是上述第一处理器通过调用图形库中用于申请存储空间的应用程序接口向第二处理器申请用于存放上述第一格式数据的存储空间。
可选的，如果第二处理器是GPU，则上述第一存储空间可以是用于存放上述第一格式数据的显存空间。
可选的，本方案所涉及的第一处理器可以是服务器(包括云服务器)中的处理器，也可以是能够完成渲染编码操作并与客户端交互的设备的处理器，本方案对第一处理器所在的设备不做具体限制，只要是能够实现本方案的功能的设备都是本方案所保护的设备。同样的，本方案对第二处理器所在的设备不做具体限制，只要是能够实现本方案的功能的设备都是本方案所保护的设备。
步骤303、所述第二处理器将所述第一存储空间中的所述第一格式数据转换为第二格式数据。
在具体的实施例中，可以由上述第一处理器指示所述第二处理器将上述第一格式数据转换为第二格式数据。
在其中一种可能的实施方式中，上述第一处理器可以通过调用视频加速库中用于格式转换的应用程序接口来指示第二处理器对上述第一格式数据进行转换。该视频加速库可以用于提供视频硬件编解码功能，同时也可以用于将第一格式数据转换为第二格式数据。
可选的，该视频加速库可以包括视频加速应用程序接口(video acceleration API，VAAPI)等。
可选的，上述VAAPI可以通过Mesa3D来实现，该Mesa3D可以以开源形式实现视频加速(video acceleration,VA)的应用程序接口。
在其中一种可能的实现方式中，上述第二格式数据可以包括图像的亮度值和色度值。
可选的，上述第二格式数据可以是YUV格式的帧数据。其中，YUV是一种颜色编码方法，常使用在各个视频处理组件中。YUV在对照片或视频编码时，考虑到人类的感知能力，允许降低色度的带宽。YUV是用于编译图像数据的一种格式，Y'UV、YUV、YCbCr、YPbPr等专有名词都可以称为YUV，彼此有重叠。“Y”表示明亮度(Luminance或Luma)，也就是灰阶值，“U”和“V”表示的则是色度(Chrominance或Chroma)，作用是描述影像色彩及饱和度，用于指定像素的颜色。
在其中一种可能的实施方式中，上述图形库中包括配置用于获取上述第一格式数据的存储资源的应用程序接口(可以称为第一接口)。上述第一处理器可以直接调用该第一接口获取上述第一格式数据的存储资源。该存储资源可以包括该第一格式数据的存储地址信息，即上述第一存储空间的具体地址信息，还可以包括该第一格式数据的容量大小等信息。
可选的，如果第二处理器是GPU，则上述存储资源可以是显存资源，该显存资源可以包括上述第一格式数据的所在的显存空间的存储地址信息，还可以包括该第一格式数据的容量大小等信息。
此外，上述视频加速库包括配置用于接收上述第一处理器调用上述第一接口发送过来的所述存储资源的应用程序接口(可以称为第二接口)，即上述第一接口获取到第一 格式数据的存储资源后，将该存储资源发送给该第二接口。该第二接口获取该存储资源之后，可以指示第二处理器将所述存储资源标记到待编码区例如编码缓冲区中，以等待格式转换后进行编码。该待编码区包括用于存放待编码的数据和/或待编码数据的存储地址的存储空间。
在其中一种可能的实施方式中，上述第一接口可以是在不能实现获取上述第一格式数据的存储资源、并将存储资源发送给上述第二接口这一功能的图形库中新增加的应用程序接口，该接口专门用于获取上述第一格式数据的存储资源，并将存储资源发送给上述第二接口。此外，上述第二接口可以是在不能实现从上述第一接口中接收上述存储资源、并将所述存储资源标记到待编码区中这一功能的视频加速库中新增的应用程序接口，该接口专门用于从上述第一接口中接收上述存储资源、并将所述存储资源标记到待编码区中。
在其中一种可能的实现方式中，可以在上述Mesa3D的OpenGL状态跟踪器里添加计算机程序实现上述第一接口，该OpenGL状态跟踪器管理着图像数据的存储资源。此外，可以在上述Mesa3D的VA状态跟踪器里添加计算机程序实现上述第二接口，VA状态跟踪器管理着编码缓冲区的存储资源，该编码缓冲区用于管理图像/视频编码的输入数据和输出数据。
在其中一种可能的实施方式中，上述第一处理器可以调用第三接口向上述第二处理器申请第二存储空间，上述第二存储空间为用于存放上述第二格式数据的地址空间；然后，再调用上述第三接口指示上述第二处理器将上述第一格式数据转换为第二格式数据。
可选的，上述第三接口可以为上述视频加速库中的应用程序接口，具体的可以是格式转换接口，该接口用于将上述第一格式数据转换为第二格式数据。
在具体实施例中，上述第一处理器调用第三接口通过上述第二接口获取到上述第一格式数据的存储资源之后，向第二处理器申请一个存储空间用于存放将第一格式数据格式转换后的数据，即第二格式数据，然后调用第三接口根据上述存储资源获取上述第一格式数据，然后指示第二处理器将该图像数据转换为第二格式数据，并将该第二格式数据存储在申请的存储空间中。
可选的，如果第二处理器是GPU，则上述第二存储空间可以是用于存放上述第二格式数据的显存空间。
在本申请实施例中，可以通过专门的应用程序接口获取渲染之后的图像数据，以使得可以在第二处理器侧对该图像数据进行格式转换，避免了将渲染后的数据复制到第一处理器转换格式后再复制到第二处理器的操作，节省了第一处理器和第二处理器的资源。
步骤304、所述第二处理器将所述第二格式数据编码为第三格式数据。
在具体实施例中，上述第一处理器指示所述第二处理器将上述第一格式数据转换为第二格式数据之后，继续指示该第二处理器将该第二格式数据编码为第三格式数据。
在其中一种可能的实现方式中，上述第三格式数据的数据容量小于上述第二格式数据的数据容量，因此，上述第三格式数据在传输过程中占用的带宽小于所述第二格式数据在传输过程中占用的带宽。
可选的，上述第三格式数据可以是H.264或H.265格式的帧数据。其中，H.264是一种高度压缩数字视频编解码器标准，H.264标准各主要部分包括访问单元分割符(access  unit delimiter)、附加增强信息(SEI)、基本图像编码(primary coded picture)和冗余图像编码(redundant coded picture)，还可以包括即时解码刷新(instantaneous decoding refresh，IDR)、假想参考解码(hypothetical reference decoder，HRD)和假想码流调度器(hypothetical stream hcheduler，HSS)。此外，H.265为高效率视频编码(high efficiency video coding，HEVC)，是继H.264之后所制定的新的视频编码标准。
在其中一种可能的实施方式中，上述第一处理器调用第四接口向上述第二处理器申请第三存储空间，上述第三存储空间为用于存放上述第三格式数据的地址空间；然后，再调用上述该第四接口指示上述第二处理器将上述第二格式数据编码为第三格式数据。
可选的，上述第四接口可以为上述视频加速库中的应用程序接口，具体的可以是编码接口，该接口用于将上述第二格式数据编码为第三格式数据。
在具体实施例中，在上述第三接口将上述第一格式数据转换为第二格式数据之后，上述第一处理器调用第四接口向第二处理器申请一个存储空间用于存放将第二格式数据编码后的数据，即第三格式数据，然后调用第四接口指示上述第二处理器将该第二格式数据编码得到第三格式数据，并将该第三格式数据存储在申请的存储空间中。
可选的，如果第二处理器是GPU，则上述第三存储空间可以是用于存放上述第三格式数据的显存空间。
步骤305、所述第二处理器将上述第三格式数据从第二处理器对应的存储空间复制到第一处理器对应的存储空间。
具体的，可以是第一处理器指示所述第二处理器将上述第三格式数据从第二处理器对应的存储空间复制到第一处理器对应的存储空间，即第一处理器指示所述第二处理器将上述第三格式数据从第二处理器对应的存储空间发送到第一处理器对应的存储空间。
可选的，如果上述第一处理器为CPU，则第一处理器对应的存储空间可以是内存空间。具体的，内存为CPU的存储器，用于保存进程运行时的程序和数据，也称可执行存储器。上述内存空间可以是上述CPU在内存中分配的用于存放上述第三格式数据的地址空间。
步骤306、所述第一处理器将上述第三格式数据发送到客户端。
在具体的实施例中，上述第一处理器指示上述第二处理器将编码得到的第三格式数据从第二处理器的存储空间复制到第一处理器的存储空间中，然后，第一处理器将该第三格式数据发送给对应的客户端以用于显示。
在其中一种可能的实施方式中，本方案的实现可以基于第二处理器开源驱动Mesa3D的状态跟踪器一层，没有依赖第二处理器设备相关驱动层，即不依赖特定的第二处理器型号，只要是基于开源驱动Mesa3D实现了OpenGL接口和VAAPI接口的第二处理器，都可以使用本方案。
在其中一种可能的实现方式中，上述第二处理器的开源驱动Mesa3D可以基于Linux系统来实现，当然也可以基于其它的系统例如伯克利软件套件(Berkeley Software Distribution，BSD)来实现，BSD是Unix的衍生系统。本方案对实现所应用的系统或平台不做限制，只要可以实现本方案的功能的系统或平台都可以。
在其中一种可能的实现方式中，上述第一接口也可以是独立于图形库的应用程序接口，其同样可以实现上述第一接口可以实现的功能，即获取上述第一格式数据的存储资 源，并将存储资源发送给上述第二接口。
在其中一种可能的实现方式中，上述第二接口也可以是独立于视频加速库的应用程序接口，该第二接口可以与图形库中的应用程序接口进行数据交互，因此同样可以实现上述第二接口可以实现的功能，即从上述第一接口中接收上述存储资源，然后将该存储资源发送给视频加速库，再将所述存储资源标记到编码缓冲区中。
在其中一种可能的实现方式中，上述步骤301至步骤306所述的操作可以是上述第一处理器执行一个主程序所实现的操作。具体的，该第一处理器将该主程序下载到内存空间，然后运行该主程序，从而实现了上述步骤301至步骤306所述的操作。
综上所述，现有技术中将渲染结果RGBA格式的图像数据传回第一处理器侧、第一处理器做RGBA格式到YUV格式的转码、再将YUV格式数据发送到第二处理器、第二处理器做YUV数据到H.264或H265转码、将H.264或H265数据传回第一处理器，这整个流程步骤较多，处理时间很长，整套流程下来平均需要80ms左右。本发明中一帧图像在渲染后直接在第二处理器内完成RGBA到YUV格式转换、YUV格式到H264或H265编码的流程，流程少，处理所需时间短，平均需8ms左右完成，性能较现有技术提升10倍，时延低，客户端能够更快的接收到画面更新，使得画面更新更流畅。
此外，现有技术额外的两次在第一处理器存储空间和第二处理器存储空间之间的数据复制，在多路云手机并发的情况下，需要消耗较大的第一处理器和第二处理器负载去做数据搬移，导致云手机并发密度受限。本发明方案减少了上述的两次数据复制，节省了第一处理器和第二处理器资源，增加了云手机并发密度。
而且，现有技术需要第一处理器做RGBA到YUV格式的转换，会耗费第一处理器资源，本发明方案由第二处理器来做该转换，节省了第一处理器资源，同时第二处理器做这类转换比第一处理器更快、效率更高。
为了便于理解本方案，下面结合图4示例性示出的客户端与服务器交互的场景图来介绍本方案实施例提供的一种图像渲染和编码的方法的一个应用实施例。
图4所示的场景包括云手机客户端和服务器端，其中，服务器上运行有云手机服务器，该云手机服务可以是安卓Android虚拟机。该云手机服务可以包括操控模块、抓图服务模块和视频流服务模块；该云手机服务还可以包括图像显示模块和编码库。服务器中还可以包括GPU，当然该GPU也可以是独立于该服务器，但是可以与该服务器进行数据交互的图形处理器。
下面就结合图4示例性介绍本方案实施例提供的一种图像渲染和编码的方法的一个应用流程，该流程可以包括但不限于如下步骤：
①云手机客户端响应于硬件的触控操作获取该触控的信息，该触控的信息可以包括触控的位置信息，以及根据该触控匹配出的需要获取显示的图像数据的信息等等。
②然后，云手机客户端通过网络将该触控信息发送给服务器。
③服务器接收到该触控信息之后触发云手机服务的操控模块对该触控信息进行处理，得到处理后的数据。
④和⑤步骤中，服务器触发云手机服务的图像显示模块请求GPU对处理后的数据进行渲染，得到渲染后的RGBA格式的图像数据。
⑥然后、GPU继续将渲染后的RGBA格式的图像数据转换为YUV格式的帧数据。
⑦格式转换后，GPU继续将转换后的YUV格式的帧数编码为H.264或H.265格式的数据。
⑧、⑨和⑩步骤中，GPU将编码得到的H.264或H.265帧数据回传给云手机服务视频流服务模块。
Figure PCTCN2020095529-appb-000001
云手机服务视频流服务模块通过网路将H.264或H.265帧数据发送给云手机客户端。
Figure PCTCN2020095529-appb-000002
云手机客户端接收该H.264或H.265帧数据后，对其进行视频解码。
Figure PCTCN2020095529-appb-000003
云手机客户端将解码后的数据显示在显示屏中。
上述步骤④、⑤、⑥和⑦中的具体实现及有益效果可以参见图3中对应的描述，此处不再赘述。
上述主要从第一处理器与第二处理器之间交互的角度对本申请实施例提供的方案进行了介绍。可以理解的是，各个网元，例如第一处理器、第二处理器等为了实现上述功能，其包含了执行各个功能相应的硬件结构和/或软件模块。本领域技术人员应该很容易意识到，结合本文中所公开的实施例描述的各示例的单元及算法步骤，本申请能够以硬件或硬件和计算机软件的结合形式来实现。某个功能究竟以硬件还是计算机软件驱动硬件的方式来执行，取决于技术方案的特定应用和设计约束条件。专业技术人员可以对每个特定的应用来使用不同方法来实现所描述的功能，但是这种实现不应认为超出本申请的范围。
本申请实施例可以根据上述方法示例对第一处理器、第二处理器等进行功能模块的划分，例如，可以对应各个功能划分各个功能模块，也可以将两个或两个以上的功能集成在一个处理模块中。上述集成的模块既可以采用硬件的形式实现，也可以采用软件功能模块的形式实现。需要说明的是，本申请实施例中对模块的划分是示意性的，仅仅为一种逻辑功能划分，实际实现时可以有另外的划分方式。
在采用对应各个功能划分各个功能模块的情况下，图5示出了上述方法实施例中所涉及的第一处理器的一种可能的逻辑结构示意图，第一处理器500包括：发送单元501、指示单元502和调用单元503。示例性的，发送单元501用于执行如前述图3所示方法实施例中步骤301和步骤306所示的第一处理器发送信息的步骤；指示单元502用于执行前述图3所示方法实施例中步骤302、步骤303、步骤304和步骤305所示的第一处理器指示第二处理器完成图像渲染、格式转换、编码以及复制的步骤；调用单元503用于执行前述图3所示方法实施例中步骤302、步骤303和步骤304的可能的实施方式所示的第一处理器调用接口的步骤。
例如，发送单元501，用于将待渲染的数据发送给第二处理器；指示单元502，用于：指示所述第二处理器根据所述待渲染的数据渲染得到第一格式数据，所述第一格式数据包括图像的三基色颜色特征；所述第一格式数据存储在所述第二处理器的第一存储空间中；指示所述第二处理器将所述第一存储空间中的所述第一格式数据转换为第二格式数据，所述第二格式数据包括图像的亮度值和色度值；以及指示所述第二处理器将所述第二格式数据编码为第三格式数据，所述第三格式数据的数据容量小于所述第二格式数据的数据容量；发送单元501，还用于向客户端发送所述第三格式数据。
在采用对应各个功能划分各个功能模块的情况下，图6示出了上述方法实施例中所涉及的第二处理器的一种可能的逻辑结构示意图，第二处理器600包括：接收单元601、渲染单元602、转换单元603、编码单元604和发送单元605。示例性的，接收单元601用于执行前述图3所示方法实施例中步骤302所示的第二处理器接收信息的步骤；渲染单元602，用于执行前述图3所示方法实施例中步骤302所示的第二处理器渲染图像的步骤；转换单元603用于执行前述图3所示方法实施例中步骤303所示的第二处理器转换格式的步骤；编码单元604用于执行前述图3所示方法实施例中步骤304所示的第二处理器编码的步骤；发送单元605用于执行前述图3所示方法实施例中步骤305所示的第二处理器发送信息的步骤。
例如，接收单元601，用于接收第一处理器发送的待渲染的数据；渲染单元602，用于根据所述待渲染的数据渲染得到第一格式数据，所述第一格式数据包括图像的三基色颜色特征；所述第一格式数据存储在所述处理器的第一存储空间中；转换单元603，用于将所述第一存储空间中的所述第一格式数据转换为第二格式数据，所述第二格式数据包括图像的亮度值和色度值；编码单元604，用于将所述第二格式数据编码为第三格式数据，所述第三格式数据的数据容量小于所述第二格式数据的数据容量；发送单元605，用于发送所述第三格式数据。
为了便于更好地实施本发明的上述方案，本发明实施例还对应提供了一种处理器，该处理器可以是上述图3中所述的第一处理器，该处理器执行如下操作：
将待渲染的数据发送给第二处理器；指示上述第二处理器根据上述待渲染的数据渲染得到第一格式数据，上述第一格式数据包括图像的三基色颜色特征；上述第一格式数据存储在上述第二处理器的第一存储空间中；指示上述第二处理器将上述第一格式数据转换为第二格式数据，上述第二格式数据包括图像的亮度值和色度值；指示上述第二处理器将上述第二格式数据编码为第三格式数据，上述第三格式数据的数据容量小于上述第二格式数据的数据容量；向客户端发送上述第三格式数据。
在其中一种可能的实施方式中，在上述处理器指示上述第二处理器根据上述待渲染的数据渲染得到第一格式数据之后，上述处理器指示上述第二处理器将上述第一格式数据转换为第二格式数据之前，还包括：上述处理器调用第一接口获取上述第一格式数据在上述第一存储空间中的存储地址；上述处理器调用上述第一接口将上述存储地址发送给第二接口；上述处理器调用上述第二接口指示上述第二处理器将上述存储地址标记到待编码区中，上述待编码区包括用于存放待编码的数据和/或待编码数据的存储地址的存储空间。
在其中一种可能的实施方式中，上述第一接口为图形库中配置的用于获取上述存储地址的应用程序接口。
在其中一种可能的实施方式中，上述第二接口为视频加速库中配置的用于接收上述第一接口发送过来的上述存储地址以及指示上述第二处理器将上述存储地址标记到上述待编码区中的应用程序接口。
在其中一种可能的实施方式中，上述处理器指示上述第二处理器将上述第一格式数 据转换为第二格式数据，包括：上述处理器调用第三接口向上述第二处理器申请第二存储空间，上述第二存储空间为用于存放上述第二格式数据的地址空间；上述第三接口为视频加速库中的应用程序接口；上述处理器调用上述第三接口指示上述第二处理器将上述第一格式数据转换为第二格式数据。
在其中一种可能的实施方式中，上述处理器指示上述第二处理器将上述第二格式数据编码为第三格式数据，包括：上述处理器调用第四接口向上述第二处理器申请第三存储空间，上述第三存储空间为用于存放上述第三格式数据的地址空间；上述第四接口为视频加速库中的应用程序接口；上述处理器调用上述第四接口指示上述第二处理器将上述第二格式数据编码为第三格式数据。
本发明实施例还对应提供了一种处理器，该处理器可以是上述图3中所述的第二处理器，该处理器执行如下操作：
接收待渲染的数据；根据上述待渲染的数据渲染得到第一格式数据，上述第一格式数据包括图像的三基色颜色特征；上述第一格式数据存储在上述处理器的第一存储空间中；将上述第一格式数据转换为第二格式数据，上述第二格式数据包括图像的亮度值和色度值；将上述第二格式数据编码为第三格式数据，上述第三格式数据的数据容量小于上述第二格式数据的数据容量；发送上述第三格式数据。
本发明实施例还对应提供了一种服务器，如图7所示。图7是本发明实施例提供的一种服务器700，该服务器700包括第一处理器701(可以是图3中所述的第一处理器)、存储器702、通信接口703和第二处理器705(可以是图3中所述的第二处理器)，所述第一处理器701、存储器702、通信接口703和第二处理器705通过总线704相互连接。
存储器702包括但不限于是随机存储记忆体(random access memory，RAM)、只读存储器(read-only memory，ROM)、可擦除可编程只读存储器(erasable programmable read only memory，EPROM)、或便携式只读存储器(compact disc read-only memory，CD-ROM)，该存储器702用于相关指令及数据的存储。通信接口703用于接收和发送数据。
该服务器700中的第一处理器701用于读取所述存储器702中存储的计算机程序，以用于执行前述图3及其可能的实施方式中的方法实施例。
本发明实施例还提供一种计算机可读存储介质，该计算机可读存储介质存储有计算机程序，所述计算机程序被处理器执行以实现前述图3及其可能的实施方式所述方法实施例。
本发明实施例还提供一种计算机程序产品，当该计算机程序产品被计算机读取并执行时，前述图3及其可能的实施方式所述方法实施例得以实现。
本发明实施例还提供一种计算机程序，当该计算机程序在计算机上执行时，将会使所述计算机实现前述图3及其可能的实施方式所述方法实施例。
综上所述，相比于现有技术中需要将在第二处理器中渲染好的第一格式的数据复制到第一处理器的内存中，在第一处理器转换为第二格式的数据之后，再复制到第二处理器内存中进行编码，本申请实施例可以在第二处理器中完成渲染、格式转换和编码的过程，不需要额外两次的数据传输，因此通过本申请实施例能够节省第一处理器和第二处 理器的资源、提高渲染和编码的效率。
综上，以上仅为本发明的较佳实施例而已，并非用于限定本发明的保护范围。凡在本发明的精神和原则之内，所作的任何修改、等同替换、改进等，均应包含在本发明的保护范围之内。

Claims (15)

  1. 一种图像渲染和编码方法，其特征在于，包括：
    第一处理器将待渲染的数据发送给第二处理器；
    所述第一处理器指示所述第二处理器根据所述待渲染的数据渲染得到第一格式数据，所述第一格式数据包括图像的三基色颜色特征；所述第一格式数据存储在所述第二处理器的第一存储空间中；
    所述第一处理器指示所述第二处理器将所述第一格式数据转换为第二格式数据，所述第二格式数据包括图像的亮度值和色度值；
    所述第一处理器指示所述第二处理器将所述第二格式数据编码为第三格式数据，所述第三格式数据的数据容量小于所述第二格式数据的数据容量；
    所述第一处理器向客户端发送所述第三格式数据。
  2. 根据权利要求1所述的方法，其特征在于，在所述第一处理器指示所述第二处理器根据所述待渲染的数据渲染得到第一格式数据之后，所述第一处理器指示所述第二处理器将所述第一格式数据转换为第二格式数据之前，还包括：
    所述第一处理器调用第一接口获取所述第一格式数据在所述第一存储空间中的存储地址；
    所述第一处理器调用所述第一接口将所述存储地址发送给第二接口；
    所述第一处理器调用所述第二接口指示所述第二处理器将所述存储地址标记到待编码区中，所述待编码区包括用于存放待编码的数据和/或待编码数据的存储地址的存储空间。
  3. 根据权利要求2所述的方法，其特征在于，所述第一接口为图形库中配置的用于获取所述存储地址的应用程序接口。
  4. 根据权利要求2或3所述的方法，其特征在于，所述第二接口为视频加速库中配置的用于接收所述第一接口发送过来的所述存储地址以及指示所述第二处理器将所述存储地址标记到所述待编码区中的应用程序接口。
  5. 根据权利要求1至4任一项所述的方法，其特征在于，所述第一处理器指示所述第二处理器将所述第一格式数据转换为第二格式数据，包括：
    所述第一处理器调用第三接口向所述第二处理器申请第二存储空间，所述第二存储空间为用于存放所述第二格式数据的地址空间；所述第三接口为视频加速库中的应用程序接口；
    所述第一处理器调用所述第三接口指示所述第二处理器将所述第一格式数据转换为第二格式数据。
  6. 根据权利要求1至5任一项所述的方法，其特征在于，所述第一处理器指示所述 第二处理器将所述第二格式数据编码为第三格式数据，包括：
    所述第一处理器调用第四接口向所述第二处理器申请第三存储空间，所述第三存储空间为用于存放所述第三格式数据的地址空间；所述第四接口为视频加速库中的应用程序接口；
    所述第一处理器调用所述第四接口指示所述第二处理器将所述第二格式数据编码为第三格式数据。
  7. 一种图像渲染和编码方法，其特征在于，所述方法应用于处理器，所述方法包括：
    接收待渲染的数据；
    根据所述待渲染的数据渲染得到第一格式数据，所述第一格式数据包括图像的三基色颜色特征；所述第一格式数据存储在所述处理器的第一存储空间中；
    将所述第一格式数据转换为第二格式数据，所述第二格式数据包括图像的亮度值和色度值；
    将所述第二格式数据编码为第三格式数据，所述第三格式数据的数据容量小于所述第二格式数据的数据容量；
    发送所述第三格式数据。
  8. 一种处理器，其特征在于，所述处理器包括：
    发送单元，用于将待渲染的数据发送给第二处理器；
    指示单元，用于：指示所述第二处理器根据所述待渲染的数据渲染得到第一格式数据，所述第一格式数据包括图像的三基色颜色特征；所述第一格式数据存储在所述第二处理器的第一存储空间中；
    指示所述第二处理器将所述第一存储空间中的所述第一格式数据转换为第二格式数据，所述第二格式数据包括图像的亮度值和色度值；以及
    指示所述第二处理器将所述第二格式数据编码为第三格式数据，所述第三格式数据的数据容量小于所述第二格式数据的数据容量；
    所述发送单元，还用于向客户端发送所述第三格式数据。
  9. 根据权利要求8所述的处理器，其特征在于，所述处理器还包括；
    调用单元，用于：在所述指示单元指示所述第二处理器根据所述待渲染的数据渲染得到第一格式数据之后，指示所述第二处理器将所述第一格式数据转换为第二格式数据之前，调用第一接口获取所述第一格式数据在所述第一存储空间中的存储地址；
    调用所述第一接口将所述存储地址发送给第二接口；以及
    调用所述第二接口指示所述第二处理器将所述存储地址标记到待编码区中，所述待编码区包括用于存放待编码的数据和/或待编码数据的存储地址的存储空间。
  10. 根据权利要求9所述的处理器，其特征在于，所述第一接口为图形库中配置的用于获取所述存储地址的应用程序接口。
  11. 根据权利要求9或10所述的处理器，其特征在于，所述第二接口为视频加速库中配置的用于接收所述第一接口发送过来的所述存储地址以及指示所述第二处理器将所述存储地址标记到所述待编码区中的应用程序接口。
  12. 根据权利要求8至11任一项所述的处理器，其特征在于，所述指示单元具体用于：
    调用第三接口向所述第二处理器申请第二存储空间，所述第二存储空间为用于存放所述第二格式数据的地址空间；所述第三接口为视频加速库中的应用程序接口；
    调用所述第三接口指示所述第二处理器将所述第一格式数据转换为第二格式数据。
  13. 根据权利要求8至12任一项所述的处理器，其特征在于，所述指示单元具体还用于：
    调用第四接口向所述第二处理器申请第三存储空间，所述第三存储空间为用于存放所述第三格式数据的地址空间；所述第四接口为视频加速库中的应用程序接口；
    调用所述第四接口指示所述第二处理器将所述第二格式数据编码为第三格式数据。
  14. 一种处理器，其特征在于，所述处理器包括：
    接收单元，用于接收第一处理器发送的待渲染的数据；
    渲染单元，用于根据所述待渲染的数据渲染得到第一格式数据，所述第一格式数据包括图像的三基色颜色特征；所述第一格式数据存储在所述处理器的第一存储空间中；
    转换单元，用于将所述第一存储空间中的所述第一格式数据转换为第二格式数据，所述第二格式数据包括图像的亮度值和色度值；
    编码单元，用于将所述第二格式数据编码为第三格式数据，所述第三格式数据的数据容量小于所述第二格式数据的数据容量；
    发送单元，用于发送所述第三格式数据。
  15. 一种服务器，其特征在于，包括第一处理器、第二处理器和存储器，所述存储器存储有计算机程序，所述第一处理器用于调用所述计算机程序来执行如权利要求1-7任一项所述的方法。
PCT/CN2020/095529 2019-09-26 2020-06-11 图像渲染和编码方法及相关装置 WO2021057097A1 (zh)

Priority Applications (2)

Application Number Priority Date Filing Date Title
EP20868794.7A EP3972253A4 (en) 2019-09-26 2020-06-11 METHOD AND APPARATUS FOR IMAGE REPRESENTATION AND CODING
US17/572,067 US11882297B2 (en) 2019-09-26 2022-01-10 Image rendering and coding method and related apparatus

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201910919459.4A CN110636305B (zh) 2019-09-26 2019-09-26 图像渲染和编码方法及相关装置
CN201910919459.4 2019-09-26

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US17/572,067 Continuation US11882297B2 (en) 2019-09-26 2022-01-10 Image rendering and coding method and related apparatus

Publications (1)

Publication Number Publication Date
WO2021057097A1 true WO2021057097A1 (zh) 2021-04-01

Family

ID=68974380

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2020/095529 WO2021057097A1 (zh) 2019-09-26 2020-06-11 图像渲染和编码方法及相关装置

Country Status (4)

Country Link
US (1) US11882297B2 (zh)
EP (1) EP3972253A4 (zh)
CN (1) CN110636305B (zh)
WO (1) WO2021057097A1 (zh)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114786040A (zh) * 2022-06-15 2022-07-22 阿里巴巴（中国）有限公司 数据通信方法、系统、电子设备和存储介质

Families Citing this family (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110636305B (zh) * 2019-09-26 2023-04-28 华为技术有限公司 图像渲染和编码方法及相关装置
CN114078078A (zh) * 2020-08-21 2022-02-22 北京小米移动软件有限公司 一种图像格式转换方法、装置及存储介质
WO2022048255A1 (zh) * 2020-09-07 2022-03-10 华为云计算技术有限公司 数据处理的方法和系统、云终端、服务器和计算设备
CN113038269B (zh) * 2021-03-11 2023-01-24 昆山龙腾光电股份有限公司 一种图像数据处理系统及方法
US20230185641A1 (en) * 2021-12-10 2023-06-15 Nvidia Corporation Application programming interface to store portions of an image

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20080143739A1 (en) * 2006-12-13 2008-06-19 Harris Jerry G Method and System for Dynamic, Luminance-Based Color Contrasting in a Region of Interest in a Graphic Image
CN105163127A (zh) * 2015-09-07 2015-12-16 浙江宇视科技有限公司 视频分析方法及装置
CN105678680A (zh) * 2015-12-30 2016-06-15 魅族科技（中国）有限公司 一种图像处理的方法和装置
CN107729095A (zh) * 2017-09-13 2018-02-23 深信服科技股份有限公司 图像处理方法、虚拟化平台及计算机可读存储介质
US10168879B1 (en) * 2017-05-12 2019-01-01 Snap Inc. Interactive image recoloring
CN110636305A (zh) * 2019-09-26 2019-12-31 华为技术有限公司 图像渲染和编码方法及相关装置

Family Cites Families (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20160132284A1 (en) * 2014-11-07 2016-05-12 Qualcomm Incorporated Systems and methods for performing display mirroring
US10503458B2 (en) * 2016-07-28 2019-12-10 Intelligent Waves Llc System, method and computer program product for generating remote views in a virtual mobile device platform using efficient macroblock comparison during display encoding, including efficient detection of unchanged macroblocks

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20080143739A1 (en) * 2006-12-13 2008-06-19 Harris Jerry G Method and System for Dynamic, Luminance-Based Color Contrasting in a Region of Interest in a Graphic Image
CN105163127A (zh) * 2015-09-07 2015-12-16 浙江宇视科技有限公司 视频分析方法及装置
CN105678680A (zh) * 2015-12-30 2016-06-15 魅族科技（中国）有限公司 一种图像处理的方法和装置
US10168879B1 (en) * 2017-05-12 2019-01-01 Snap Inc. Interactive image recoloring
CN107729095A (zh) * 2017-09-13 2018-02-23 深信服科技股份有限公司 图像处理方法、虚拟化平台及计算机可读存储介质
CN110636305A (zh) * 2019-09-26 2019-12-31 华为技术有限公司 图像渲染和编码方法及相关装置

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
See also references of EP3972253A4 *

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114786040A (zh) * 2022-06-15 2022-07-22 阿里巴巴（中国）有限公司 数据通信方法、系统、电子设备和存储介质

Also Published As

Publication number Publication date
US20220132147A1 (en) 2022-04-28
CN110636305A (zh) 2019-12-31
CN110636305B (zh) 2023-04-28
EP3972253A4 (en) 2022-07-20
US11882297B2 (en) 2024-01-23
EP3972253A1 (en) 2022-03-23

Similar Documents

Publication Publication Date Title
WO2021057097A1 (zh) 图像渲染和编码方法及相关装置
CN1856819B (zh) 通过分布式应用程序的图形数据的网络传输的系统和方法
US9619916B2 (en) Method for transmitting digital scene description data and transmitter and receiver scene processing device
KR101962990B1 (ko) 복잡도가 낮은 원격 프리젠테이션 세션 인코더
CN108881916A (zh) 远程桌面的视频优化处理方法及装置
JP7359521B2 (ja) 画像処理方法および装置
WO2022257750A1 (zh) 图像处理方法、装置、电子设备、程序及可读存储介质
US20120218292A1 (en) System and method for multistage optimized jpeg output
CN114567784B (zh) 一种用于飞腾显卡的vpu视频解码输出方法及系统
WO2023011033A1 (zh) 图像处理方法、装置、计算机设备及存储介质
WO2021147463A1 (zh) 视频处理方法、装置及电子设备
CN111935483B (zh) 一种视频图像无损编解码方法及系统
CN111225214B (zh) 视频处理方法、装置及电子设备
US20240031525A1 (en) Efficient electro-optical transfer function (eotf) curve for standard dynamic range (sdr) content
CN105025359A (zh) 一种基于spice协议的远程视频显示方法及系统
KR20240096800A (ko) 렌더링 파이프라인들의 크로미넌스 최적화들
WO2021169817A1 (zh) 视频处理方法及电子设备
CN113450293A (zh) 视频信息处理方法、装置、系统、电子设备及存储介质
WO2023185856A1 (zh) 数据传输方法、装置、电子设备及可读存储介质
WO2024120031A1 (zh) 处理视频数据的方法、装置、计算机设备和存储介质
WO2023029689A1 (zh) 多媒体数据的共享方法、媒体共享服务器、终端、电子设备和计算机可读存储介质
CN116489132A (zh) 虚拟桌面数据传输方法、服务器、客户端和存储介质
CN118101853A (zh) 图像处理方法、装置和系统
TW202215373A (zh) 圖像空間函數傳輸

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 20868794

Country of ref document: EP

Kind code of ref document: A1

ENP Entry into the national phase

Ref document number: 2020868794

Country of ref document: EP

Effective date: 20211214

NENP Non-entry into the national phase

Ref country code: DE