WO2020006739A1

WO2020006739A1 - Image processing method and apparatus

Info

Publication number: WO2020006739A1
Application number: PCT/CN2018/094716
Authority: WO
Inventors: 郑萧桢; 封旭阳; 张李亮; 赵丛
Original assignee: 深圳市大疆创新科技有限公司
Priority date: 2018-07-05
Filing date: 2018-07-05
Publication date: 2020-01-09
Also published as: CN110720224A; CN110720224B

Abstract

An image processing method and apparatus. The method comprises: obtaining code stream data of a current image, the code stream data comprising identification information, the identification information being used for identifying at least one target object in the current image, the identification information comprising image area information and pixel information, the image area information comprising the position and the size of an image area where the target object is located, and the pixel information comprising an attribute of at least one pixel in the image area; decoding the code stream data to obtain the current image and the identification information; and performing pixel-level processing on the current image according to the identification information. According to the image processing method and apparatus, the position and the size of an image area where a target object is located are indicated by means of image area information, and attributes of multiple pixels in the image area are indicated by means of pixel information, so that the target object is identified with a finer granularity, and thus a decoding device can perform pixel-level processing on the target object more efficiently and accurately.

Description

Image processing method and device

Copyright statement

The content disclosed in this patent document contains material which is subject to copyright protection. The copyright is owned by the copyright owner. The copyright owner has no objection to the facsimile reproduction by anyone of the patent document or the patent disclosure as it appears in the official records and archives of the Patent and Trademark Office.

Technical field

The present application relates to the field of image processing, and in particular, to an image processing method and device.

Background technique

In scenes such as video surveillance, human-computer interaction, and security inspections, the objects of interest (including people, animals, plants, public facilities, vehicles or landscapes, and scenery) usually need to be identified so that the decoder or the observer can update To track the change of the object in the video stream, so as to better assist the observer to observe or interact with the object. This type of method in image processing can generally be referred to as object tracking technology.

Existing object tracking technologies usually use image processing, computer vision, and computer analysis and understanding technologies to identify the content of the video stream and identify the objects that require attention. In an existing solution, the position and size of an object of interest in each frame of the image are identified by a rectangular area at the encoding end or the decoding end. The decoder performs additional operations based on the rectangular region, which results in poor processing results and low processing efficiency.

Summary of the invention

The present application provides an image processing method and apparatus, so that a decoding device can perform pixel-level processing on a target object more efficiently and accurately.

According to a first aspect, an image processing method is provided, including: obtaining code stream data of a current image, where the code stream data includes identification information, and the identification information is used to identify at least one target object in the current image. The identification information includes image area information and pixel information, the image area information includes a position and a size of an image area where the target object is located, and the pixel information includes an attribute of at least one pixel in the image area; Decode the stream data to obtain the current image and the identification information; and perform pixel-level processing on the current image according to the identification information.

According to a second aspect, an image processing apparatus is provided, including: at least one memory for storing computer-executable instructions; at least one processor, individually or collectively, for: accessing the at least one memory and executing the computer-executable Execute an instruction to implement the following operations: obtaining code stream data of the current image, the code stream data including identification information, the identification information used to identify at least one target object in the current image, the identification information including an image Area information and pixel information, the image area information including a position and a size of an image area where the target object is located, the pixel information including an attribute of at least one pixel in the image area, and decoding the code stream data To obtain the current image and the identification information; and perform pixel-level processing on the current image according to the identification information.

According to a third aspect, a computer-readable storage medium is provided, on which instructions are stored, and when the instructions run on the computer, the computer is caused to execute the method of the first aspect.

The image processing method and device of the present application indicate the position and size of the image area where the target object is located through the image area information, and indicate the attributes of multiple pixels in the image area through the pixel information, thereby identifying the target object with a finer granularity, This enables the decoding device to perform pixel-level processing on the target object more efficiently and accurately.

The image processing method and device of the present application enable recognition of a target object to be performed on an encoding end, and a decoding device only needs to perform subsequent image processing. Therefore, on the one hand, the image processing method of the present application can be implemented on platforms such as mobile phones and tablets; on the other hand, the computing resources of the decoding device can be used for more complex image processing, so that the decoding device can present a higher quality and more delicate Image.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic flowchart of an encoding method according to an embodiment of the present application.

FIG. 2 is a schematic diagram of a target object in an image according to an embodiment of the present application.

FIG. 3 is a schematic flowchart of a decoding method according to an embodiment provided in the present application.

FIG. 4 is a schematic flowchart of an encoding device according to an embodiment of the present application.

FIG. 5 is a schematic flowchart of an encoding device according to another embodiment provided in the present application.

FIG. 6 is a schematic flowchart of a decoding device according to an embodiment provided in this application.

FIG. 7 is a schematic flowchart of a decoding device according to another embodiment provided in this application.

FIG. 8 is a schematic flowchart of an image processing method according to an embodiment of the present application.

9A and 9B are schematic diagrams of two images obtained by fusion according to an embodiment of the present application.

10A and 10B are schematic diagrams of adding an indicator halo on a target object according to an embodiment of the present application.

FIG. 11 is a schematic diagram of changing the brightness of a target object according to an embodiment of the present application.

FIG. 12A is an original image of the current image; FIG. 12B is an object category segmentation image corresponding to the current image.

FIG. 13 is a schematic diagram of images identified by different colors at different parts of an embodiment of the present application.

FIG. 14A is an original image of the current image; FIG. 14B is a reflection intensity segmented image corresponding to the current image.

FIG. 15A is an original image of the current image; FIG. 15B is a depth map corresponding to the current image.

FIG. 16 is a schematic block diagram of an image processing apparatus according to an embodiment of the present application.

FIG. 17 is a schematic block diagram of an image processing apparatus according to another embodiment of the present application.

detailed description

The technical solutions in the embodiments of the present application will be described below with reference to the drawings.

Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terms used herein in the specification of the present application are only for the purpose of describing specific embodiments, and are not intended to limit the present application.

First, related technologies and concepts involved in the embodiments of the present application are introduced.

The target object can refer to the objects in the image that need to be focused on, to be identified, identified or to be observed, and can include people, animals, plants, public facilities, vehicles or landscapes, scenery, etc., and can also include other types of objects ; Can also include specific parts of people, animals, plants, public facilities, vehicle landscapes, scenery or other types of objects.

The image area may refer to a regular or irregular shape area where the target object is located. Generally speaking, the position and size of the image area should be such that all parts of the target object fall into the image area, or at least 80% of the area on the target object should fall into the image area. The image area can be roughly delimited, so that the decoder can determine the position and size of the target object faster.

The sub-image area may be a piece of area in the image area where pixels have the same attribute.

For a system from the encoding end to the decoding end, an existing method of object tracking technology is to encode the video content at the encoding end; analyze the video content at the decoding end to find the objects that need to be focused on, and respond to the need The objects of interest are identified, that is, the identification is done at the decoding end.

The problem with completing the identification at the decoding end is that video encoding is usually a lossy process, and the information of the video content will suffer loss after encoding. Compared with the video content on the encoding side, the video content decoded by the decoder has a certain degree of degradation in quality and information. The decoder analyzes a damaged video content and extracts the objects that need attention. The effect is usually not satisfactory. In addition, analyzing video content and extracting objects at the decoding end will consume a lot of computing resources at the decoding end. However, decoders are widely used in mobile devices such as mobile phones, which are more sensitive to power consumption. Therefore, the calculation of video content at the decoding end will affect the user experience to a certain extent.

In this application, the function of analyzing video content is transferred from the decoding end to the encoding end for execution. These technologies identify the extracted object at the encoding end and write the identification information into the video file to ensure that the decoding end can identify the object extracted from the encoding end by parsing the identification information. The advantages of this approach are: 1. Analysis of the original uncompressed loss video content at the encoding end can more efficiently and accurately extract the objects that need attention. 2. Since the encoding device usually has stronger computing capabilities, and the encoding device usually needs to analyze the video content in order to perform some additional operations, the calculation and analysis of the original decoding end will be transferred to the encoding end, without bringing Comes with a bad user experience. These additional operations may be, for example, obstacle avoidance operations performed after the captured video content is analyzed on the drone system.

In some implementations, the encoding end may use a common video coding standard, for example, H.264 / advanced video coding (AVC) standard, H.265 / high efficiency video coding (HEVC) Standards, source coding standards (audio video coding standard (AVS) 1-P2, AVS2-P2, VP9 standards, Open Media Alliance Video (AV) 1 standard, universal video coding (VVC) standard pair The video content is encoded to obtain a video file. then,

FIG. 1 is a schematic flowchart of an encoding method 100 according to an embodiment of the present application. The encoding method 100 is performed by an encoding device. As shown in FIG. 1, the encoding method 100 includes: S110, encoding processing of a current image to generate code stream data, the code stream data includes identification information, and the identification information is used to identify at least one target object in the current image The identification information includes image area information and pixel information, the image area information includes a position and a size of an image area where the target object is located, and the pixel information includes an attribute of at least one pixel in the image area.

In the encoding method provided in the embodiment of the present application, the position and size of an image area where a target object is located are indicated by image area information, and attributes of multiple pixels in the image area are indicated by pixel information, thereby identifying the target object with finer granularity, It is beneficial for the decoding end to perform operations on the target object more efficiently and accurately.

In some possible implementation manners, before S110 performs encoding processing on the current image and generates code stream data, the encoding method 100 may further include: performing image recognition on the current image, determining the target object, and obtaining the identifier of the target object. information. Among them, image recognition can be based on technologies such as image processing, computer vision, and computer analysis and understanding. Of course, the identification information in the embodiment of the present application may also be obtained by other methods, for example, by receiving external input. The form and content of the obtained identification information can be various, which will be described in detail below.

In some possible implementation manners, the identification information may be located in auxiliary enhancement information or extended data of the current image. For example, the auxiliary enhancement information may be Supplemental Enhancement Information (SEI), and the extension data may be ED (Extension Data). SEI and ED can generally be considered as part of the stream data. When the decoding device receives the SEI and / or ED, it may decode according to the SEI and / or ED, or discard the SEI and / or ED. Whether to decode the identification information may not affect the decoding of the content of the current image. This will also be described in detail below.

Optionally, in some embodiments of the present application, the image area may be a rectangular area. In some implementations, the image area is the smallest rectangular area or a smaller rectangular area that can frame the target object. There are various ways to indicate the position and size of the rectangular area with the image area information. For example, the image region information may include coordinates of any corner of the rectangular region (for example, coordinates of the upper left corner), height information of the rectangular region, and width information of the rectangular region. As another example, the image area information may include the coordinates of the center point of the rectangular area, the height information of the rectangular area, and the width information of the rectangular area. The height information of the rectangular area may be the full height or half height of the rectangular area. The width information of the rectangular area may be the full width or half width of the rectangular area, which is not limited herein. As another example, the image region information may include coordinates of the upper left corner of the rectangular region and coordinates of the lower right corner of the rectangular region. Of course, the image region information may include coordinates of the upper-right corner of the rectangular region and coordinates of the lower-left corner of the rectangular region, and so on. The specific content of the image area information is not limited in the embodiments of the present application.

In addition, in other embodiments of the present application, the image region may have other shapes, such as a circle, a polygon, or a curved edge, and so on. When the image area is circular, the image area information may include the coordinates of the center of the circle (that is, the coordinates of the center point) and the radius information. When the image area is a polygon, such as a regular hexagon, the image area information may include the coordinates of the center point and the distance information between the center point and the vertex of the regular hexagon. Those skilled in the art can understand that the image area and the image area information can also be obtained by other alternative forms or solutions, which are not listed here one by one.

It should be understood that, in the embodiments of the present application, the image region may include multiple sub-image regions. The sub-image area may be a piece of area in the image area where pixels have the same attribute. For example, one sub-image region may be a region corresponding to a target object, and another sub-image region may be a region corresponding to a background. As another example, a sub-image area may be a region corresponding to one part of the target object, another sub-image region may be a region corresponding to another part of the target object, and another sub-image region may be a region corresponding to the background.

In the embodiments of the present application, the attributes may be measured in pixels, that is, each pixel corresponds to its own attribute. Correspondingly, the pixel information includes information about the attributes of each pixel. The attributes may also be measured in pixels. Ground, the pixel information includes information of attributes of at least one pixel block, and the pixel block includes at least two pixels.

A pixel block may be an area with a finer or smaller granularity than an image area. The attribute of a pixel block means that the attributes of all pixel points in the pixel block are attributes of the pixel block. A pixel block may be a regular shaped block, such as a square or rectangular block. A pixel block can also be an irregularly shaped block. A pixel block may include multiple pixels (e.g., 2, 4, 9, or 16 pixels). When the attributes are measured in pixel blocks, the sizes of multiple pixel blocks may be the same or different. The current image can be down-sampled first to obtain the attribute information corresponding to the pixel block.

Compared with the case where the attributes are measured in pixels, the attributes are measured in pixel blocks to reduce the amount of data stored or transmitted by the encoding device. Those skilled in the art can understand that the pixel information may also be obtained from other alternative forms or solutions, which are not listed here one by one.

Optionally, in some embodiments of the present application, the pixel information may include a value assigned to at least one pixel in the image area; wherein pixels in different sub-image areas are assigned the same or different values. It should be understood that the pixel values of different sub-image regions in the same image region may be the same or different. For example, the two sub-image regions in the graphics region that are not connected are all regions other than the target object, then the The values assigned by the pixels may be the same or different. The pixel values of the sub-image areas in different image areas can be the same or different. For example, the values assigned by the sub-image areas belonging to the target object in different image areas can be the same or different. The sub-image areas outside the target object in different image areas The assigned values may be the same or different. Of course, the pixel information may also be represented by a non-numeric indicator, which is not limited in the embodiment of the present application.

Optionally, in some embodiments of the present application, the attribute of the at least one pixel may include whether the at least one pixel belongs to the target object. For example, in the pixel information, different values are assigned to at least one pixel to indicate whether the at least one pixel belongs to the target object.

In a possible implementation manner, among the at least one pixel, the first portion of pixels is assigned a first value to indicate that the first portion of pixels does not belong to the target object. That is, the pixel information includes values of pixels that do not belong to the target object. For example, the image region includes one (or more) sub-image regions as target objects; the image region also includes several sub-image regions as backgrounds that do not belong to the target object. The pixel information may include only attributes of pixels that do not belong to the target object, or the pixel information may include only values of pixels that do not belong to the target object. In other words, the pixel information may include only attributes or numerical values of pixels belonging to the several sub-image regions of the background.

In another possible implementation manner, among the at least one pixel, the second part of the pixels is assigned a second value to indicate that the second part of the pixels belong to the target object. That is, the pixel information includes numerical values of pixels belonging to the target object. For example, the image region includes one (or more) sub-image regions as target objects; the image region also includes several sub-image regions as backgrounds that do not belong to the target object. The pixel information may include only attributes of pixels belonging to the target object, or the pixel information may include only values of pixels belonging to the target object. In other words, the pixel information may include only attributes or numerical values of pixels belonging to one (or more) sub-image regions of the target object.

In another possible implementation manner, among the at least one pixel, the first partial pixel is assigned a first value to indicate that the first partial pixel does not belong to the target object; the second partial pixel is assigned a second value to indicate the second partial pixel. Belongs to the target object. That is, the pixel information includes numerical values of all pixels. For example, the image region includes one (or more) sub-image regions as target objects; the image region also includes several sub-image regions as backgrounds that do not belong to the target object. The pixel information may include both the attributes of pixels belonging to the target object and the attributes of pixels belonging to the background; or in other words, the pixel information may include both the values of pixels belonging to the target object and the values of pixels belonging to the background. In other words, the pixel information may include both attributes or values of pixels belonging to one (or more) sub-image areas of the target object; and attributes or values of pixels belonging to several sub-image areas of the background.

In an example in which the attribute is measured in pixels, the pixel information may be represented by a mask. The template value can be identified by the

binary values

0 and 1. The template value of the pixels belonging to the target object in the pixel information is 1; the template value of the pixels belonging to the background is 0. Take the image area of the target object i as a rectangular area as an example; the image area information of the target object i includes the coordinates of the upper left corner of the rectangular area, the height information of the rectangular area, and the width information of the rectangular area; Taking a template as an example, the specific content of the identification information of the target object i can be as follows. Those skilled in the art can understand that this content is only schematic and can be obtained from other alternative forms or solutions, which are not listed here one by one.

Among them, ar_object_top [i], ar_object_left [i], ar_object_width [i], and ar_object_height [i] represent the position and size of the target object i, ar_object_top [i] and ar_object_left [i] represent the position of the upper left corner of the target object i; ar_object_width [i] and ar_object_height [i] represent the width and height of the target object i. mask [m] [n] represents the template value corresponding to pixels with coordinates offset m and n in the vertical and horizontal directions relative to the upper left corner of the rectangular area. When the pixel belongs to the target object, the value of mask [m] [n] is 1; otherwise, when the pixel belongs to the background, the value of mask [m] [n] is 0.

In addition, for the identification of the mask, a point-by-point identification method may be adopted, or the target frame class indicated by the target object identified by ar_object_top [i], ar_object_left [i], ar_object_width [i], and ar_object_height [i] may be used. The starting position of each line and the length of the target object in the line are identified. The specific method is as follows:

Among them, mask_pos [i] [m] represents the starting position of the i-th object in the m-th row in the target frame, and mask_len [i] [m] represents the length of the i-th object in the m-th row in the target frame. .

FIG. 2 is a schematic diagram of a target object in an image 200 according to an embodiment of the present application. As shown in FIG. 2, the image 200 includes a target object 1 and a target object 2. The image region 1 corresponding to the target object 1 is a rectangular region; the image region 2 corresponding to the target object 2 is also a rectangular region. In the image region 1, pixels with a value of 1 belong to the target object 1, and pixels with a value of 0 do not belong to the target object 1. In the image region 2, pixels with a value of 1 belong to the target object 2, and pixels with a value of 0 do not belong to the target object 2.

Optionally, in some embodiments of the present application, the attribute of the at least one pixel includes a part of the target object to which the at least one pixel belongs. For example, in the pixel information, different pixels are assigned different values to indicate that different pixels belong to different parts of the target object. Of course, there may also be some pixels used to indicate that they do not belong to the target object but belong to the background.

In a specific example, the target object is a person; a first part of the at least one pixel is assigned a third value to indicate that the first part of the pixel belongs to the head of the target object; and / or, the at least one pixel The second portion of pixels is given a fourth value to indicate that the second portion of pixels belong to the hand of the target object. The at least one pixel may further include a third partial pixel, which is used to indicate that the third partial pixel does not belong to the target object but belongs to the background. For example, the third portion of pixels is assigned 0 to indicate that the third portion of pixels does not belong to the target object, but belongs to the background; the first portion of pixels is assigned 1 to indicate that the first portion of pixels belongs to the head of the target object; The two partial pixels are assigned 2 to indicate that the second partial pixels belong to the hand of the target object.

In another specific example, the target object is a car; a first part of the at least one pixel is assigned a fifth value to indicate that the first part of the pixel belongs to the head of the target object; and / or, the The second part of the pixels is assigned a sixth value to indicate that the second part of the pixels belong to the rear of the target object. The at least one pixel may further include a third partial pixel, which is used to indicate that the third partial pixel does not belong to the target object but belongs to the background. For example, the third portion of pixels is assigned 0 to indicate that the third portion of pixels does not belong to the target object, but belongs to the background; the first portion of pixels is assigned 1 to indicate that the first portion of pixels belongs to the head of the target object; the second Part of the pixel is assigned 2 to indicate that the second part of the pixel belongs to the rear of the target object.

Optionally, in some embodiments of the present application, the attribute of the at least one pixel includes a description feature corresponding to the at least one pixel. For example, the descriptive feature may be point cloud data. In a specific example, the description feature corresponding to at least one pixel may include at least one of the following: the reflection intensity of the point cloud corresponding to the at least one pixel, the infrared intensity corresponding to the at least one pixel, and the Describe the depth value corresponding to at least one pixel. Among them, depth is a measure of distance, such as the distance to the lens.

The above specifically describes the identification information used by the encoding device to identify the target object. The following will give a specific identification scheme that can effectively identify the target object, can effectively improve the efficiency of the identification, and reduce data storage and transmission. The core idea of this specific identification scheme is to identify one or more target objects that have changed relative to the encoded image by comparing the current image with the encoded image. Among them, the identification object in the current image and the identification object in the encoded image can be compared one by one.

Optionally, in some embodiments of the present application, the target object may be at least one of the following: an identification object added to the current image relative to the encoded image; and an identification that changes the position of the current image relative to the encoded image. An object; an identification object whose size of the current image changes relative to the encoded image; an identification object whose pixel information in the image area of the current image changes relative to the encoded image.

Alternatively, from the perspective of performing steps, the encoding method 100 may further include at least one of the following steps: determining the current image relative to the encoded image, and determining the newly added object to be identified as the target object; comparing the current image to the encoded image, the position and The object to be identified whose size is changed is determined as the target object; the object to be identified whose pixel information in the image area is changed relative to the encoded image is determined as the target object.

From the perspective of the stream data, the identification information of the stream data further includes a category identification bit for indicating at least one of the following situations: the target object is an identification object newly added to the current image relative to the encoded image; the The target object is an identification object whose position of the current image is changed relative to the encoded image; the target object is an identification object whose size of the current image is changed relative to the encoded image; the target object is the current image which is relatively encoded An identification object in which pixel information in the image area in the image changes. The change of the identification object is identified through the category identification bit. For example, the category identification bit indicates whether the identification object is newly added, or the position / size and pixel information are changed.

It should be understood that, in the embodiments of the present application, the identification object whose position of the current image changes relative to the encoded image may refer to a change in the position of the identification object itself, or a change in the position of the image region where the identification object is located. The identification object whose size of the current image changes relative to the encoded image may refer to a change in the size of the identification object itself, or may refer to a change in the size of an image area where the identification object is located.

In some implementations, the target object includes a newly added identification object of the current image relative to the encoded image, and the image region information includes an absolute value of a position and a size of the image region where the newly added identification object is located. When a new identification object is added to the coded image in the current image, the image area information and pixel information should be marked. The image area information may include an absolute value of a position and an absolute value of a size of an image area where the newly-identified object is located.

In some implementations, the target object may include an identification object whose current image is relatively encoded with a changed position, and then the image region information of the target object (that is, the identification object whose position is changed) includes the image where the target object is located. The absolute value of the location of a region or the relative value of a change in location. The absolute value of the position refers to the position of the image area where the target object is located in the current image; the relative value of the position change refers to the position of the image area where the target object is in the encoded image, and the target object The difference between the positions of the image area in the current image.

In the above implementation manner in which a target object (that is, an identification object whose position changes) exists in the current image relative to the encoded image, there may be a size of the image region in which the target object is located in the current image compared to the image in the decoded image. The size of the area changes or remains the same.

In the case of a change, optionally, the image area information of the target object includes an absolute value of a size of the image area where the target object is located or a relative value of the size change. The absolute value of the size refers to the size of the image area where the target object is located in the current image; the relative value of the size change refers to the size of the area where the target object is located in the encoded image, and The difference between the size of the image in the current image.

In the case where it remains unchanged, optionally, the image area information of the target object includes identification bits for indicating that the size of the image area where the target object is located remains unchanged in the decoded image. Optionally, the size of the image area is not encoded in the image area information of the target object in the code stream data.

In the foregoing implementation manner in which a target object (that is, an identification object whose position changes) exists in the current image relative to the encoded image, there may be pixel information of the target object in the image region where the target object is located compared to the pixel image in the decoded image. Pixels in the image area change or stay the same.

In the case where the change occurs, optionally, the pixel information of the target object includes an absolute value of an attribute of at least one pixel of the image region where the target object is located or a relative value of the attribute change of at least one pixel. The absolute value of the attribute refers to the attribute of at least one pixel of the image region where the target object is in the current image; the attribute of at least one pixel may refer to the absolute value of the attributes of all pixels in the image region, or it may be Refers to the absolute value of the attributes of some pixels in the image region whose attributes have changed. The relative value of the attribute change refers to the difference between the value assigned to the pixel of the image area of the target object in the current image and the value assigned to the pixel of the image area of the target object in the encoded image. . The relative value may be a difference value corresponding to all pixels in the image area, or may be a difference value corresponding to some pixels in the image area where attributes are changed.

In the case where it remains unchanged, optionally, the image area information of the target object includes an identification bit, which is used to indicate that the pixel information of the image area where the target object is located remains unchanged in the decoded image. Optionally, the pixel information of the target object is not encoded in the bitstream data.

In some implementations, the target object may include an identification object whose current image is relatively encoded with a changed size, and then the image region information of the target object (that is, the identification object whose size is changed) includes the image where the target object is located. The absolute value of the size of a region or the relative value of a change in size.

In the above implementation manner in which a target object (that is, an identification object whose size changes) exists in the current image relative to the encoded image, there may be a position of the target object in an image region in the current image compared to the image in the decoded image The location of the area changes or remains the same.

In the case of a change, optionally, the image area information of the target object includes an absolute value of a position of the image area where the target object is located or a relative value of a position change.

In the case where it remains unchanged, optionally, the image area information of the target object includes identification bits for indicating that the position of the image area where the target object is located remains unchanged in the decoded image. Optionally, the position of the image region is not encoded in the image region information of the target object in the code stream data.

In the case where the change occurs, optionally, the pixel information of the target object includes an absolute value of an attribute of at least one pixel of the image region where the target object is located or a relative value of the attribute change of at least one pixel.

In some implementations, the target object may include an identification object in which the pixel information of the image area where the current image is relative to the encoded image is changed. Then, the target object (that is, the identification object where the pixel information is changed) is in the image where the current image is located. The pixel information of the region includes an absolute value of an attribute of the at least one pixel of the image region where the target object is located or a relative value of a change in the attribute.

In the above implementation manner in which a target object (that is, an identification object whose pixel information changes) exists in the current image relative to the encoded image, there may be a position of the target object in the image region where the current image is located compared to the position in the decoded image. The position of the image area changes or remains the same.

In the above implementation manner in which a target object (that is, an identification object whose pixel information changes) exists in the current image relative to the encoded image, there may be a size of the image region where the target object is located in the current image compared to the size of the image region in the decoded image. The size of the image area changes or remains the same.

In the case of a change, optionally, the image area information of the target object includes an absolute value of a size of the image area where the target object is located or a relative value of the size change.

It should be added that at least part of the implementation methods described above can be combined. For example, for the case where the position and size of the image area where the target object is located remain unchanged, the image area information may include an identification bit, which is used to indicate that the size and position of the image area where the target object is located remain inferior to the encoded image change. It should be understood that the identification bit may be a single identification bit, which indicates that the size and position are unchanged; the identification bit may also include two sub-identification bits, which indicate that the size is unchanged and the position is unchanged.

Those skilled in the art can understand that when only one or two of the three parameters of the position of the image area where the target object is located, the size of the image area where the target object is located, and the pixels of the target object in the image area change, Each of the identification information may include both image area information and pixel information. In this scheme, the specific content of the identification information can be as follows.

Among them, ar_object_mask_present_flag indicates whether mask information of an object needs to be identified in the current image; ar_num_objects_minus1 indicates the number of objects that need to be identified in the current image; ar_object_idx [i] represents the label of the i-th object in the current image that needs to be identified; ar_bounding_box_mask_present_flag [ar_object_idx [i ]] Indicates whether the label ar_object_idx [i] has a mask identifying the shape of the object; ar_bounding_box_mask_infer_flag [ar_object_idx [i]] indicates whether the label value is from the previously encoded image when the label ar_object_idx [i] contains mask information ar_object_idx [i] mask of the object; ar_new_object_flag [ar_object_idx [i]] indicates whether the object labeled ar_object_idx [i] in the current image is a newly appeared object; ar_object_bounding_box_update_flag [ar_object_idx [i]] indicates the current image and the encoded image , Whether the position and size of the object labeled ar_object_idx [i] have changed in the current image; ar_object_top [ar_object_idx [i]], ar_object_left [ar_object_idx [i]], ar_object_width [ar_object_idx [i]], and ar_object_height [ar_ object_idx [i]] indicates the position and size of the object labeled ar_object_idx [i], where ar_object_top [ar_object_idx [i]] and ar_object_left [ar_object_idx [i]] indicate the position of the upper left corner of the object labeled ar_object_idx [i]; ar_object_width [ar_object_idx [i]] and ar_object_height [ar_object_idx [i]] represent the width and height of the object labeled ar_object_idx [i]. mask [m] [n] represents the template value corresponding to pixels with coordinates offset m and n in the vertical and horizontal directions relative to the upper left corner of the rectangular area. When the pixel belongs to the target object, the value of mask [m] [n] is 1; otherwise, when the pixel belongs to the background, the value of mask [m] [n] is 0.

In addition, for the identification of the mask, a point-by-point identification method may be adopted, or the target object may be represented by ar_object_top [ar_object_idx [i]], ar_object_left [ar_object_idx [i]], ar_object_width [ar_object_idx [i]], and ar_object_height [ar_object_idx [i]] identifies the starting position of each line of the target frame class and the length of the target object in the line. The specific method is as follows:

Among them, mask_pos [ar_object_idx [i]] [m] represents the starting position of the ar_object_idx [i] th object in the target frame, and mask_len [ar_object_idx [i]] [m] represents the ar_object_idx [i] th The length of the m-th row of the object in the target frame.

It should be understood that ar_new_object_flag and the like can be considered as the category identification bits mentioned above. ar_object_idx [i] is the label of the target object, which can also be called the indicator bit, number or index of the target object, which is used to indicate which target object it is.

Optionally, in some embodiments of the present application, the code stream data and / or the identification information may further include an indication bit of an encoded image, which is used to indicate which encoded image is currently referenced. The indication bit may be the number of the encoded image, or the number of frames from the current image in the encoding order. Of course, the code stream data and the identification information may not include the reference bit of the encoded image, but use the previous frame image or the previous N frame image specified or defaulted by the protocol as the reference encoded image.

Optionally, in other embodiments of the present application, the encoded image may be determined by the following method. The label of one or more target objects in the current image is used as a search condition, and among the multiple images that have been encoded, the image of the target object that is closest to the current image is searched as the encoded image used as a reference.

Optionally, in still other embodiments of the present application, the encoded image may be determined by the following method. Use at least one parameter value of the three parameters: the position, size, and pixel information of the image area where the target object in the current image is located as the search condition. From the multiple images that have been encoded, search for the value of the at least one parameter. The closest image is the encoded image used as a reference. The target object may be one or more. The search can be based on at least one parameter value of the same target object in the current image, that is, when the same target object and the position and / or size and / or pixel information are closest, it is considered that a coded image is searched for reference . The search may not be based on the same target object in the current image, but only based on at least one of the position, size, and pixel information, that is, regardless of whether the target objects are the same, the position and / or size and / or pixel information are When approached, it is considered that a coded image was searched for reference.

In one application scenario of drones, drones control camera equipment through the PTZ, so that target objects, such as people, remain at the center of the screen or at a specific position on the screen. In combination with the encoding method in the embodiment of the present application, the center of the image area where the target object is located is always maintained at the center of the picture or a specific position of the picture. In this application scenario, or other application scenarios where the position of the target object in the image area remains unchanged in multiple frames, the image area can be a rectangular area, and the image area information can include the coordinates of the center point of the rectangular area, the Height information and width information of the rectangular area. Because the position of the image area in this application scenario remains the same in multiple frames, but the size of the image area and / or the pixel information in the image area changes, the code stream data may not be centered on the image area in the image area information. The specific value of the point coordinate is encoded, but the flag is used to indicate that the value is unchanged.

For the above application scenario, the image area is a rectangular area, and the image area information includes the coordinates of the center point of the rectangular area, the height information of the rectangular area, and the width information of the rectangular area. The image area information may include an identification bit, which is used to indicate that the coordinates of the center point of the image area where the target object is located remain unchanged compared to the encoded image.

Optionally, in some embodiments of the present application, the identification information may also be used to identify a removed object of the current image relative to the encoded image. It should be understood that each identification object in the embodiments of the present application may have a unique label or index. In addition, the label or index of the same identified object in different images may be the same. In some possible implementation manners, the identification information includes label information of the removed object or position information of the removed object. In one example, the specific identification scheme of the removed object may be as follows.

Among them, ar_num_cancel_objects represents objects that the current image no longer exists relative to the encoded image; ar_cancel_object_idx [i] represents the labels of the objects that no longer exist.

The target objects mentioned above can be people, cars and public facilities. Optionally, in some embodiments of the present application, the identification information may further include content information, and the content information is used to indicate the content of the target object.

In one example, the content information may be label information. The label can use natural language to directly indicate the content of the target object. The natural language can use the Internet Engineering Task Force (IETF) Request For Comments (RFC) 5646 standard, that is, the IETF RFC 5646 standard. To represent. In another example, the content information may be a numerical value. That is, one-dimensional values can be added, and different values can be used to indicate what kind of content the target object is. For example, a value of content information of 1 indicates that the content of the target object is a person; a value of content information of 2 indicates that the content of the target object is a car.

Optionally, in some embodiments of the present application, the code stream data may further include image content data of the current image.

In a possible implementation manner, the image content data of the current image includes reference frame data of the current image and residual data between the current image and the reference frame.

FIG. 3 is a schematic flowchart of a decoding method 300 according to an embodiment provided in the present application. The decoding method 300 is performed by a decoding device. As shown in FIG. 3, the decoding method 300 includes: S310, obtaining code stream data of a current image, where the code stream data includes identification information, the identification information is used to identify at least one target object in the current image, and the identification information Including image area information and pixel information, the image area information includes the position and size of the image area where the target object is located, the pixel information includes attributes of at least one pixel in the image area; S320, at least part of the bitstream data Perform decoding processing.

In the decoding method provided in the embodiment of the present application, the position and size of the image area where the target object is located are indicated by the image area information, and the attributes of multiple pixels in the image area are indicated by the pixel information, thereby identifying the target object with finer granularity, It is beneficial for the decoding device to perform operations on the target object more efficiently and accurately.

In the decoding method provided in this embodiment, the bitstream data of the current image obtained in step 310 may be the same as the bitstream data in the encoding method provided in the present invention. For the interpretation of the bitstream data in step 310, refer to Explanation of code stream data in the above coding method.

Optionally, in some embodiments of the present application, the attribute of the at least one pixel may include whether the at least one pixel belongs to the target object.

Optionally, in some embodiments of the present application, the image area may include multiple sub-image areas, and the pixel information may include a value assigned to at least one pixel in the image area; wherein pixels in different sub-image areas are assigned different values .

Optionally, in some embodiments of the present application, at least one pixel may be assigned a different value in the pixel information, S320. Decoding the at least part of the bitstream data may include: Pixel information determines whether at least one pixel in the image area belongs to the target object.

In a possible implementation manner, the first part of the at least one pixel may be assigned a first value, and determining whether at least one pixel in the image area belongs to the target object according to the pixel information in the bitstream data may include: when In the pixel information in the bitstream data, the first portion of pixels corresponds to the first value, and it is determined that the first portion of pixels does not belong to the target object. For example, if the first part of the pixel information corresponds to 0, the first part of the pixel does not belong to the target object.

In another possible implementation manner, the second part of the at least one pixel may be assigned a second value. According to the pixel information in the bitstream data, determining whether at least one pixel in the image area belongs to the target object may include: : When the second portion of pixels in the pixel information in the bitstream data corresponds to the second value, it is determined that the second portion of pixels belongs to the target object. For example, if the second part of the pixel information corresponds to 1, the second part of the pixel belongs to the target object.

It should be understood that similar to the encoding method, the above two possible implementation manners may be implemented separately or in combination with each other, which is not limited in the embodiments of the present application.

Optionally, in some embodiments of the present application, the attribute of the at least one pixel may include a part of the target object to which the at least one pixel belongs.

In a possible implementation manner, different pixels may be assigned different values in the pixel information. S320 decoding at least part of the bitstream data may include: determining an image area according to the pixel information in the bitstream data. At least one of the pixels is in a part to which the target object belongs.

In a specific example, the target object may be a person; the first part of the at least one pixel may be assigned a third value, and according to the pixel information in the code stream data, it is determined that at least one pixel in the image area belongs to the target object. The part may include: when the first part of the pixel information in the bitstream data corresponds to the third value, determining that the first part of the pixel belongs to the head of the target object; and / or, the second part of the at least one pixel may be given The fourth value, according to the pixel information in the code stream data, determining at least one pixel in the image area at the part to which the target object belongs, may include: when the second part of the pixels in the pixel information in the code stream data corresponds to the fourth value, Determine that the second part of the pixel belongs to the hand of the target object.

In another specific example, the target object may be a car; the first part of the at least one pixel may be assigned a fifth value, and at least one pixel in the image area is determined to belong to the target object according to the pixel information in the code stream data. The part of the pixel information may include: when the first part of the pixel information in the bitstream data corresponds to the fifth value, determining that the first part of the pixel belongs to the head of the target object; and / or, the second part of the at least one pixel may be given The sixth value, according to the pixel information in the code stream data, determining that at least one pixel in the image area belongs to the target object, may include: when the second part of the pixel information in the pixel information in the code stream data corresponds to the sixth value, determining The second part of the pixels belongs to the rear of the target object.

In a possible implementation manner, the attribute of the at least one pixel may include a description feature corresponding to the at least one pixel. For example, the description feature corresponding to at least one pixel may include at least one of the following: the reflection intensity of a point cloud corresponding to the at least one pixel, the infrared intensity corresponding to the at least one pixel, and the depth value corresponding to the at least one pixel .

Optionally, in some embodiments of the present application, the attributes are measured in pixel blocks, and the pixel information may include attribute information of at least one pixel block, and the pixel block may include at least two pixels.

Optionally, in some embodiments of the present application, the code stream data may further include a category identification bit. The decoding method 300 may further include: determining that the target object may be at least one of the following cases according to the category identification bit: a newly added identification object of the current image relative to the decoded image; and an identification of a change in the position of the current image relative to the decoded image. An object; an identification object whose size of the current image changes relative to the decoded image; an identification object whose pixel information in the image area of the current image changes relative to the decoded image.

In some implementations, the target object may include a newly added identification object of the current image relative to the decoded image. The image area information may include the absolute value of the position and the size of the image area where the target object is located. When a new identification object is added to the decoded image in the current image, the image area information and pixel information should be marked. S320 decoding the at least part of the bitstream data may include: determining the position and size of the target object, that is, the image region where the newly identified object is located, according to the image region information in the bitstream data.

In some implementations, the target object may include an identification object whose current image is relative to the decoded image and whose position is changed. Then, the image region information of the target object (that is, the identification object whose position is changed) includes the image where the target object is located. The absolute value of the location of a region or the relative value of a change in location. Among them, the absolute value of the position refers to the position of the image area where the target object is located in the current image; the relative value of the position change refers to the position of the image area where the target object is in the decoded image, and the target object The difference between the positions of the image area in the current image.

When the image area information includes the relative value of the position change of the image area where the target object is located, S320 decoding at least part of the bitstream data may include: according to where the target object is located in the decoded image. The position of the image region and the relative value of the position change of the image region determine the position of the image region where the target object is located in the current image. For example, the decoding device may determine the position of the target image region in the decoded image; based on the position of the target image region in the decoded image, and the position of the target image region in the decoded image and the target object The difference between the positions of the image regions in the current image determines the position of the image region in which the target object is located in the current image.

In the above implementation manner in which the current image has a target object (that is, an identification object whose position changes) relative to the decoded image, there may be a size of the image region where the target object is located in the current image compared to the image in the decoded image. The size of the area changes or remains the same.

In the case of a change, optionally, the image area information of the target object includes an absolute value of a size of the image area where the target object is located or a relative value of the size change. The absolute value of the size refers to the size of the image area where the target object is located in the current image; the relative value of the size change refers to the size of the area where the target object is in the decoded image, and The difference between the size of the image in the current image.

When the image area information includes the relative value of the size change of the image area where the target object is located, S320 decoding at least part of the bitstream data may include: according to where the target object is located in the decoded image The size of the image area and the relative value of the size change of the image area determine the size of the image area where the target object is located in the current image. For example, the decoding device may determine the size of the image area where the target object is in the decoded image; based on the size of the image area where the target object is in the decoded image, and the size of the image area where the target object is in the decoded image and the target object The difference between the sizes of the image areas where the current image is located determines the size of the image area where the target object is located in the current image.

In the case where it remains unchanged, optionally, the image area information of the target object includes identification bits for indicating that the size of the image area where the target object is located remains unchanged in the decoded image. Optionally, the size of the image area is not encoded in the image area information of the target object in the code stream data. S320 decodes at least part of the bitstream data, and may further include: determining, based on a size of an image area in which the target object is located in the decoded image, an image area of the target object in the current image. size. That is, the size of the image area where the target object is located in the decoded image is determined as the size of the image area where it is located in the current image.

In the above implementation manner in which a target object (that is, an identification object whose position changes) exists in the current image relative to the decoded image, there may be pixel information of the target object in the image area where the target object is located compared to the pixel information in the decoded image. Pixels in the image area change or stay the same.

In the case where the change occurs, optionally, the pixel information of the target object includes an absolute value of an attribute of at least one pixel of the image region where the target object is located or a relative value of the attribute change of at least one pixel. The absolute value of the attribute refers to the attribute of at least one pixel of the image region where the target object is in the current image; the attribute of at least one pixel may refer to the absolute value of the attributes of all pixels in the image region, or it may be Refers to the absolute value of the attributes of some pixels in the image region whose attributes have changed. The relative value of the attribute change refers to the difference between the value assigned to the pixels of the image area of the target object in the current image and the value assigned to the pixels of the image area of the target object in the decoded image. . The relative value may be a difference value corresponding to all pixels in the image area, or a difference value corresponding to some pixels in the image area where attributes are changed, that is, when the difference value is 0, the difference value may be omitted.

When the pixel information includes the relative value of the attribute change of at least one pixel of the image area where the target object is located, S320 decodes at least part of the bitstream data, and may further include: according to the target object in the decoded The pixel information in the image and the relative value of the attribute change of the at least one pixel determine the pixel information of the target object in the current image. For example, the decoding device may determine the attributes of at least one pixel of the image region where the target object is in the decoded image; according to the attributes of at least one pixel of the image region where the target object is in the decoded image, and where the target object is in the decoded image A difference between the attribute of at least one pixel of the image region and the attribute of at least one pixel of the image region where the target object is in the current image determines the attribute of at least one pixel of the image region where the target object is in the current image.

When the pixel information includes information about a part of the pixels whose attributes have changed in the image region where the target object is located in the current image, the decoding device may consider that the attributes of the remaining pixels have not changed.

In the case where it remains unchanged, optionally, the image area information of the target object includes an identification bit, which is used to indicate that the pixel information of the image area where the target object is located remains unchanged in the decoded image. Optionally, the pixel information of the target object is not encoded in the code stream data. Correspondingly, S320 decodes at least part of the bitstream data, and may further include: determining pixel information of the target object in the image region where the current image is located according to the pixel information of the image region where the target object is in the decoded image.

In some implementations, the target object may include an identification object whose current image is relative to the decoded image and whose size changes, and then the image region information of the target object (that is, the identification object whose size has changed) includes the image where the target object is located. The absolute value of the size of a region or the relative value of a change in size. When the image area information includes the relative value of the size change of the image area where the target object is located, S320 decoding at least part of the bitstream data may include: according to where the target object is located in the decoded image The size of the image area and the relative value of the size change of the image area determine the size of the image area where the target object is located in the current image.

In the above implementation manner in which a target object (that is, an identification object whose size changes) exists in the current image relative to the decoded image, there may be a position where the target object is located in an image region in the current image compared to the image in the decoded image. The location of the area changes or remains the same.

In the case of a change, optionally, the image area information of the target object includes an absolute value of a position of the image area where the target object is located or a relative value of a position change. When the image area information includes the relative value of the position change of the image area where the target object is located, S320 decodes at least part of the bitstream data, and may further include: according to the target object in the decoded image The position of the image region where the image is located and the relative value of the position change of the image region determine the position of the image region where the target object is located in the current image.

In the case where it remains unchanged, optionally, the image area information of the target object includes identification bits for indicating that the position of the image area where the target object is located remains unchanged in the decoded image. Optionally, the position of the image region is not encoded in the image region information of the target object in the code stream data. S320 decodes at least part of the bitstream data, and may further include: determining, based on the position of the target object in the image region where the target object is located, the image region of the target object in the current image. position. That is, the position of the image region where the target object is in the decoded image is determined as the position of the image region where it is in the current image.

In the case where the change occurs, optionally, the pixel information of the target object includes an absolute value of an attribute of at least one pixel of the image region where the target object is located or a relative value of the attribute change of at least one pixel. When the pixel information includes the relative value of the attribute change of at least one pixel of the image area where the target object is located, S320 decodes at least part of the bitstream data, and may further include: according to the target object in the decoded The pixel information in the image and the relative value of the attribute change of the at least one pixel determine the pixel information of the target object in the current image.

In the case where it remains unchanged, optionally, the image area information of the target object includes an identification bit, which is used to indicate that the pixel information of the image area where the target object is located remains unchanged in the decoded image. Optionally, the pixel information of the target object is not encoded in the bitstream data. Optionally, the pixel information of the target object is not encoded in the code stream data. Correspondingly, S320 decodes at least part of the bitstream data, and may further include: determining pixel information of the target object in the image region where the current image is located according to the pixel information of the image region where the target object is in the decoded image.

In some implementations, the target object may include an identification object in which the pixel information of the image area where the current image is relative to the decoded image is changed. Then, the target object (that is, the identification object where the pixel information is changed) is in the image where the current image is located. The pixel information of the region includes an absolute value of an attribute of the at least one pixel of the image region where the target object is located or a relative value of a change in the attribute.

In the above implementation manner in which a target object (that is, an identification object whose pixel information changes) exists in the current image relative to the decoded image, there may be a position of the target object in the image region where the current image is located compared to the position in the decoded image The position of the image area changes or remains the same. When the pixel information includes the relative value of the attribute change of at least one pixel of the image region where the target object is located, S320 decoding at least part of the bitstream data may include: according to the target object in the decoded image And the relative value of the attribute change of the at least one pixel to determine the pixel information of the target object in the current image.

In the above implementation manner in which a target object (that is, an identification object whose pixel information changes) exists in the current image relative to the decoded image, there may be a size of the image area where the target object is located in the current image compared to the size of the image area in the decoded image. The size of the image area changes or remains the same.

In the case of a change, optionally, the image area information of the target object includes an absolute value of a size of the image area where the target object is located or a relative value of the size change. When the image area information includes the relative value of the size change of the image area where the target object is located, S320 decodes at least part of the bitstream data, which may include: The size of the image area and the relative value of the size change of the image area determine the size of the image area where the target object is located in the current image.

In the case where it remains unchanged, optionally, the image area information of the target object includes identification bits for indicating that the size of the image area where the target object is located remains unchanged in the decoded image. Optionally, the size of the image area is not encoded in the image area information of the target object in the code stream data. S320 decodes at least part of the bitstream data, and may further include: determining, based on a size of an image area in which the target object is located in the decoded image, an image area of the target object in the current image. size.

It should be added that at least part of the implementation methods described above can be combined.

In a specific application scenario of the drone, the image area may be a rectangular area. The image region information may include coordinates of a center point of the rectangular region, height information of the rectangular region, and width information of the rectangular region. When the position of the image area where the target object is kept unchanged and the size changes, the code stream data may not include the numerical value of the coordinates of the center point of the image area in the image area information, but use the identification bit to indicate its content constant. The image area information may further include an identification bit, which is used to indicate that the coordinates of the center point of the image area where the target object is located remain unchanged. S320 performing decoding processing on at least a part of the bitstream data may include: determining a center point of the target object in the image area according to coordinates of a center point of the image area where the target object is in the decoded image. coordinate. The decoding device can determine the coordinates of the center point of the identified object in the image area based on the coordinates of the center point of the image area where the identified object remains in the decoded image; determine the image according to the image area information of the current image The height information and width information of the area; determine the image area where the identity object remains unchanged according to the coordinates of the center point of the image area, and the height information and width information of the image area.

Optionally, in some embodiments of the present application, the identification information may also be used to identify a removed object of the current image relative to the decoded image.

In a possible implementation manner, the identification information may include label information of the removed object or position information of the removed object.

Optionally, in some embodiments of the present application, the code stream data may further include image content data of the current image. S320 performing decoding processing on at least a part of the code stream data may include: performing decoding processing on image content data of the current image in the code stream data.

Optionally, in some embodiments of the present application, S320 decoding at least part of the code stream data may include: decoding the identification information in the code stream data, obtaining the current image and the decoded identification information.

Optionally, in some embodiments of the present application, S320 decoding the at least part of the code stream data may include: discarding the identification information and not decoding the identification information.

Optionally, in some embodiments of the present application, the identification information may further include content information. S320 performing decoding processing on at least a part of the bitstream data may include: determining content of the target object according to content information in the bitstream data.

In a possible implementation manner, the content information may be label information.

In another possible implementation manner, the content information may be a numerical value.

Optionally, in some embodiments of the present application, the image area may be a rectangular area.

In a possible implementation manner, the image region information may include coordinates of any corner of the rectangular region, height information of the rectangular region, and width information of the rectangular region.

Alternatively, the image area information may include the coordinates of the center point of the rectangular area, the height information of the rectangular area, and the width information of the rectangular area.

Alternatively, the image region information may include coordinates of the upper left corner of the rectangular region and coordinates of the lower right corner of the rectangular region.

Alternatively, the image region information may include coordinates of the upper right corner of the rectangular region and coordinates of the lower left corner of the rectangular region.

Optionally, in some embodiments of the present application, the identification information may be located in auxiliary enhancement information or extended data of the current image.

binary values

0 and 1. The template value of the pixels belonging to the target object in the pixel information is 1; the template value of the pixels belonging to the background is 0. The image area of the target object i is a rectangular area; the image area information of the target object i includes the coordinates of the upper left corner of the rectangular area, the height information of the rectangular area, and the width information of the rectangular area; the pixel information of the target object i is a template As an example, for a decoding device, the specific content of the identification information of the target object i can be as follows. Those skilled in the art can understand that this content is only schematic and can be obtained from other alternative forms or solutions, which are not listed here one by one.

Among them, ar_object_top [i], ar_object_left [i], ar_object_width [i], and ar_object_height [i] represent the position and size of the target object i, ar_object_top [i] and ar_object_left [i] represent the position of the upper left corner of the target object i; ar_object_width [i] and ar_object_height [i] represent the width and height of the target object i. mask [m] [n] represents the template value corresponding to pixels with coordinates offset m and n in the vertical and horizontal directions relative to the upper left corner of the rectangular area. When the value of the decoded mask_value is 1, the value of mask [m] [n] is 1, indicating that the pixel belongs to the target object i; when the value of the decoded mask_value is 0, mask [m] [n] A value of 0 indicates that the pixel belongs to the background.

For the decoding device, the related information of the identification object that decodes the current image can refer to the situation of the decoded image. The specific content of the identification information received by the decoding device may be as follows.

Among them, ar_object_mask_present_flag indicates whether mask information is required to identify the object in the current image; ar_num_cancel_objects indicates objects that the current image no longer exists relative to the decoded image; ar_cancel_object_idx [i] indicates the label of the object that no longer exists; ar_num_objects_minus1 indicates that the current image requires The number of objects identified; ar_object_idx [i] indicates the label of the i-th object in the current image; ar_bounding_box_mask_present_flag [ar_object_idx [i]] indicates whether the label ar_object_idx [i] has a mask identifying the shape of the object; ar_bounding_box_mask_infer_flag [ar_object_idx [ i]] indicates that when the label ar_object_idx [i] contains mask information, whether the mask value is from the mask of the previously encoded image label ar_object_idx [i]; ar_new_object_flag [ar_object_idx [i]] indicates the label in the current image Whether the object of ar_object_idx [i] is a newly appearing object; ar_object_bounding_box_update_flag [ar_object_idx [i]] represents the position and position of the object labeled ar_object_idx [i] in the current image and the decoded image. Whether the size has changed; ar_object_top [ar_object_idx [i]], ar_object_left [ar_object_idx [i]], ar_object_width [ar_object_idx [i]], and ar_object_height [ar_object_idx [i]] indicate the position and size of the object labeled ar_object_idx [i] , Ar_object_top [ar_object_idx [i]] and ar_object_left [ar_object_idx [i]] indicate the position of the upper left corner of the object labeled ar_object_idx [i]; ar_object_width [ar_object_idx [i]] and ar_object_height [ar_object_idx [i]] indicate that the label is ar_object_id i] the width and height of the object. If there is no change, the position, size, and pixel information of the rectangular area corresponding to ar_object_idx [i] are consistent with the position, size, and pixel information of the rectangular area corresponding to the label ar_object_idx [i] in the decoded image; if there is a change , The position, size, and pixel information of the rectangular region corresponding to ar_object_idx [i] are re-decoded. mask [m] [n] represents the template value corresponding to pixels with coordinates offset m and n in the vertical and horizontal directions relative to the upper left corner of the rectangular area. When the value of the decoded mask_value is 1, the value of mask [m] [n] is 1, indicating that the pixel belongs to the object labeled ar_object_idx [i]; when the value of the decoded mask_value is 0, mask [ The value of m] [n] is 0, which means that the pixel belongs to the background.

Optionally, in some embodiments of the present application, the bitstream data and / or the identification information may further include an indication bit of a decoded image, which is used to indicate which decoded image is currently referenced. The indication bit may be the number of the decoded image, or the number of frames from the current image in the decoding order. Of course, the code stream data and the identification information may not include the reference bit of the decoded image, but use the previous frame image or the previous N frame image specified or defaulted by the protocol as the reference decoded image.

The method of the embodiment of the present application has been described in detail above, and the encoding device and the decoding device of the embodiment of the present application are described in detail below.

FIG. 4 is a schematic block diagram of an encoding device 400 according to an embodiment of the present application. As shown in FIG. 4, the encoding device 400 includes:

At least one memory 410, configured to store computer-executable instructions;

At least one processor 420 is used alone or collectively to access the at least one memory 410 and execute the computer-executable instructions to perform the following operations:

Encoding the current image to generate code stream data, the code stream data includes identification information, the identification information is used to identify at least one target object in the current image, and the identification information includes image area information and pixels Information, the image region information includes a position and a size of an image region where the target object is located, and the pixel information includes attributes of at least one pixel in the image region.

The encoding device in the embodiment of the present application indicates the position and size of the image area where the target object is located through the image area information, and indicates the attributes of multiple pixels in the image area through the pixel information, thereby identifying the target object with finer granularity. It is beneficial for the decoding device to perform operations on the target object more efficiently and accurately.

In some embodiments, the attributes of the at least one pixel include whether the at least one pixel belongs to the target object.

In some embodiments, the image region includes a plurality of sub-image regions, and the pixel information includes a value assigned to at least one pixel in the image region; wherein pixels in different sub-image regions are assigned different values.

In some embodiments, the pixel information assigns different values to the at least one pixel to indicate whether the at least one pixel belongs to the target object.

In some embodiments, among the at least one pixel, a first partial pixel is assigned a first value to indicate that the first partial pixel does not belong to the target object.

In some embodiments, of the at least one pixel, a second partial pixel is assigned a second value to indicate that the second partial pixel belongs to the target object.

In some embodiments, the attributes of the at least one pixel include a part of the target object to which the at least one pixel belongs.

In some embodiments, different values are assigned to different pixels in the pixel information to indicate that the different pixels belong to different parts of the target object.

In some embodiments, the target object is a person;

A first partial pixel of the at least one pixel is assigned a third numerical value to indicate that the first partial pixel belongs to a head of the target object;

and / or,

A second partial pixel of the at least one pixel is given a fourth numerical value, which is used to indicate that the second partial pixel belongs to a hand of the target object.

In some embodiments, the target object is a car;

A first partial pixel of the at least one pixel is assigned a fifth value, which is used to indicate that the first partial pixel belongs to a head of the target object;

and / or,

A second part of the at least one pixel is assigned a sixth value for indicating that the second part of the pixel belongs to the rear of the vehicle of the target object.

In some embodiments, the attributes of the at least one pixel include descriptive features corresponding to the at least one pixel.

In some embodiments, the description feature corresponding to the at least one pixel includes at least one of the following: a reflection intensity of a point cloud corresponding to the at least one pixel, an infrared intensity corresponding to the at least one pixel, and the at least one The depth value corresponding to the pixel.

In some embodiments, the attributes are measured in pixel blocks, the pixel information includes information on attributes of at least one pixel block, and the pixel blocks include at least two pixels.

In some embodiments, the target object is an object that meets at least one of the following conditions:

An identification object added to the current image relative to the encoded image;

An identification object whose position of the current image changes relative to the encoded image;

An identification object whose size of the current image changes relative to the encoded image;

An identification object in which the pixel information in the image area of the current image is changed relative to the encoded image.

In some embodiments, the code stream data further includes a category identification bit for indicating at least one of the following situations:

The target object is a newly added identification object of the current image relative to the encoded image;

The target object is an identification object whose position of the current image is changed relative to the encoded image;

The target object is an identification object that changes the size of the current image relative to the encoded image;

The target object is an identification object in which the current image is changed relative to pixel information in the image area in the encoded image.

In some embodiments, the target object includes a new identification object of the current image relative to the encoded image, and the image area information includes an absolute value and a size of a position of an image area where the new identification object is located. Absolute value.

In some embodiments, the target object includes an identification object whose position is changed relative to the encoded image and the current image;

The image region information includes an absolute value of a position of an image region where the target object is located or a relative value of a position change.

In some embodiments, the image area information includes an identification bit, which is used to indicate that the size of the image area where the target object is located remains unchanged compared to the encoded image.

In some embodiments, the target object includes an identification object in which the current image is changed in size relative to the encoded image;

The image area information includes an absolute value of a size of an image area where the target object is located or a relative value of a size change.

In some embodiments, the pixel information includes an identification bit, which is used to indicate that the pixel information of the image region where the target object is located remains unchanged from the encoded image.

In some embodiments, the target object includes an identification object in which the current image is changed with respect to the encoded image and pixel information, and the pixel information includes an absolute value of the pixel information or a relative value of the pixel information change.

In some embodiments, the pixel information includes an identification bit, which is used to indicate that the pixel information of the image area where the target object is located is changed compared to the encoded image.

In some embodiments, the image region information includes an identification bit, which is used to indicate that the size and / or position of the image region where the target object is located remains unchanged from the encoded image.

In some embodiments, the image area is a rectangular area, and the image area information includes coordinates of a center point of the rectangular area, height information of the rectangular area, and width information of the rectangular area;

The image region information includes an identification bit, which is used to indicate that the coordinates of the center point of the image region where the target problem is located remain unchanged compared to the encoded image.

In some embodiments, the identification information is further used to identify a removed object of the current image relative to the encoded image.

In some embodiments, the identification information includes label information of the removed object or position information of the removed object.

In some embodiments, the processor 420 is further configured to:

Determining that the current image is relatively encoded with the newly added object to be identified as the target object;

Determining, as the target object, an object to be identified in which the current image has a changed position and / or size relative to the encoded image;

An object to be identified whose pixel information in the image area changes from the current image to the encoded image is determined as the target object.

In some embodiments, the identification information further includes content information, and the content information is used to indicate content of the target object.

In some embodiments, the content information is label information.

In some embodiments, the content information is a numerical value.

In some embodiments, the image area is a rectangular area.

In some embodiments, the image region information includes coordinates of any corner of the rectangular region, height information of the rectangular region, and width information of the rectangular region;

or,

The image area information includes coordinates of a center point of the rectangular area, height information of the rectangular area, and width information of the rectangular area;

or,

The image area information includes coordinates of an upper left corner of the rectangular area and coordinates of a lower right corner of the rectangular area;

or,

The image region information includes coordinates of an upper right corner of the rectangular region and coordinates of a lower left corner of the rectangular region.

In some embodiments, before the encoding process is performed on the current image to generate code stream data, the processor 420 may be further configured to:

Performing image recognition on the current image, determining the target object, and obtaining the identification information of the target object.

In some embodiments, the identification information is located in auxiliary enhancement information or extended data of the current image.

It should be understood that the encoding device of each embodiment of the present application may be implemented based on a module. For example, FIG. 5 is a schematic block diagram of an encoding device 500 according to an embodiment of the present application. As shown in FIG. 5, the encoding device 500 may include an encoding module 510 for performing encoding processing, generating code stream data, and the like. Each module in the encoding device may be used to execute the methods in the embodiments of the present application, and details are not described herein again.

FIG. 6 is a schematic block diagram of a decoding device 600 according to an embodiment of the present application. As shown in Figure 6,

At least one memory 610, configured to store computer-executable instructions;

At least one processor 620, alone or collectively, is configured to access the at least one memory 610 and execute the computer-executable instructions to perform the following operations:

Obtaining bitstream data of a current image, the bitstream data including identification information, the identification information being used to identify at least one target object in the current image, the identification information including image area information and pixel information, the The image region information includes a position and a size of an image region where the target object is located, and the pixel information includes attributes of at least one pixel in the image region;

Performing decoding processing on at least part of the code stream data.

The decoding device provided in the embodiment of the present application indicates the position and size of the image area where the target object is located through the image area information, and indicates the attributes of multiple pixels in the image area through the pixel information, thereby identifying the target object with finer granularity, It is beneficial for the decoding device to perform operations on the target object more efficiently and accurately.

In some embodiments, the pixel information assigns different values to the at least one pixel, and the processor 620 decodes at least a part of the bitstream data, including:

Determining whether the at least one pixel in the image region belongs to the target object according to pixel information in the code stream data.

In some embodiments, the determining, by the processor 620 according to the pixel information in the code stream data, whether the at least one pixel in the image region belongs to the target object may include:

When the first partial pixel in the pixel information in the bitstream data corresponds to the first value, it is determined that the first partial pixel does not belong to the target object.

When the second partial pixel in the pixel information in the bitstream data corresponds to the second value, it is determined that the second partial pixel belongs to the target object.

In some embodiments, different values are assigned to different pixels in the pixel information,

The processing performed by the processor 620 on at least a part of the code stream data includes:

Determining, according to the pixel information in the code stream data, a part of the image area where the at least one pixel belongs to the target object.

In some embodiments, the target object is a person;

The determining, by the processor 620 according to the pixel information in the bitstream data, the at least one pixel in the image region at a part to which the target object belongs includes:

When the first partial pixel in the pixel information in the bitstream data corresponds to the third value, determining that the first partial pixel belongs to the head of the target object;

and / or,

When the second partial pixel in the pixel information in the bitstream data corresponds to the fourth value, it is determined that the second partial pixel belongs to the hand of the target object.

In some embodiments, the target object is a car;

When the first partial pixel in the pixel information in the bitstream data corresponds to the fifth value, determining that the first partial pixel belongs to the head of the target object;

and / or,

A second part of the at least one pixel is assigned a sixth value, and the processor 620 determines that the at least one pixel in the image area belongs to the target object according to the pixel information in the code stream data Parts, including:

It is determined that the second part of the pixels belong to the rear of the target object according to the second part of the pixels in the bitstream information corresponding to the sixth value.

In some embodiments, the attributes of the at least one pixel include descriptive characteristics corresponding to the at least one pixel.

In some embodiments, the attribute is a measurement unit of a pixel block, the pixel information includes information of an attribute of at least one pixel block, and the pixel block includes at least two pixels.

In some embodiments, the code stream data includes a category identification bit, and the processor 620 is further configured to:

Determining that the target object is at least one of the following cases according to the category identifier:

An identification object added to the current image relative to the decoded image;

An identification object whose position of the current image is changed relative to the decoded image;

An identification object in which the size of the current image relative to the decoded image changes;

An identification object in which the pixel information in the image area of the current image is changed relative to the decoded image.

In some embodiments, the target object includes an identification object newly added to the decoded image of the current image, and the image region information includes an absolute value of a position and an absolute value of a size of an image region where the target object is located.

In some embodiments, the target object includes an identification object that changes a position of the current image relative to a decoded image;

The image region information includes an absolute value of a position of an image region where the target object is located,

or,

The image region information includes a relative value of a position change of an image region where the target object is located, and the processor 620 decodes at least a part of the bitstream data, which may include:

The position of the image region where the target object is in the current image is determined according to the position of the image region where the target object is in the decoded image and the relative value of the position change of the image region.

In some embodiments, the image region information includes an identification bit, which is used to indicate that a size of an image region where the target object is located remains unchanged in the decoded image;

The processor 620 may perform decoding processing on at least a part of the code stream data, and further includes:

Determining the size of the image area in which the target object is located in the current image according to the size of the image area in which the target object is located in the decoded image.

In some embodiments, the target object includes an identification object whose size of the current image changes relative to the decoded image;

The image area information includes an absolute value of a size of the image area,

or,

The image region information includes a relative value of a size change of the image region, and the processor 620 decodes at least a part of the bitstream data, including:

The size of the image region in which the target object is located in the current image is determined according to the size of the image region in which the target object is located in the decoded image and the relative value of the size change of the image region.

In some embodiments, the pixel information includes an identification bit for indicating that the pixel information of an image area where the target object is located remains unchanged from the decoded image;

The processor 620 decodes at least a part of the code stream data, and further includes:

Determine pixel information of the target object in the image region where the current image is located according to the pixel information of the image region where the target object is in the decoded image.

In some embodiments, the code stream data includes the pixel information;

Decoding pixel information of the target object in an image area where the current image is located.

In some embodiments, the bitstream data further includes an identification bit for indicating that pixel information of an image area where the target object is located is changed compared to the decoded image.

In some embodiments, the target object includes an identification object that changes pixel information of the current image relative to the decoded image;

The pixel information includes an absolute value of an attribute of the at least one pixel;

or,

The pixel information includes a relative value of an attribute change of the at least one pixel, and the processor 620 decodes at least a part of the code stream data, including:

Determining pixel information of the target object in the current image according to pixel information of the target object in the decoded image and a relative value of an attribute change of the at least one pixel.

In some embodiments, the image area information further includes an identification bit, which is used to indicate that the target object is unchanged in the image area where the current image is located compared to the decoded image;

Determining image area information of the target object in the current image according to image area information of the target object in the decoded image.

The image area information further includes an identification bit, which is used to indicate that the coordinates of the center point of the image area where the target object is located remain unchanged;

Determine the center point coordinates of the target object in the image area according to the center point coordinates of the image area where the target object is in the decoded image.

In some embodiments, the identification information is further used to identify a removed object of the current image relative to the decoded image.

In some embodiments, the identification information includes label information of the removed object or position information of the removed object in the decoded image.

In some embodiments, the processor 620 decodes at least a portion of the code stream data, including:

Decoding the identification information in the bitstream data to obtain the current image and the decoded identification information.

The identification information is discarded, and the identification information is not decoded.

In some embodiments, the code stream data further includes image content data of the current image;

Decoding processing is performed on image content data of the current image in the code stream data.

In some embodiments, the image content data of the current image includes reference frame data of the current image and residual data between the current image and the reference frame.

In some embodiments, the identification information further includes content information,

Determine the content of the target object according to the content information in the code stream data.

In some embodiments, the content information is label information.

In some embodiments, the content information is a numerical value.

In some embodiments, the image area is a rectangular area.

or,

It should be understood that the decoding device in each embodiment of the present application may be implemented based on a module. For example, FIG. 7 is a schematic block diagram of a decoding device 700 according to an embodiment of the present application. As shown in FIG. 7, the decoding device 700 may include an obtaining module 710 to obtain code stream data of a current image, and further include a decoding module 720 to perform decoding processing on at least part of the code stream data. Each module in the decoding device may be used to execute the methods in the embodiments of the present application, and details are not described herein again.

The present application also provides an image processing method. FIG. 8 is a schematic flowchart of an image processing method 800 according to an embodiment of the present application. As shown in FIG. 8, the method 800 includes the following steps.

S810. Obtain code stream data of the current image. The code stream data includes identification information, where the identification information is used to identify at least one target object in the current image. The identification information includes image area information and pixel information. The image area information. It includes the position and size of the image area where the target object is located, and the pixel information includes attributes of at least one pixel in the image area.

S820: Decode the bitstream data to obtain the current image and the identification information.

S830. Perform pixel-level processing on the current image according to the identification information.

In the image processing method of the embodiment of the present application, the position and size of the image area where the target object is located are indicated by the image area information, and the attributes of multiple pixels in the image area are indicated by the pixel information, thereby identifying the target object with finer granularity The decoding device can perform pixel-level processing on the target object more efficiently and accurately.

It should be understood that, in the existing solutions, because the operation of identifying the target object is more complicated, the hardware of the decoding device is usually higher, and the decoding device is usually a computer or a server. The image processing method in the embodiment of the present application enables recognition of a target object to be performed on an encoding end, and a decoding device only needs to perform subsequent image processing. Therefore, on the one hand, the image processing method in the embodiment of the present application can be implemented on platforms such as mobile phones and tablet computers; on the other hand, the computing resources of the decoding device can be used for more complex image processing, so that the decoding device can present better quality More beautiful images.

In some embodiments of the present application, S830 performs pixel-level processing on the current image according to the identification information, which may include: changing display content of the current image according to the identification information.

In other embodiments of the present application, S830 performs pixel-level processing on the current image according to the identification information, and may include: performing statistics on data information in the current image according to the identification information.

In other words, the identification information may include image area information and finer pixel information. When the decoding device performs display processing or statistics on one or more pixels in the current image, referring to the image area information and finer pixel information, it can reduce its own calculation amount, can save computing resources, and reduce processing requirements. time. The image processing method according to the embodiment of the present application will be described in more detail in terms of display and statistics respectively.

As described above, the attribute of the at least one pixel includes whether the at least one pixel belongs to the target object. In one embodiment, in the pixel information, at least one pixel is assigned a different value to indicate whether the at least one pixel belongs to the target object. In one example, a first portion of pixels in at least one pixel is assigned a first value to indicate that the first portion of pixels does not belong to the target object; and / or a second portion of pixels in at least one pixel is assigned a second value to indicate a second portion Some pixels belong to the target object.

Optionally, in some embodiments of the present application, the method 800 may further include: acquiring a first image. S830: Performing pixel-level processing on the current image according to the identification information may include: performing fusion processing on the current image and the first image based on the identification information to obtain a second image, and the second image. Including at least part of the content of the current image and at least part of the content of the first image.

It should be understood that the parameters of the current image and the second image may be the same, such as the same size, the same number of pixels, and the same resolution. The parameters of the current image and the second image may also be different, which is not limited in the embodiment of the present application.

In an embodiment, the performing fusion processing on the current image and the first image based on the identification information may include: weighting the current image and the first image based on the identification information. And, a weight value of a pixel corresponding to the target object in the current image is different from a weight value of at least a part of pixels in the current image except the target object. The weight value of the pixel corresponding to the target object is large, and the weight value of at least some pixels other than the target object is small, so that the target object of the current image in the second image obtained after the fusion process is more prominent than the non-target object. In addition, the following processing may be adopted: weighting and summing the pixels in the current image and the pixels in the first image, and the weight of the pixels belonging to the target object in the current image is greater than that of the first image A weight of a pixel at a corresponding position in the, and a weight of a pixel in the current image that does not belong to the target object is smaller than a weight of a corresponding pixel in the first image.

For example, the weight value of the pixels corresponding to the target object is 0.6, and the weight value of the pixels corresponding to the position of the target object in the first image is 0.4; the weight value of at least some pixels except the target object in the current image is 0.2, and the first image corresponds to The weighting value of at least some pixels at positions other than the target object is 0.8. The final image effect is that the current image floats semi-transparently on the first image as the background, and the target object is more prominent in the fused picture. The target object floats on the background shown in the first image.

In another embodiment, the performing fusion processing on the current image and the first image based on the identification information may include: determining, based on the image area information and the pixel information, whether the Pixels that do not belong to the target object; replacing pixels in the current image that do not belong to the target object with corresponding pixels in the first image to obtain a second image.

In a specific example, the at least one pixel is assigned a different value. The determining, according to the image area information and the pixel information, pixels in the current image that do not belong to the target object may include: according to the image area information, removing pixels outside the image area where the target object is located. The pixels are determined to be pixels that do not belong to the target object in the current image; when a first portion of pixels in the pixel information corresponds to a first value, it is determined that the first portion of pixels does not belong to the target object. In this way, the decoding device can simply determine the details such as the position, size, and boundary of the target object without performing complicated calculations.

In a specific example, replacing pixels that do not belong to the target object in the current image with corresponding pixels in the first image to obtain a second image may include: comparing the pixels in the current image with all pixels in the current image. The pixels in the first image are weighted and summed, and pixels in the current image that do not belong to the target object are replaced with corresponding pixels in the first image. Among them, some pixels of the first image are used as the background. For example, the pixels in the current image and the pixels in the first image are weighted and summed together, the weight of the pixels belonging to the target object in the current image is 1, and the corresponding position in the first image is The weight of a pixel is 0; the weight of a pixel that does not belong to the target object in the current image is 0, and the weight of a pixel at a corresponding position in the first image is 1. If the pixel information that belongs to the target object is assigned a value of 1 and the pixels that do not belong to the target object are assigned a value of 0, the value in the pixel information may directly correspond to the weight of the pixel during fusion. 9A and 9B are schematic diagrams of two images obtained by fusion according to an embodiment of the present application. The figure shows the target object and the background, respectively.

Optionally, before determining, according to the image area information and the pixel information, pixels in the current image that do not belong to the target object, the method may further include: according to the pixel information, Determining a boundary of the target object, and performing an expansion operation on the target object based on the boundary; and determining, according to the image area information and the pixel information, pixels in the current image that do not belong to the target object, The method includes: determining pixels in the current image that do not belong to the target object according to the image area information, the pixel information, and a boundary of the expanded target object. It should be understood that, in the embodiment of the present application, the expansion operation is to expand the field of the target object so that pixels that originally did not belong to the target object become pixels that belong to the target object. Specifically, the value of the pixels in the pixel information near the original boundary of the target object is changed from 0 to 1, so that when these pixels are fused, they are not replaced by pixels in the first image, thereby protecting the target object.

In another embodiment, the pixels of the target object can be extracted for other occasions based on the identification information. Accordingly, the performing pixel-level processing on the current image according to the identification information may include: determining, according to the image area information and the pixel information, pixels in the current image that belong to the target object, Extract pixels belonging to the target object.

Optionally, the shooting angle of the current image is the same as the shooting angle of the first image. The embodiments of the present application can be applied to a single-frame image or video. For example, a scene in which the method is applied to a single frame image can be as follows. Someone has a photo of a person (ie, a target object) taken at a certain position on the top of Mount Tai and at a certain shooting angle. However, because the shooting day was cloudy, the sunrise was not captured. In addition, there is a first image, which is also a picture of the scenery taken at the position and angle of the top of Mount Tai. By merging the person's photo with the scene's photo, or reducing or enlarging the fusion, a very realistic fusion effect can be obtained, and a commemorative photo of the person bathing in the rising sun can be obtained. Similarly, this method can be applied to video scenes to obtain better Augmented Reality (AR) effects, which will be described in detail below.

It should be understood that the shooting angle of the current image may be carried in the code stream data or more specifically in the identification information. Correspondingly, the identification information further includes viewing angle information, which is used to indicate a shooting angle of the target object relative to the shooting device, or a shooting angle of the current image. Before replacing the pixels in the current image that do not belong to the target object with the corresponding pixels in the first image, the method may further include: determining a shooting angle of the target object and a shooting of the first image Same angle. Of course, the identification information may not include the perspective information. The decoding device determines the shooting angle of the target object relative to the shooting device through other methods or algorithms, which is not limited in the embodiment of the present application.

Optionally, in some embodiments, S830, performing pixel-level processing on the current image according to the identification information may include: determining, based on the image area information and the pixel information, For the target object, an augmented reality AR special effect is added to the target object.

In an embodiment, the identification information may further include perspective information, the perspective information is used to indicate a shooting angle of the target object relative to the shooting device, and the attribute of the at least one pixel may further include the at least one pixel Corresponding depth value. Determining the target object in the current image and adding an AR special effect to the target object according to the image region information and the pixel information includes: determining the target object according to the image region information and the pixel information The target object in the current image; adding an AR special effect to the target object according to the target object, the shooting angle, and a depth value corresponding to the at least one pixel. It should be understood that the AR special effect may be adding an icon (for example, an arrow, a halo, etc.), a text, a layer, and the like.

It should be understood that the shooting angle of the current image and / or the depth value corresponding to at least one pixel may be carried in the code stream data or more specifically in the identification information, or the decoding device may determine the target object through other methods or algorithms, etc. Shooting angle relative to the camera. For example, the viewing angle information and / or the depth value corresponding to at least one pixel may be obtained by calculating the ground attitude of the drone, which is not limited in the embodiment of the present application.

In a specific example, in the pixel information, the at least one pixel is assigned a different value. For example, a first portion of pixels in at least one pixel is assigned a first value to indicate that the first portion of pixels does not belong to the target object; and / or a second portion of pixels in at least one pixel is assigned a second value to indicate that the second portion of pixels belongs to Target object. The determining the boundary of the target object in the current image and adding an augmented reality AR effect to the target object according to the image area information and the pixel information may include: when the second part of the pixel information When the pixel corresponds to the second value, it is determined that the second part of the pixel belongs to the target object; based on the boundary of the target object, an indicator halo is added to the target object. 10A and 10B are schematic diagrams of adding an indicator halo on a target object according to an embodiment of the present application. As shown in FIG. 10A, when the bright portion of the indication halo is rotated in front of the target object, the bright portion of the indication halo blocks the target object. As shown in FIG. 10B, when the bright portion of the indicating halo is rotated behind the target object, the bright portion of the indicating halo is blocked by the target object.

Optionally, in some embodiments, the performing the pixel-level processing on the current image according to the identification information in S830 may include: determining, based on the image area information and the pixel information, whether the Pixels that belong to the target object or pixels of non-target objects that do not belong to the target object in the current image; change at least one of the brightness, color, and grayscale of the target object, or change the non-target At least one of the brightness, color, and grayscale of the object, or changing the contrast of the target object and the non-target object.

Specifically, changing the brightness or color of the target object or changing the brightness or color of the non-target object may include: modifying the YUV value, the RGB value, or γ of the target object or the non-target object Curve to change the brightness or color of the target object, or change the brightness or color of the non-target object. When the image is a grayscale image, the grayscale of the target object can be changed according to the identification information. When the contrast between the target object and the non-target object needs to be emphasized, the contrast between the target object and the non-target object can be improved.

Let's take a few specific examples to explain how to change the brightness, color, grayscale or contrast. In these examples, the at least one pixel in the pixel information may be assigned a different value. For example, a first portion of pixels in at least one pixel is assigned a first value to indicate that the first portion of pixels does not belong to the target object; and / or a second portion of pixels in at least one pixel is assigned a second value to indicate that the second portion of pixels belongs to Target object.

For example, when the second partial pixel in the pixel information corresponds to a second value, it is determined that the second partial pixel belongs to the target object; and the brightness of the second partial pixel is increased.

For another example, when the first partial pixel in the pixel information corresponds to a first value, it is determined that the first partial pixel does not belong to the target object; and the brightness of the first partial pixel is reduced.

For another example, when a first partial pixel in the pixel information corresponds to a first value, it is determined that the first partial pixel belongs to the target object; the first partial pixel is identified by a preset color.

For another example, when the first partial pixel in the pixel information corresponds to a first value, it is determined that the first partial pixel belongs to the target object; when the second partial pixel in the pixel information corresponds to a second value, the first The two partial pixels belong to the target object; the contrast between the first partial pixels and the second partial pixels is improved.

Optionally, the identification information further includes content information, which is used to indicate a content category of the target object. The changing at least one of the brightness, color, and grayscale of the target object may include: when the target object is a first content category, changing the brightness of the target object to a preset first brightness value Change the color of the target object to a preset first color value, or change the gray level of the target object to a preset first gray value. FIG. 11 is a schematic diagram of changing the brightness of a target object according to an embodiment of the present application. For example, the current image includes multiple target objects. Part of the content categories of the multiple target objects are people, and the other content categories are not people. The content category can be changed to a brightness of a target object of a person to a preset first brightness value. As shown in FIG. 11, the content category is that the target object of a person is displayed by being highlighted or popped out to facilitate observation by an observer. For example, this example can be used for playback and viewing of the recording results of a smart camera, or real-time playback.

Optionally, when a plurality of the target objects are included in the current image, the changing at least one of the brightness, color, and grayscale of the target object may include: Content categories, which assign different brightness values, color values, or grayscale values to the target objects of different content categories. That is, target objects of different content categories are identified with different brightness, color, or grayscale, so as to facilitate observation by an observer.

Optionally, in some embodiments, S830 performing pixel-level processing on the current image according to the identification information may include: according to the image area information, the pixel information, and the content category, based on the The current image generates an object class segmentation image. In object segmentation images, target objects of different content categories can be given different colors. FIG. 12A is an original image of the current image, and FIG. 12B is an object category segmentation image corresponding to the current image. As shown in FIG. 12B, for example, cars are marked with blue, buildings are marked with gray, and ground is marked with purple. In addition, people are marked with red, street lights are marked with yellow, plants are marked with green, etc., which are not shown in FIG. 12B.

It should be understood that, as described above, the content information may be a label or a value.

Optionally, in some embodiments, the attributes of the at least one pixel may include a part of the target object to which the at least one pixel belongs. For example, the at least one pixel in the pixel information is assigned different values to indicate that the at least one pixel belongs to a different part of the target object.

Optionally, S830 performing pixel-level processing on the current image according to the identification information may include: using different brightness and colors for different parts of the target object according to the image area information and the pixel information. Or grayscale identification, or different contrast between different parts. In the following, different parts of the target object are marked with different colors as an example. FIG. 13 is a schematic diagram of images identified by different colors at different parts of an embodiment of the present application. For example, the person at the lower left of FIG. 13 forms a target object together with a bag and a bicycle. In this target object, people are identified by yellow, bags are identified by red, and bicycles are identified by green. In addition, in this image, target objects of other content types, such as cars are marked in blue, and people are marked in yellow.

Optionally, the attribute of the at least one pixel may include a description feature corresponding to the at least one pixel. The description feature corresponding to the at least one pixel may include at least one of the following: a reflection intensity of a point cloud corresponding to the at least one pixel, an infrared intensity corresponding to the at least one pixel, and a depth value corresponding to the at least one pixel. .

Optionally, in some embodiments of the present application, the current image may include a plurality of the target objects. S830 performing pixel-level processing on the current image according to the identification information may include: according to the reflection intensity, infrared intensity, or depth value of multiple target objects, the The target object is given different brightness, color, and gray values. Taking color as an example, S830 performs pixel-level processing on the current image according to the identification information, which may include at least one of the following processing.

In the processing of one embodiment, a reflection intensity segmented image is generated based on the current image according to the image area information, pixel information, and the reflection intensity of a point cloud corresponding to the at least one pixel. In the reflection intensity segmentation image, each part of the target object can be distinguished, and the parts with different reflection intensities are identified by different colors; the parts of the target object can also be distinguished, such as the average of the reflection intensity of each part of the target object (or the reflection The intensity itself corresponds to the average reflection intensity of the entire target object), a target object is identified by a single color. Target objects with different reflection intensities are assigned different colors. FIG. 14A is an original image of the current image, and FIG. 14B is a reflection intensity segmented image corresponding to the current image. As shown in FIG. 14B, target objects with different reflection intensities are given different colors, and the portions in FIG. 14B that do not belong to any target object, such as the background portion, are identified by white.

In the processing of one embodiment, a depth map is generated based on the current image according to the image area information, pixel information, and a depth value corresponding to the at least one pixel. In the depth map, each part of the target object can be distinguished, and the parts with different depth values are identified by different colors; the parts of the target object can also be distinguished, such as averaging the depth values of the parts of the target object (or the depth value itself). Is the average depth value corresponding to the entire target object), a target object is identified by a single color. FIG. 15A is an original image of the current image, and FIG. 15B is a depth map corresponding to the current image. As shown in FIG. 15B, pixels of different depth values are assigned different colors.

Similarly, an infrared image may be generated based on the current image according to the image area information, the pixel information, and an infrared intensity corresponding to the at least one pixel, and details are not described herein again.

Optionally, in some embodiments of the present application, S830 performs pixel-level processing on the current image according to the identification information, which may include: performing statistics on data information in the current image according to the identification information.

Optionally, in some embodiments of the present application, the identification information further includes content information, which is used to indicate a content category of the target object. Some statistics can be made using the content category of the target object included in the content information.

Correspondingly, S830 performing pixel-level processing on the current image according to the identification information may include: performing statistics on the target object in the current image according to the content category of the target object to obtain a statistical result.

For example, the statistics of the target object in the current image according to the content category of the target object to obtain a statistical result may include: performing statistics on the target object whose content category is a person in the current image to obtain People flow results and / or crowd density results. This scenario can be used by the municipal management department for crowd management during peak commutes or holidays, or it can be used for commercial layout purposes, and used to count passenger traffic.

As another example, the statistics of the target object in the current image according to the content category of the target object to obtain a statistical result may include: counting the target object whose content category is a car in the current image To obtain the results of traffic flow and / or density results. This scenario can be used by traffic management departments for rush-hour commute or traffic management at public transit stations.

Optionally, in some embodiments of the present application, the pixel-level processing in S830 may be expression recognition or motion recognition.

For example, when the target object is a person, S830 performs pixel-level processing on the current image according to the identification information, which may include: when a first partial pixel in the pixel information corresponds to a third value, determining the first partial pixel Belonging to the head of the target object; performing facial expression recognition according to the head of the target object; and / or, performing the pixel-level processing on the current image according to the identification information in S830 may include: when the When the second part of the pixel information corresponds to the fourth value, it is determined that the second part of the pixel belongs to the hand of the target object; hand motion recognition is performed according to the hand of the target object.

This scenario can be used in the field of drones. The decoding device may send a control instruction to the drone based on the result of the facial expression recognition or the result of hand motion recognition. For example, when the hand is in a "T" shape, the drone is suspended or returned to home. Another example is when the operator nods, the drone speeds up and so on. The meaning represented by the facial expressions or hand movements may be agreed in advance by the drone and the control terminal, which is not described in the embodiment of the present application.

Optionally, in some embodiments of the present application, the pixel-level processing in S830 may involve traffic management and the like. For example, if the target object is a car, S830 performs pixel-level processing on the current image according to the identification information, which may include: determining a first partial pixel when the first partial pixel in the pixel information corresponds to a fifth value. The head of the target object; determining the driving direction of the target object according to the head of the target object; and / or, S830 performing pixel-level processing on the current image according to the identification information may include: when When the first partial pixel in the pixel information corresponds to a sixth value, it is determined that the second partial pixel belongs to the rear of the target object; and the driving direction of the target object is determined according to the rear of the target object. The method implemented in this application can quickly find the retrograde vehicle and assist the traffic police to deal with it in time.

Optionally, in some embodiments of the present application, the description feature corresponding to the at least one pixel includes at least one of the following: a reflection intensity of a point cloud corresponding to the at least one pixel, and an infrared corresponding to the at least one pixel. An intensity and a depth value corresponding to the at least one pixel. S830: Performing pixel-level processing on the current image according to the identification information may include: performing statistics on the target object in the current image according to the description feature to obtain a statistical result. Taking the depth value as an example, the number of target objects whose depth value is a certain value, or the number of target objects whose depth value is within a certain range, is used for some distance-related statistics, which are not described here one by one.

Optionally, after the statistics are completed, the method 800 may further include: generating a heat map for the statistical results of the target object according to the statistical results. That is, after the statistics are completed, a heat map is generated for displaying the statistical results.

It should be understood that, in the embodiments of the present application, the attributes may be measured in pixels; the attributes may also be measured in pixel blocks, and the pixel information may include information about attributes of at least one pixel block. A block includes at least two pixels.

It should be understood that, in the embodiments of the present application, the identification information may be located in auxiliary enhancement information or extended data of the current image.

The image processing method of the embodiment of the present application has been described in detail above, and the image processing apparatus of the embodiment of the present application is described in detail below.

FIG. 16 is a schematic block diagram of an image processing apparatus 1600 according to an embodiment of the present application. As shown in FIG. 16, the device 1600 includes:

At least one memory 1610 for storing computer-executable instructions;

At least one processor 1620, alone or collectively, is used to access the at least one memory 1610 and execute the computer-executable instructions to perform the following operations:

Decoding the bitstream data to obtain the current image and the identification information;

Pixel level processing is performed on the current image according to the identification information.

The image processing apparatus according to the embodiment of the present application indicates the position and size of the image area where the target object is located through the image area information, and indicates the attributes of multiple pixels in the image area through the pixel information, thereby identifying the target object with finer granularity This allows the image processing device to perform pixel-level processing on the target object more efficiently and accurately.

It should be understood that the image processing apparatus 1600 may be a decoding device. In the existing solutions, because the operation of identifying the target object is more complicated, the hardware of the decoding device is usually higher, and the decoding device is usually a computer or a server. The image processing method in the embodiment of the present application enables recognition of a target object to be performed on an encoding end, and a decoding device only needs to perform subsequent image processing. Therefore, on the one hand, the image processing method in the embodiment of the present application can be implemented on platforms such as mobile phones and tablet computers; on the other hand, the computing resources of the decoding device can be used for more complex image processing, so that the decoding device can present better quality More beautiful images.

Optionally, in some embodiments, the processor 1620 performs pixel-level processing on the current image according to the identification information, including: changing display content of the current image according to the identification information.

Optionally, in some embodiments, the attributes of the at least one pixel include whether the at least one pixel belongs to the target object.

Optionally, in some embodiments, the processor 1620 is further configured to obtain a first image; the processor performing pixel-level processing on the current image according to the identification information includes: based on the identification information Fusion processing is performed on the current image and the first image to obtain a second image, where the second image includes at least part of the content of the current image and at least part of the content of the first image.

Optionally, in some embodiments, the processor 1620 performs fusion processing on the current image and the first image based on the identification information, including: performing a fusion process on the current image and the first image based on the identification information. The first image performs weighted summing, wherein the weighting value of pixels corresponding to the target object in the current image is different from the weighting value of at least some pixels in the current image except for the target object.

Optionally, in some embodiments, the processor 1620 performs fusion processing on the current image and the first image based on the identification information, including: determining according to the image area information and the pixel information Pixels that do not belong to the target object in the current image; replace pixels that do not belong to the target object in the current image with corresponding pixels in the first image to obtain a second image.

Optionally, in some embodiments, in the pixel information, the at least one pixel is assigned a different value, and the processor 1620 determines the current image based on the image area information and the pixel information. Pixels that do not belong to the target object include: determining pixels outside the image region where the target object is located as pixels in the current image that do not belong to the target object according to the image region information; when the pixels When the first portion of pixels in the information corresponds to the first value, it is determined that the first portion of pixels does not belong to the target object.

Optionally, in some embodiments, before the processor 1620 determines pixels that do not belong to the target object in the current image according to the image area information and the pixel information, the processor 1620 further uses At: determining a boundary of the target object based on the pixel information, and performing an expansion operation on the target object based on the boundary; the processor 1620 determining the current based on the image area information and the pixel information Pixels that do not belong to the target object in the image include: determining pixels that do not belong to the target object in the current image according to the image area information, the pixel information, and the boundary of the expanded target object.

Optionally, in some embodiments, the shooting angle of the current image is the same as the shooting angle of the first image.

Optionally, in some embodiments, the identification information further includes viewing angle information, where the viewing angle information is used to indicate a shooting angle of the target object relative to the shooting device; in the processor 1620, the Before the corresponding pixels replace pixels in the current image that do not belong to the target object, the processor 1620 is further configured to determine that the shooting angle of the target object is the same as the shooting angle of the first image.

Optionally, in some embodiments, the processor 1620 performs pixel-level processing on the current image according to the identification information, including: determining the current image according to the image area information and the pixel information. Adding the augmented reality AR special effect to the target object in.

Optionally, in some embodiments, the identification information further includes viewing angle information, the viewing angle information is used to indicate a shooting angle of the target object relative to the shooting device, and the attribute of the at least one pixel further includes the at least A depth value corresponding to one pixel; the processor 1620 determines the target object in the current image according to the image area information and the pixel information, and adds an AR special effect to the target object, including: Image area information and the pixel information, determining the target object in the current image; adding an AR special effect to the target object according to the target object, the shooting angle, and a depth value corresponding to the at least one pixel .

Optionally, in some embodiments, the processor 1620 performs pixel-level processing on the current image according to the identification information, including: determining the current image according to the image area information and the pixel information. A pixel belonging to the target object or a pixel of a non-target object that does not belong to the target object in the current image; changing at least one of the brightness, color, and grayscale of the target object, or changing the non-target object At least one of the brightness, color, and grayscale of the target object, or changing the contrast of the target object and the non-target object.

Optionally, in some embodiments, the processor 1620 changes the brightness or color of the target object, or changes the brightness or color of the non-target object, including: modifying the target object or the non-target The YUV value, RGB value, or γ curve of the object to change the brightness or color of the target object, or change the brightness or color of the non-target object.

Optionally, in some embodiments, the identification information further includes content information, which is used to indicate a content category of the target object.

Optionally, in some embodiments, the processor 1620 changes at least one of brightness, color, and grayscale of the target object, including: when the target object is a first content category, changing the The brightness of the target object is a preset first brightness value, the color of the target object is changed to a preset first color value, or the grayscale of the target object is changed to a preset first grayscale value.

Optionally, in some embodiments, the current image includes a plurality of the target objects, and the processor 1620 changes at least one of the target object's brightness, color, and grayscale, including: Content categories of the target objects, and assign different brightness values, color values, or grayscale values to the target objects of different content categories.

Optionally, in some embodiments, the processor 1620 performs pixel-level processing on the current image according to the identification information, including: according to the image area information, the pixel information, and the content category, An object category segmentation image is generated based on the current image.

Optionally, in some embodiments, the content information is a label or a value.

Optionally, in some embodiments, the attributes of the at least one pixel include a part of the target object to which the at least one pixel belongs.

Optionally, in some embodiments, in the pixel information, the at least one pixel is assigned a different value to indicate that the at least one pixel belongs to a different part of the target object.

Optionally, in some embodiments, the processor 1620 performs pixel-level processing on the current image according to the identification information, including: according to the image area information and the pixel information, performing processing on the target object Different parts of the logo are marked with different brightness, color, or grayscale, or have different contrast between different parts.

Optionally, in some embodiments, the attribute of the at least one pixel includes a description feature corresponding to the at least one pixel.

Optionally, in some embodiments, the description feature corresponding to the at least one pixel includes at least one of the following: a reflection intensity of a point cloud corresponding to the at least one pixel, an infrared intensity corresponding to the at least one pixel, and A depth value corresponding to the at least one pixel.

Optionally, in some embodiments, the processor 1620 performs pixel-level processing on the current image according to the identification information, including at least one of the following: according to the image area information, the pixel information A reflection intensity segmentation image of a point cloud corresponding to the at least one pixel is generated based on the current image; based on the image area information, the pixel information, and an infrared intensity corresponding to the at least one pixel, based on the An infrared image is generated from the current image; a depth map is generated based on the current image according to the image area information, the pixel information, and a depth value corresponding to the at least one pixel.

Optionally, in some embodiments, the current image includes a plurality of the target objects, and the processor 1620 performs pixel-level processing on the current image according to the identification information, including: The reflection intensity, infrared intensity, or depth value of the target object is described, and different brightness values, color values, and grayscale values are assigned to the target object with different reflection intensity, infrared intensity, or depth value.

Optionally, in some embodiments, the processor 1620 performs pixel-level processing on the current image according to the identification information, including: performing statistics on data information in the current image according to the identification information. .

Optionally, in some embodiments, the processor 1620 performs pixel-level processing on the current image according to the identification information, including: according to the content category of the target object, processing all the current images in the current image. Statistics on the target objects are obtained, and statistical results are obtained.

Optionally, in some embodiments, the target object is a person, and the processor 1620 performs pixel-level processing on the current image according to the identification information, including: when the first portion of pixels in the pixel information corresponds to At the third value, it is determined that the first part of the pixels belong to the head of the target object; and facial expression recognition is performed according to the head of the target object;

and / or,

The processor 1620 performs pixel-level processing on the current image according to the identification information, including: when the second partial pixel in the pixel information corresponds to a fourth value, determining that the second partial pixel belongs to the target The hand of the object; performing hand motion recognition according to the hand of the target object.

Optionally, in some embodiments, the processor 1620 is further configured to send a control instruction to the drone based on a result of facial expression recognition or a result of hand motion recognition.

Optionally, in some embodiments, the target object is a car, and the processor 1620 performs pixel-level processing on the current image according to the identification information, including: when a first portion of pixels in the pixel information corresponds to At a fifth value, determining that the first portion of pixels belongs to the head of the target object; and determining the driving direction of the target object according to the head of the target object;

and / or,

The processor 1620 performs pixel-level processing on the current image according to the identification information, including: when a first partial pixel in the pixel information corresponds to a sixth value, determining that the second partial pixel belongs to the target object The rear of the vehicle; determining the driving direction of the target according to the rear of the vehicle.

Optionally, in some embodiments, the processor 1620 performs statistics on the target object in the current image according to the content category of the target object, and obtains a statistical result, including: The content category collects statistics on the target objects of people, and obtains the results of the flow of people and / or the results of the density of people.

Optionally, in some embodiments, the processor 1620 performs statistics on the target object in the current image according to the content category of the target object, and obtains a statistical result, including: The content category collects statistics on the target objects of the vehicle, and obtains the results of the traffic flow and / or the results of the traffic density.

Optionally, in some embodiments, the description feature corresponding to the at least one pixel includes at least one of the following: a reflection intensity of a point cloud corresponding to the at least one pixel, an infrared intensity corresponding to the at least one pixel, and A depth value corresponding to the at least one pixel; and the processor 1620 performs pixel-level processing on the current image according to the identification information, including: according to the description feature, the target object in the current image Perform statistics and get statistical results.

Optionally, in some embodiments, the processor 1620 is further configured to generate a heat map for the statistical result of the target object according to the statistical result.

Optionally, in some embodiments, the attributes are measured in pixel blocks, the pixel information includes information on attributes of at least one pixel block, and the pixel blocks include at least two pixels.

Optionally, in some embodiments, the identification information is located in auxiliary enhancement information or extended data of the current image.

It should be understood that the image processing apparatus according to each embodiment of the present application may be implemented based on a module. For example, FIG. 17 is a schematic block diagram of an image processing apparatus 1700 according to an embodiment of the present application. As shown in FIG. 17, the image processing device 1700 may include an obtaining module 1710 for obtaining code stream data of the current image; a decoding module 1720 for decoding the code stream data to obtain the current image and the identification information; processing Module 1730 is configured to perform pixel-level processing on the current image according to the identification information. Each module in the image processing apparatus 1700 can be used to execute the image processing method in each embodiment of the present application, and details are not described herein again.

It should be understood that the processor mentioned in the embodiments of the present application may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), and application-specific integrated circuits (DSPs). application specific integrated circuit (ASIC), ready-made programmable gate array (field programmable gate array, FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may be any conventional processor or the like.

It should also be understood that the memory mentioned in the embodiments of the present application may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), or Erase programmable read-only memory (EPROM, EEPROM) or flash memory. The volatile memory may be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (synchronous DRAM, SDRAM), double data rate synchronous dynamic random access memory (double SDRAM, DDR SDRAM), enhanced synchronous dynamic random access memory (enhanced SDRAM, ESDRAM), synchronous connection dynamic random access memory (synchlink DRAM, SLDRAM ) And direct memory bus random access memory (direct RAMbus RAM, DR RAM).

It should be noted that when the processor is a general-purpose processor, a DSP, an ASIC, an FPGA, or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component, the memory (memory module) is integrated in the processor.

It should be noted that the memories described herein are intended to include, but are not limited to, these and any other suitable types of memory.

An embodiment of the present application further provides a computer-readable storage medium having instructions stored thereon. When the instructions are run on the computer, the computer is caused to execute the methods of the foregoing method embodiments.

An embodiment of the present application further provides a computer program, which causes a computer to execute the methods of the foregoing method embodiments.

An embodiment of the present application further provides a computing device, where the computing device includes the computer-readable storage medium described above.

The embodiments of the present application can be applied in the field of aircraft, especially in the field of drones.

It should be understood that the division of circuits, sub-circuits, and sub-units in the embodiments of the present application is merely schematic. Those of ordinary skill in the art may realize that the circuits, sub-circuits, and sub-units of the examples described in the embodiments disclosed herein can be split or combined again.

In the above embodiments, it may be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented in software, it may be implemented in whole or in part in the form of a computer program product. A computer program product includes one or more computer instructions. When computer instructions are loaded and executed on a computer, the processes or functions according to the embodiments of the present application are generated in whole or in part. The computer may be a general purpose computer, a special purpose computer, a computer network, or other programmable device. The computer instructions may be stored in or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from a web site, computer, server, or data center via a wired (e.g., Coaxial cable, optical fiber, digital subscriber line (DSL), or wireless (such as infrared, wireless, microwave, etc.) transmission to another website site, computer, server, or data center. The computer-readable storage medium may be any available medium that can be accessed by a computer or a data storage device such as a server, a data center, or the like that includes one or more available medium integrations. Usable media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., high-density digital video discs (DVDs)), or semiconductor media (e.g., solid state disks (SSDs) )Wait.

It should be understood that “an embodiment” or “an embodiment” mentioned throughout the specification means that a particular feature, structure, or characteristic related to the embodiment is included in at least one embodiment of the present application. Thus, the appearances of "in one embodiment" or "in an embodiment" appearing throughout the specification are not necessarily referring to the same embodiment. Furthermore, the particular features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

It should be understood that, in various embodiments of the present application, the size of the sequence numbers of the above processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic. The implementation process constitutes any limitation.

It should be understood that in the embodiment of the present application, “B corresponding to A” means that B is associated with A, and B can be determined according to A. However, it should also be understood that determining B based on A does not mean determining B based solely on A, but also determining B based on A and / or other information.

It should be understood that the term “and / or” herein is only an association relationship describing an associated object, and indicates that there can be three kinds of relationships, for example, A and / or B can mean: A exists alone, and A and B exist simultaneously. There are three cases of B alone. In addition, the character "/" in this text generally indicates that the related objects are an "or" relationship.

Those of ordinary skill in the art may realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be implemented by electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. A professional technician can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices, and units described above can refer to the corresponding processes in the foregoing method embodiments, and are not repeated here.

In the several embodiments provided in this application, it should be understood that the disclosed systems, devices, and methods may be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the unit is only a logical function division. In actual implementation, there may be another division manner. For example, multiple units or components may be combined or Can be integrated into another system, or some features can be ignored or not implemented. In addition, the displayed or discussed mutual coupling or direct coupling or communication connection may be indirect coupling or communication connection through some interfaces, devices or units, which may be electrical, mechanical or other forms.

The units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, may be located in one place, or may be distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the objective of the solution of this embodiment.

In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each of the units may exist separately physically, or two or more units may be integrated into one unit.

The above is only a specific implementation of this application, but the scope of protection of this application is not limited to this. Any person skilled in the art can easily think of changes or replacements within the technical scope disclosed in this application. It should be covered by the protection scope of this application. Therefore, the protection scope of this application shall be subject to the protection scope of the claims.

Claims

An image processing method, comprising:

Obtaining bitstream data of a current image, the bitstream data including identification information, the identification information being used to identify at least one target object in the current image, the identification information including image area information and pixel information, the The image region information includes a position and a size of an image region where the target object is located, and the pixel information includes attributes of at least one pixel in the image region;

Decoding the bitstream data to obtain the current image and the identification information;

Pixel level processing is performed on the current image according to the identification information.
The method according to claim 1, wherein:

The performing pixel-level processing on the current image according to the identification information includes:

Changing the display content of the current image according to the identification information.
The method according to claim 1 or 2, wherein the attribute of the at least one pixel includes whether the at least one pixel belongs to the target object.
The method according to claim 3, further comprising:

Obtaining a first image;

The performing pixel-level processing on the current image according to the identification information includes:

Fusion processing the current image and the first image based on the identification information to obtain a second image, where the second image includes at least part of the content of the current image and at least part of the content of the first image.
The method according to claim 4, wherein the performing fusion processing on the current image and the first image based on the identification information comprises:

Weighting and summing the current image and the first image based on the identification information, wherein a weight value of a pixel corresponding to the target object in the current image is equal to a weight value of a pixel other than the target object in the current image At least some pixels have different weighting values.
The method according to claim 4 or 5, wherein the performing fusion processing on the current image and the first image based on the identification information comprises:

Determine, according to the image area information and the pixel information, pixels in the current image that do not belong to the target object;

Replace pixels in the current image that do not belong to the target object with corresponding pixels in the first image to obtain a second image.
The method according to claim 6, wherein in the pixel information, at least one pixel is assigned a different value,

The determining, according to the image area information and the pixel information, pixels in the current image that do not belong to the target object includes:

Determining, according to the image region information, pixels outside the image region where the target object is located as pixels in the current image that do not belong to the target object;

When the first partial pixel in the pixel information corresponds to a first value, it is determined that the first partial pixel does not belong to the target object.
The method according to claim 7, wherein before the determining, according to the image area information and the pixel information, pixels in the current image that do not belong to the target object, the method further comprises:

Determining a boundary of the target object according to the pixel information, and performing an expansion operation on the target object based on the boundary;

The determining, according to the image area information and the pixel information, pixels in the current image that do not belong to the target object includes:

Determining pixels in the current image that do not belong to the target object according to the image region information, the pixel information, and the boundary of the target object after expansion.
The method according to any one of claims 4 to 8, wherein a shooting angle of the current image is the same as a shooting angle of the first image.
The method according to claim 9, wherein the identification information further comprises viewing angle information, and the viewing angle information is used to indicate a shooting angle of the target object relative to the shooting device;

Before replacing the pixels in the current image that do not belong to the target object with corresponding pixels in the first image, the method further includes:

It is determined that the shooting angle of the target object is the same as the shooting angle of the first image.
The method according to claim 3, wherein:

The performing pixel-level processing on the current image according to the identification information includes:

Determining the target object in the current image according to the image area information and the pixel information, and adding an augmented reality AR special effect to the target object.
The method according to claim 11, wherein the identification information further comprises perspective information, the perspective information is used to indicate a shooting angle of the target object relative to the shooting device, and the attribute of the at least one pixel further comprises A depth value corresponding to the at least one pixel;

The determining the target object in the current image according to the image area information and the pixel information, and adding an AR special effect to the target object includes:

Determining the target object in the current image according to the image area information and the pixel information;

Adding an AR special effect to the target object according to the target object, the shooting angle, and a depth value corresponding to the at least one pixel.
The method according to claim 3, wherein:

The performing pixel-level processing on the current image according to the identification information includes:

Determining, according to the image area information and the pixel information, pixels in the current image that belong to the target object or pixels in the current image that do not belong to the target object;

Changing at least one of the brightness, color, and grayscale of the target object, or changing at least one of the brightness, color, and grayscale of the non-target object, or changing the target object and the non-target object Contrast.
The method according to claim 13, wherein:

The changing the brightness or color of the target object, or changing the brightness or color of the non-target object includes:

By modifying the YUV value, RGB value, or γ curve of the target object or the non-target object, the brightness or color of the target object is changed, or the brightness or color of the non-target object is changed.
The method according to claim 1, 3, 13 or 14, wherein the identification information further comprises content information, which is used to indicate a content category of the target object.
The method according to claim 15, wherein:

The changing at least one of brightness, color, and grayscale of the target object includes:

When the target object is a first content category, changing the brightness of the target object to a preset first brightness value, changing the color of the target object to a preset first color value, or changing the target object The grayscale is a preset first grayscale value.
The method according to claim 15, wherein the current image includes a plurality of the target objects,

The changing at least one of brightness, color, and grayscale of the target object includes:

According to the content categories of a plurality of the target objects, different brightness values, color values, or grayscale values are assigned to the target objects of different content categories.
The method according to claim 15, wherein:

The performing pixel-level processing on the current image according to the identification information includes:

An object category segmented image is generated based on the current image based on the image area information, the pixel information, and the content category.
The method according to any one of claims 15 to 18, wherein the content information is a label or a value.
The method according to claim 1 or 2, wherein the attribute of the at least one pixel includes a part of the target object to which the at least one pixel belongs.
The method according to claim 1, 2 or 20, wherein in the pixel information, the at least one pixel is assigned a different value to indicate that the at least one pixel belongs to different parts of the target object.
The method according to claim 1, 2, 20 or 21, wherein:

The performing pixel-level processing on the current image according to the identification information includes:

According to the image area information and the pixel information, different parts of the target object are identified with different brightness, color, or grayscale, or different parts have different contrasts.
The method according to any one of claims 1 to 3, wherein the attribute of the at least one pixel includes a description characteristic corresponding to the at least one pixel.
The method according to claim 23, wherein the description feature corresponding to the at least one pixel includes at least one of the following: a reflection intensity of a point cloud corresponding to the at least one pixel, and a The infrared intensity and a depth value corresponding to the at least one pixel.
The method according to claim 24, wherein:

The performing pixel-level processing on the current image according to the identification information includes at least one of the following:

Generating a reflection intensity segmentation image based on the current image according to the image area information, the pixel information, and the reflection intensity of a point cloud corresponding to the at least one pixel;

Generating an infrared image based on the current image according to the image area information, the pixel information, and an infrared intensity corresponding to the at least one pixel;

Generating a depth map based on the current image according to the image area information, the pixel information, and a depth value corresponding to the at least one pixel.
The method according to claim 24, wherein the current image includes a plurality of the target objects,

The performing pixel-level processing on the current image according to the identification information includes:

According to the reflection intensity, infrared intensity, or depth value of a plurality of the target objects, different brightness values, color values, and grayscale values are assigned to the target objects with different reflection intensity, infrared intensity, or depth values.
The method according to claim 1, wherein:

The performing pixel-level processing on the current image according to the identification information includes:

Statistics on data information in the current image according to the identification information.
The method according to claim 1 or 27, wherein the identification information further comprises content information, which is used to indicate a content category of the target object.
The method according to claim 28, wherein:

The performing pixel-level processing on the current image according to the identification information includes:

According to the content category of the target object, statistics are performed on the target object in the current image to obtain a statistical result.
The method according to claim 1, 27, or 28, wherein the target object is a person,

The performing pixel-level processing on the current image according to the identification information includes:

When the first partial pixel in the pixel information corresponds to a third value, determining that the first partial pixel belongs to the head of the target object;

Performing facial expression recognition according to the head of the target object;

and / or,

The performing pixel-level processing on the current image according to the identification information includes:

When a second partial pixel in the pixel information corresponds to a fourth value, determining that the second partial pixel belongs to a hand of the target object;

Hand motion recognition is performed according to the hand of the target object.
The method according to claim 30, further comprising:

Send control instructions to the drone based on the results of facial expression recognition or hand motion recognition.
The method according to claim 1, 27, or 28, wherein the target object is a car,

The performing pixel-level processing on the current image according to the identification information includes:

When the first partial pixel in the pixel information corresponds to a fifth value, determining that the first partial pixel belongs to the head of the target object;

Determining the traveling direction of the target object according to the front of the target object;

and / or,

The performing pixel-level processing on the current image according to the identification information includes:

When the first partial pixel in the pixel information corresponds to a sixth value, determining that the second partial pixel belongs to the rear of the target object;

The traveling direction of the target object is determined according to the rear of the target object.
The method according to claim 29, wherein:

The statistic of the target object in the current image according to the content category of the target object to obtain a statistical result includes:

Statistic is performed on the target objects whose content category is a person in the current image to obtain the result of the flow of people and / or the result of the density of people.
The method according to claim 29, wherein:

The statistic of the target object in the current image according to the content category of the target object to obtain a statistical result includes:

Statistic is performed on the target object whose content category is a car in the current image to obtain a traffic flow result and / or a traffic flow density result.
The method according to claim 1 or 27, wherein the description feature corresponding to the at least one pixel comprises at least one of the following: a reflection intensity of a point cloud corresponding to the at least one pixel, the at least one pixel A corresponding infrared intensity and a depth value corresponding to the at least one pixel;

The performing pixel-level processing on the current image according to the identification information includes:

According to the description feature, statistics are performed on the target object in the current image to obtain a statistical result.
The method according to claim 27 or 35, wherein the method further comprises:

According to the statistical result, a heat map of the statistical result for the target object is generated.
The method according to any one of claims 1 to 36, wherein the attributes are measured in pixel blocks, the pixel information includes information on attributes of at least one pixel block, and the pixel blocks include at least two Pixels.
The method according to any one of claims 1 to 37, wherein the identification information is located in auxiliary enhancement information or extended data of the current image.
An image processing device, comprising:

At least one memory for storing computer-executable instructions;

At least one processor, individually or collectively, for accessing the at least one memory and executing the computer-executable instructions to perform the following operations:

Obtaining bitstream data of a current image, the bitstream data including identification information, the identification information being used to identify at least one target object in the current image, the identification information including image area information and pixel information, the The image region information includes a position and a size of an image region where the target object is located, and the pixel information includes attributes of at least one pixel in the image region;

Decoding the bitstream data to obtain the current image and the identification information;

Pixel level processing is performed on the current image according to the identification information.
The device according to claim 39, wherein:

The processor performing pixel-level processing on the current image according to the identification information includes:

Changing the display content of the current image according to the identification information.
The device according to claim 39 or 40, wherein the attribute of the at least one pixel includes whether the at least one pixel belongs to the target object.
The apparatus according to claim 41, wherein the processor is further configured to acquire a first image;

The processor performing pixel-level processing on the current image according to the identification information includes:

Fusion processing the current image and the first image based on the identification information to obtain a second image, where the second image includes at least part of the content of the current image and at least part of the content of the first image.
The apparatus according to claim 42, wherein the processor performs fusion processing on the current image and the first image based on the identification information, comprising:

Weighting and summing the current image and the first image based on the identification information, wherein a weight value of a pixel corresponding to the target object in the current image is equal to a weight value of a pixel other than the target object in the current image At least some pixels have different weighting values.
The apparatus according to claim 42 or 43, wherein the processor performs fusion processing on the current image and the first image based on the identification information, comprising:

Determine, according to the image area information and the pixel information, pixels in the current image that do not belong to the target object;

The pixels in the current image that do not belong to the target object are replaced with corresponding pixels in the first image to obtain a second image.
The device according to claim 44, wherein in the pixel information, the at least one pixel is assigned a different value,

The determining, by the processor according to the image area information and the pixel information, pixels in the current image that do not belong to the target object include:

Determining, according to the image region information, pixels outside the image region where the target object is located as pixels in the current image that do not belong to the target object;

When the first partial pixel in the pixel information corresponds to a first value, it is determined that the first partial pixel does not belong to the target object.
The apparatus according to claim 45, wherein before the processor determines pixels in the current image that do not belong to the target object according to the image area information and the pixel information, the processor Also used for:

Determining a boundary of the target object according to the pixel information, and performing an expansion operation on the target object based on the boundary;

The determining, by the processor according to the image area information and the pixel information, pixels in the current image that do not belong to the target object include:

Determining pixels in the current image that do not belong to the target object according to the image region information, the pixel information, and the boundary of the target object after expansion.
The device according to any one of claims 42 to 46, wherein a shooting angle of the current image is the same as a shooting angle of the first image.
The device according to claim 47, wherein the identification information further includes angle of view information, and the angle of view information is used to indicate a shooting angle of the target object relative to the shooting device;

Before the processor replaces pixels in the current image that do not belong to the target object with corresponding pixels in the first image, the processor is further configured to:

It is determined that the shooting angle of the target object is the same as the shooting angle of the first image.
The device according to claim 41, wherein:

The processor performing pixel-level processing on the current image according to the identification information includes:

Determining the target object in the current image according to the image area information and the pixel information, and adding an augmented reality AR special effect to the target object.
The device according to claim 49, wherein the identification information further includes perspective information, the perspective information is used to indicate a shooting angle of the target object relative to the shooting device, and the attribute of the at least one pixel further includes A depth value corresponding to the at least one pixel;

Determining, by the processor, the target object in the current image according to the image area information and the pixel information, and adding an AR special effect to the target object includes:

Determining the target object in the current image according to the image area information and the pixel information;

Adding an AR special effect to the target object according to the target object, the shooting angle, and a depth value corresponding to the at least one pixel.
The device according to claim 41, wherein:

The processor performing pixel-level processing on the current image according to the identification information includes:

Determining, according to the image area information and the pixel information, pixels in the current image that belong to the target object or pixels in the current image that do not belong to the target object;

Changing at least one of the brightness, color, and grayscale of the target object, or changing at least one of the brightness, color, and grayscale of the non-target object, or changing the target object and the non-target object Contrast.
The device according to claim 51, wherein:

The processor changing the brightness or color of the target object or changing the brightness or color of the non-target object includes:

By modifying the YUV value, RGB value, or γ curve of the target object or the non-target object, the brightness or color of the target object is changed, or the brightness or color of the non-target object is changed.
The device according to claim 39, 41, 51 or 52, wherein the identification information further includes content information, which is used to indicate a content category of the target object.
The device according to claim 53, wherein:

The processor changing at least one of brightness, color, and grayscale of the target object includes:

When the target object is a first content category, changing the brightness of the target object to a preset first brightness value, changing the color of the target object to a preset first color value, or changing the target object The grayscale is a preset first grayscale value.
The apparatus according to claim 53, wherein the current image includes a plurality of the target objects,

The processor changing at least one of brightness, color, and grayscale of the target object includes:

According to the content categories of a plurality of the target objects, different brightness values, color values, or grayscale values are assigned to the target objects of different content categories.
The device according to claim 55, wherein:

The processor performing pixel-level processing on the current image according to the identification information includes:

An object category segmented image is generated based on the current image based on the image area information, the pixel information, and the content category.
The device according to any one of claims 53 to 56, wherein the content information is a label or a value.
The device according to claim 39 or 40, wherein the attribute of the at least one pixel includes a part of the target object to which the at least one pixel belongs.
The device according to claim 39, 40 or 58, wherein in the pixel information, the at least one pixel is assigned a different value to indicate that the at least one pixel belongs to a different part of the target object.
The device according to claim 39, 40, 58 or 59, wherein:

The processor performing pixel-level processing on the current image according to the identification information includes:

According to the image area information and the pixel information, different parts of the target object are identified with different brightness, color, or grayscale, or different parts have different contrasts.
The device according to any one of claims 39 to 41, wherein the attribute of the at least one pixel includes a description feature corresponding to the at least one pixel.
The device according to claim 51, wherein the description feature corresponding to the at least one pixel comprises at least one of the following: a reflection intensity of a point cloud corresponding to the at least one pixel, and a The infrared intensity and a depth value corresponding to the at least one pixel.
The device according to claim 62, wherein:

The processor performs pixel-level processing on the current image according to the identification information, including at least one of the following:

Generating a reflection intensity segmentation image based on the current image according to the image area information, the pixel information, and the reflection intensity of a point cloud corresponding to the at least one pixel;

Generating an infrared image based on the current image according to the image area information, the pixel information, and an infrared intensity corresponding to the at least one pixel;

Generating a depth map based on the current image according to the image area information, the pixel information, and a depth value corresponding to the at least one pixel.
The apparatus according to claim 62, wherein the current image includes a plurality of the target objects,

The processor performing pixel-level processing on the current image according to the identification information includes:

According to the reflection intensity, infrared intensity, or depth value of a plurality of the target objects, different brightness values, color values, and grayscale values are assigned to the target objects with different reflection intensity, infrared intensity, or depth values.
The device according to claim 39, wherein:

The processor performing pixel-level processing on the current image according to the identification information includes:

Statistics on data information in the current image according to the identification information.
The device according to claim 39 or 65, wherein the identification information further comprises content information, which is used to indicate a content category of the target object.
The device according to claim 66, wherein

The processor performing pixel-level processing on the current image according to the identification information includes:

According to the content category of the target object, statistics are performed on the target object in the current image to obtain a statistical result.
The device according to claim 39, 65 or 66, wherein the target object is a person,

The processor performing pixel-level processing on the current image according to the identification information includes:

When the first partial pixel in the pixel information corresponds to a third value, determining that the first partial pixel belongs to the head of the target object;

Performing facial expression recognition according to the head of the target object;

and / or,

The processor performing pixel-level processing on the current image according to the identification information includes:

When the second part of the pixel information corresponds to a fourth value, determining that the second part of the pixel belongs to the hand of the target object;

Hand motion recognition is performed according to the hand of the target object.
The apparatus according to claim 68, wherein the processor is further configured to:

Send control instructions to the drone based on the results of facial expression recognition or hand motion recognition.
The device according to claim 39, 65 or 66, wherein the target object is a car,

The processor performing pixel-level processing on the current image according to the identification information includes:

When the first partial pixel in the pixel information corresponds to a fifth value, determining that the first partial pixel belongs to the head of the target object;

Determining the traveling direction of the target object according to the front of the target object;

and / or,

The processor performing pixel-level processing on the current image according to the identification information includes:

When the first partial pixel in the pixel information corresponds to a sixth value, determining that the second partial pixel belongs to the rear of the target object;

The traveling direction of the target object is determined according to the rear of the target object.
The device according to claim 67, wherein

According to the content category of the target object, the processor performs statistics on the target object in the current image, and obtains a statistical result, including:

Statistic is performed on the target objects whose content category is a person in the current image to obtain the result of the flow of people and / or the result of the density of people.
The device according to claim 67, wherein

According to the content category of the target object, the processor performs statistics on the target object in the current image, and obtains a statistical result, including:

Statistic is performed on the target object whose content category is a car in the current image to obtain a traffic flow result and / or a traffic flow density result.
The device according to claim 39 or 65, wherein the description feature corresponding to the at least one pixel comprises at least one of the following: a reflection intensity of a point cloud corresponding to the at least one pixel, the at least one pixel A corresponding infrared intensity and a depth value corresponding to the at least one pixel;

The processor performing pixel-level processing on the current image according to the identification information includes:

According to the description feature, statistics are performed on the target object in the current image to obtain a statistical result.
The apparatus according to claim 65 or 73, wherein the processor is further configured to:

According to the statistical result, a heat map of the statistical result for the target object is generated.
The device according to any one of claims 39 to 74, wherein the attributes are measured in pixel blocks, the pixel information includes attribute information of at least one pixel block, and the pixel block includes at least two Pixels.
The device according to any one of claims 39 to 75, wherein the identification information is located in auxiliary enhancement information or extended data of the current image.
A computer-readable storage medium, characterized in that instructions are stored thereon, and when the instructions are run on a computer, the computer is caused to execute the method according to any one of claims 1 to 38.