CN112040240A

CN112040240A - Data processing method, device and storage medium

Info

Publication number: CN112040240A
Application number: CN202011212659.5A
Authority: CN
Inventors: 李文广; 赵文军; 周长波
Original assignee: SZ DJI Technology Co Ltd
Current assignee: SZ DJI Technology Co Ltd
Priority date: 2020-11-03
Filing date: 2020-11-03
Publication date: 2020-12-04
Anticipated expiration: 2040-11-03
Also published as: CN112040240B

Abstract

The embodiment of the application provides a data processing method, equipment and a storage medium, to-be-processed image data are obtained from an original image, a first preset coding rule is used for coding base layer image data, a second preset coding rule is used for coding enhancement layer image data, so that the precision of the to-be-processed image data is reserved as far as possible, the dynamic range and the later editing space of the to-be-processed image data are guaranteed, and a final target image code stream is generated according to the base layer image code stream and the enhancement layer image code stream, so that the purpose of compatibility is achieved.

Description

Data processing method, device and storage medium

Technical Field

The present invention relates to the field of data processing, and in particular, to a data processing method, device, and storage medium.

Background

With the gradual increase of the hardware performance of the image sensor, the dynamic range of the actual scene that can be shot by the camera is larger and larger. Currently, the internet uses the most widely used mainstream image compression format, such as JPEG (Joint Photographic Experts Group, which is a lossy compression international standard for still images proposed by the Joint Photographic Experts Group), and only part of the dynamic range of the scene shot by the camera can be reserved due to the limitation of the data bit depth.

In recent years, some image compression techniques supporting High bit depth have been proposed in order to accommodate High Dynamic Range (HDR) images, but are not compatible with some mainstream image formats.

Disclosure of Invention

The invention provides a data processing method, equipment and a storage medium, which can be used for making a high dynamic range image compatible with an image in a mainstream image format.

A first aspect of the present invention is to provide a data processing method, including: acquiring original image data, and performing nonlinear image transformation processing on the original image data to obtain image data to be processed; extracting base layer image data with preset data volume from the image data to be processed or the original image data; coding the basic layer image data based on a first preset coding rule to obtain a basic layer image code stream; calculating a residual error based on the image data to be processed and the base layer image data, and determining enhancement layer image data based on the residual error; coding the enhancement layer image data based on a second preset coding rule to obtain an enhancement layer image code stream; and writing the enhancement layer image code stream into a preset field in the base layer image code stream to generate a target image code stream.

A second aspect of the present invention is to provide a data processing method, including: acquiring a target image code stream; and decoding the target image code stream based on a first preset decoding rule to obtain base layer image data.

A third aspect of the present invention is to provide an electronic apparatus, comprising: the device comprises a memory, a processor and an image acquisition device; the memory for storing a computer program; the image acquisition device is used for acquiring original image data; the processor calls the computer program to realize the following steps: acquiring original image data, and performing nonlinear image transformation processing on the original image data to obtain image data to be processed; extracting base layer image data with preset data volume from the image data to be processed or the original image data; coding the basic layer image data based on a first preset coding rule to obtain a basic layer image code stream; calculating a residual error based on the image data to be processed and the base layer image data, and determining enhancement layer image data based on the residual error; coding the enhancement layer image data based on a second preset coding rule to obtain an enhancement layer image code stream; and writing the enhancement layer image code stream into a preset field in the base layer image code stream to generate a target image code stream.

A fourth aspect of the present invention is to provide an electronic apparatus, comprising: a memory, a processor, and a communication component; the memory for storing a computer program; the communication component is used for acquiring a target image code stream; the processor calls the computer program to realize the following steps: and decoding the target image code stream based on a first preset decoding rule to obtain base layer image data.

A fifth aspect of the present invention is to provide a computer-readable storage medium, wherein the storage medium is a computer-readable storage medium, and program instructions are stored in the computer-readable storage medium, and the program instructions are used for the method according to the first aspect and the second aspect.

The embodiment of the application provides a data processing method, which comprises the steps of obtaining original image data, and carrying out nonlinear image transformation processing on the original image data to obtain image data to be processed; extracting base layer image data with preset data volume from the image data to be processed or the original image data; coding the basic layer image data based on a first preset coding rule to obtain a basic layer image code stream; calculating a residual error based on the image data to be processed and the base layer image data, and determining enhancement layer image data based on the residual error; coding the enhancement layer image data based on a second preset coding rule to obtain an enhancement layer image code stream; and writing the enhancement layer image code stream into a preset field in the base layer image code stream to generate a target image code stream.

The encoding of the image data to be processed is completed by encoding the image data of the basic layer and encoding the image data of the enhancement layer and generating a target image code stream, and meanwhile, the image data code stream of the enhancement layer and the image data code stream of the basic layer can be compatible.

In addition, since the original image data is subjected to the nonlinear image transformation processing to obtain the image data to be processed, the image data to be processed may be HDR image data, and then the HDR image data may be finally encoded. The base layer image data may belong to image data in a mainstream image format, such as image data in JPEG. Then writing the enhancement layer image code stream into a preset field in the base layer image code stream to generate a target image code stream, and then encoding HDR image data by using the image code stream in the main stream image format as a carrier, so that the image data in the main stream image format can be decoded and the HDR image data can also be decoded according to different requirements.

Based on this, the transmissibility of the HDR image can be greatly improved.

In addition, if the decoding end can decode all information in the target image code stream, an image with a high dynamic range close to the image to be processed can be obtained. The image data to be processed is close to the information content of the original image, so that the post-editing space is large. Providing more authoring space and freedom to the user.

Meanwhile, the image data to be processed (namely HDR image data) is obtained by carrying out nonlinear transformation on the original image, so that the image to be processed corresponding to the image data to be processed is more similar to the base layer image in vision than the original image, the appearance of the image to be processed is close to the appearance of the image to be processed when a user shoots, the unified and more consistent appearance is kept in image transmission, the later-stage editing operation of the user is facilitated, and the method is more friendly to the user.

In contrast, the method, the corresponding device and the storage medium provided by the embodiment of the present application based on the data processing method may also have the corresponding technical effects.

Drawings

The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this application, illustrate embodiment(s) of the application and together with the description serve to explain the application and not to limit the application. In the drawings:

fig. 1 is a schematic flow chart of a data processing method according to an embodiment of the present invention;

FIG. 2 is a diagram illustrating a data processing method according to an embodiment of the present invention;

fig. 3 is a schematic diagram of dividing into slices according to an embodiment of the present invention;

fig. 4 is a schematic diagram of slice division according to an embodiment of the present invention;

fig. 5 is a schematic diagram of a scan block according to an embodiment of the present invention;

fig. 6 is a schematic diagram of a scanning slice according to an embodiment of the present invention;

FIG. 7 is a diagram illustrating an enhancement layer image code stream according to an embodiment of the present invention;

FIG. 8 is a diagram illustrating header information of an enhancement layer image bitstream according to an embodiment of the present invention;

FIG. 9 is a diagram illustrating header information of an enhancement layer image bitstream according to an embodiment of the present invention;

fig. 10 is a flowchart illustrating a data processing method according to an embodiment of the present invention;

fig. 11 is a flowchart illustrating a data processing method according to an embodiment of the present invention;

fig. 12 is a schematic structural diagram of a data processing apparatus according to an embodiment of the present invention;

fig. 13 is a schematic structural diagram of a data processing apparatus according to an embodiment of the present invention;

fig. 14 is a schematic structural diagram of a data processing apparatus according to an embodiment of the present invention;

fig. 15 is a schematic structural diagram of an electronic device according to an embodiment of the present invention;

fig. 16 is a schematic structural diagram of an electronic device according to an embodiment of the present invention;

fig. 17 is a schematic structural diagram of an electronic device according to an embodiment of the present invention.

Detailed Description

In order to make the objects, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are some, but not all, embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.

Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. The terminology used in the description of the invention herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention.

In order to facilitate understanding of the technical solutions and technical effects of the present application, the following briefly describes the prior art:

based on the foregoing, as HDR images are increasing, the current compression method for HDR images is often to directly compress images with high bit depth, and thus, the HDR images are hardly widely supported by the market in the case that display devices with high bit depth are not popularized yet and are not compatible with the conventional image format, such as JPEG.

In addition, although the JPEG XT image compression standard can compress HDR images on the basis of being compatible with JPEG, the compression scheme is specific to the transmission end, and the decoded images are in a linear space, which is not friendly to users who want to perform post-editing on the decoded images. And because the images are compressed by 8-bit JPEG. The insufficient data precision influences the space of image post-editing.

Therefore, in order to solve the above problem, the embodiments of the present application propose a solution. The method comprises the steps of obtaining image data to be processed from an original image, coding the image data of a basic layer based on a first preset coding rule, and coding the image data of an enhancement layer based on a second preset coding rule, so that the precision of the image data to be processed is kept as much as possible, the dynamic range and the post-editing space of the image data to be processed are guaranteed, and a final target image code stream is generated according to the image code stream of the basic layer and the image code stream of the enhancement layer, so that the purpose of compatibility is achieved.

Some embodiments of the invention are described in detail below with reference to the accompanying drawings. The features of the embodiments and examples described below may be combined with each other without conflict between the embodiments.

Fig. 1 is a schematic flow chart of a data processing method according to an embodiment of the present invention; the method 100 provided by the embodiment of the application can be executed by an electronic device which can be provided with an encoder, the encoder can be arranged in the electronic device, the electronic device can be a camera, an unmanned aerial vehicle, a handheld cloud deck, a mobile phone, a tablet personal computer and the like, and the electronic device with the camera shooting and/or photographing functions can be used. The method 100 comprises the steps of:

101: and acquiring original image data, and performing nonlinear image transformation processing on the original image data to obtain image data to be processed.

102: and extracting the base layer image data of a preset data amount from the image data to be processed or the original image data.

103: and coding the base layer image data based on a first preset coding rule to obtain a base layer image code stream.

104: and calculating a residual error based on the image data to be processed and the base layer image data, and determining the enhancement layer image data based on the residual error.

105: and coding the image data of the enhancement layer based on a second preset coding rule to obtain an image code stream of the enhancement layer.

106: and writing the enhancement layer image code stream into a preset field in the base layer image code stream to generate a target image code stream.

In contrast, after the image data to be processed is encoded, the target image code stream may be decoded, where the decoding process may be an inverse process of encoding the image data to be processed, and may be implemented by an electronic device having a decoder, and the decoder may be simultaneously disposed in one electronic device with the encoder, or may be respectively and independently paired in different electronic devices.

The following is set forth in detail with respect to the above steps:

The original image data is image data with a high dynamic range. The raw image data includes linear image data or non-linear image data. The linear image data may be obtained image data acquired by an image sensor. The nonlinear image data may be high dynamic range image data generated based on a plurality of images, or may be obtained image data acquired by an image sensor.

The raw image data may refer to raw image data acquired from an image acquisition device in an electronic device, and the raw image data may be linear image data, and the image data may be linear HDR image data. The image acquisition device may be a camera, a video camera, or the like, and for example, the raw image data may be acquired by a mobile phone, and the raw image data may be 16bit linear image data.

After the original image data is acquired, the image data to be processed, which may be nonlinear image data, may be acquired from the original image data, and the size of the image data may be 12 bits. The acquisition mode can include the following modes:

for example, according to the above, the camera of the mobile phone captures an image of the external environment, which is 16bit linear image data. After the encoder in the mobile phone acquires the image data, 12-bit nonlinear image data can be obtained through nonlinear image transformation and used as intermediate layer image data and can be directly used as image data to be processed, and the image data to be processed can be nonlinear HDR image data.

It should be noted that, linear image transformation processing may also be performed on the original image data to obtain image data to be processed, and the specific implementation process is similar to that described above, and will not be described herein again. Only the description is as follows: for example, 16bit raw image data is subjected to linear image transform processing (e.g., linear cropping) to 15bit image data. Thus, the data size can be compressed, and the subsequent image processing is convenient.

Since the HDR image is encoded for the non-linear HDR image data and is obtained by non-linearly transforming/linearly transforming an original image, the HDR image is closer to a presentable image supported by a display device (i.e. a display device of an electronic device which may have a decoder) than the original image. For example, conventional display devices typically support lower bit depth images, such as 8bit images, for a non-linear transformation of a 16bit original image to a 12bit hdr image, the amount of data is not reduced much in practice. In the nonlinear transformation, detail information which is more sensitive to human eyes is reserved, and detail information which is not sensitive to human eyes is compressed more. And, in visual effect, the nonlinear 12-bit HDR image is closer to the 8-bit image, and the linear 16-bit original image is less close to the 8-bit image. Since the 12-bit HDR image is closer to the 8-bit image in visual effect, the 12-bit HDR image is more friendly to the user, and the 12-bit HDR image also retains the information of the high bit depth image, so that the user is also left with an editing space. The method is very friendly to users in visual effect and editing.

In order to be closer to the image displayed by the display device in visual effect or retain more information of the high bit depth image, the image data to be processed can be obtained by performing nonlinear transformation not only once. It is also possible to continue the non-linear transformation again on the basis of the non-linear transformation.

Specifically, the processing of the original image data by the nonlinear image conversion to obtain the image data to be processed includes: and carrying out at least one time of nonlinear image transformation processing on the original image data to obtain image data to be processed.

For example, as described above, after the encoder in the mobile phone obtains the 12-bit image data, the encoder may further perform nonlinear change to obtain 10-bit image data as the nonlinear HDR image data.

Correspondingly, the process of linear image transformation on the original image data to obtain the image data to be processed includes: and performing at least one linear image transformation process on the original image data to obtain image data to be processed.

And performing at least one time of nonlinear image conversion processing and at least one time of linear image conversion processing on the original image data to obtain image data to be processed. It should be understood that the order of processing for the nonlinear image conversion processing and the linear image conversion processing is not limited.

For example, as described above, the mobile phone continues to perform linear compression, such as linear cropping, on the basis of the 12-bit image data, so as to obtain 10-bit image data as the final nonlinear HDR image data. Or, as described above, the encoder in the mobile phone may perform nonlinear image conversion processing on 16-bit original image data to obtain 14-bit image data, and then perform linear cropping on the 14-bit image data once to obtain 12-bit image data. On the basis of the 12-bit image data, linear clipping is performed again to obtain 10-bit image data as final nonlinear HDR image data.

It should be noted that, for the original image, a larger image, such as 20bit original image data, etc., can be acquired. Then, as the original image data increases, the above transformation processes for the original image data may also increase, and regardless of the primary and secondary compression processes, it should be understood that the process of transforming the original image data for the first time may be a non-linear image transformation process, and then a plurality of times of linear image transformation processes are continued.

In contrast, the data size of the final image data to be processed may also be selected for different display devices, if the display device can support larger display image data, such as 10-bit image data, 12-bit image data, and the like. The number of times of transformation and the target of image transformation can be changed along with the transformation, but the specific implementation steps of the transformation are similar and are not described again.

In order to satisfy the user's autonomy, the user may also take the initiative in the above-described image transformation process.

Specifically, the processing of the original image data by the nonlinear image conversion to obtain the image data to be processed includes: determining a target of a non-linear image transformation of the original image data based on the image precision loss; and performing nonlinear image transformation processing on the original image based on the target of the nonlinear image transformation to obtain image data to be processed.

The loss of image precision refers to the precision lost by the image in the transformation process, and is determined according to the final transformation target of the image data. In the embodiment of the present application, the image precision loss is preset, and may be multiple, and each loss may correspond to a final transformation target, that is, a data amount of a final transformation of an image. For example, the image data is finally converted into 10-bit image data or 12-bit image data or 14-bit image data, or the like. And each transformation target can also correspond to a preset transformation process.

For example, as described above, the encoder in the mobile phone may display each transformation object in advance for the user to select through a display device, such as a screen of the mobile phone where the encoder is located. And an encoder in the mobile phone responds to the selection of a user, determines a final transformation target, transforms the original image through the process and the transformation target to obtain image data to be processed, and takes the image data as final nonlinear HDR image data.

It should be noted that, in addition to the above-described transformation target, the number of times of transformation may be set so that the transformation process thereof is determined according to the number of times of the transformation process. Since similar matters have been described above, they will not be described in detail here.

Further, for the linear image conversion processing, it is also possible to: the method for obtaining the image data to be processed by carrying out linear image transformation processing on the original image data comprises the following steps: determining a target of linear image transformation of the original image data based on the image precision loss; and performing linear image transformation processing on the original image based on the target of linear image transformation to obtain image data to be processed. And will not be described in detail herein.

In another embodiment, original image data is obtained, and image transformation processing is performed on the original image data to obtain image data to be processed. The image transformation process may include a linear transformation process, a non-linear transformation process, or a combination of a linear transformation process and a non-linear transformation process.

Further, the base layer image data refers to image data of a base data bit depth, i.e., low bit depth image data, and its dynamic range is also a base dynamic range. Such as 8bit image data. The base layer image data is compliant with the presentable image data of a conventional display device.

The method for extracting the base layer image data with the preset data size from the image data to be processed or the original image data comprises the following steps: and performing linear image transformation processing on the image data to be processed or the original image data to obtain the base layer image data with preset data volume.

The predetermined data amount may be determined according to the requirements of the electronic device associated with or located in the decoder, such as 8 bits.

For example, as described above, the encoder in the mobile phone may perform linear cropping after acquiring the 12-bit image data, and directly obtain 8-bit base layer image data after the linear cropping.

It should be noted that, the base layer image data of the preset data amount may also be obtained by performing linear image transformation processing on the original image data. It is not described here in detail, but it should be understood that the dynamic range of the base layer image data is smaller than that of the image data to be processed.

The first preset encoding rule is an encoding rule determined according to a JPEG encoding rule. In addition, the encoding rule may be determined according to an encoding rule of a current mainstream Image Format, such as TIFF (Tag Image File Format), BMP (Bitmap), and the like. The following can be described by taking JPEG as an example:

for example, as described above, as shown in fig. 2, an encoder in a mobile phone may obtain the base layer image data, and encode the base layer image data according to a JPEG encoding rule to obtain a base layer image code stream.

The residual refers to image data obtained by removing base layer image data from image data to be processed. The residual may be directly used as the enhancement layer image data.

The enhancement layer image data refers to image data between the high bit depth image data and the base layer image data. Its dynamic range is also high and is higher than that of the base layer image data.

In one embodiment, the residual may be calculated based on the image data to be processed extracted in step 101 and the base layer data extracted in step 102.

In another embodiment, the following may be specifically mentioned: calculating a residual based on the image data to be processed and the pre-processed base layer image data, comprising: and subtracting the preprocessed basic layer image data from the image data to be processed to obtain a residual error.

For example, as described above, after the encoder obtains the image data to be processed, the image data of 12 bits, and the image data of 8 bits of the base layer, the encoder may subtract the corresponding image data of 8 bits from the image data of 12 bits, so as to obtain the image data of the enhancement layer.

In addition, after the image data to be processed is encoded and decoded, as for the electronic device for decoding the target image code stream, it can only acquire decoded base layer image data (for example, corresponding to the code stream encoded by the JPEG encoding rule) from the decoded target image, and cannot acquire uncoded base layer image data, so that in order to better decode the target image subsequently, the decoded target image is acquired by the decoded enhancement layer image data and the decoded base layer image data (here, the base layer image compressed by the first image format or encoded), so that when the image data to be processed is encoded, the enhancement layer image data can be determined for the coded and decoded base layer image data.

In addition, when the base layer image data belongs to the coded and decoded base layer image data, the enhancement layer image data can be obtained from the image data to be processed and the coded and decoded base layer image data.

Specifically, the following may be used: the method 100 further comprises: preprocessing the base layer image data to obtain preprocessed base layer image data; calculating a residual based on the image data to be processed and the base layer image data, comprising: a residual is calculated based on the image data to be processed and the pre-processed base layer image data.

The method for preprocessing the base layer image data to obtain preprocessed base layer image data comprises the following steps: and decoding the base layer image code stream based on a first preset decoding rule corresponding to the first preset encoding rule, and taking the decoded data as preprocessed base layer image data.

For example, as shown in the foregoing, as shown in fig. 2, an encoder in a mobile phone acquires base layer image data, that is, 8-bit image data that is not coded and decoded, and then performs JPEG encoding on the base layer image data. Obtaining the encoded base layer image data, and then obtaining the decoded base layer image data through image decoding. Then, the corresponding decoded image data of the base layer is subtracted from the image data to be processed to obtain the image data of the enhancement layer.

It should be noted that, in JPEG, the implementation of the standard software and hardware algorithm is simple, and the quality of an image can be ensured under a higher compression multiple, so that the standard becomes an image compression standard which is widely used. Besides JPEG, other image format coding and decoding can be carried out, which image format is selected for coding and decoding, and when the whole image data to be processed is coded subsequently, the image in the image format can be compatible with HDR image.

In addition, in some regions of the relatively flat enhancement layer image, after the subsequent encoding process of the enhancement layer image data, such as DCT transform and quantization, only the dc component of the enhancement layer image data remains, thereby causing the final decoded target image, such as a 12-bit image, to exhibit blocking artifacts. In order to improve the quality of the final decoded target image. Random noise can be added on the basis of the obtained base layer image data (which can be the base layer image data or the preprocessed base layer image data) (which is equivalent to adding random noise on the enhancement layer image data because the enhancement layer image data is obtained according to the base layer image data added with the random noise), so that the blocking effect can be improved at the cost of increasing the code stream and partial noise.

Specifically, the method 100 may further include: adding random noise data to the base layer image data; calculating a residual based on the image data to be processed and the base layer image data, comprising: residual errors are calculated based on the image data to be processed and the base layer image data to which random noise data is added.

The random noise may be added to the base layer image data or the preprocessed base layer image data.

Since this situation occurs only where the image data is particularly dark, 0,1 random noise is added only where the gray scale value of the base layer image data or the preprocessed base layer image data is less than 3. The residual is then determined. And will not be described in detail herein.

Furthermore, the method 100 may further include: after the residual is calculated, random noise data is added to the residual.

In addition, the method 100 further includes: in the process of calculating the residual, random noise data is added so that the resulting residual has random noise data.

In order to determine the enhancement layer image data more conveniently, when calculating the residual, the dynamic range of the image data to be processed may be aligned with the dynamic range of the base layer image data (here and below, the base layer image may be the base layer image data, or may be the pre-processed base layer image data).

Specifically, the dynamic range represented by the image data to be processed is aligned with the dynamic range represented by the preprocessed base layer image data.

For example, the alignment may be: HDR/16, where HDR is image data to be processed, i.e. the above nonlinear HDR image data. When hdr is 12bit and the base layer image data is 8bit image data, hdr/16 can get an 8bit integer and 4bit decimal to align with the dynamic range of the base layer image data.

When determining the residual, if it is determined by the pre-processed base layer image data, the pre-processed base layer image data will tend to be lost, and in order to compensate for this loss, the loss can be compensated for.

Specifically, the aligned image data to be processed is compensated according to the ratio of the dynamic range of the image data of the enhancement layer to the dynamic range of the image data of the base layer; and subtracting the preprocessed base layer image data from the compensated image data to be processed to obtain a residual error.

For example, the HDR/16 data is compensated by compensating HDR/16 drccoef, where drccoef is an inverse of a ratio of a dynamic range of the base layer image data to a dynamic range of the HDR image (i.e., HDR) data. After compensation, the enhanced layer image data can be obtained by subtracting the preprocessed base layer image data.

Based on the foregoing discussion, wherein the calculating the residual based on the image data to be processed and the pre-processed base layer image data comprises: aligning the dynamic range represented by the image data to be processed with the dynamic range represented by the preprocessed base layer image data; compensating the aligned image data to be processed according to the proportion of the dynamic range of the image data of the enhancement layer to the dynamic range of the image data of the basic layer; and subtracting the preprocessed base layer image data from the compensated image data to be processed to obtain a residual error.

Since the foregoing has been set forth, further description is omitted herein.

In order to optimize the calculation mode, reduce the calculation amount, and improve the calculation efficiency and time, the data of the enhancement layer image may be selected.

Specifically, the method 100 may further include: determining enhancement layer image data based on the residual, comprising: obtaining data to be reserved based on the residual error; the retention data is selected from the data to be retained and is used as the enhancement layer image data.

The selecting the retained data refers to performing data retention or data selection on the obtained residual error.

The data retention may be selected, for example, by selecting (hdr/16 drccoef-base) 2n, where n is the number of bits of data desired to be retained. As can be seen from the foregoing, hdr/16 can be obtained as 4-digit decimal, so that n is 4 at the maximum and 1 at the minimum. Here n can be chosen to be 3, so three decimal places can be kept for a long time. (hdr/16 drccoef-base) is the residual. Random noise data may be added to the residual.

Since the calculation processes of the subsequent images are all calculating positive integers, the decimal is required to be converted into a positive integer.

Specifically, selecting reserved data from the data to be reserved and using the data as the image data of the enhancement layer includes: and adjusting the reserved data to obtain the enhanced layer image data with positive integer data.

For example, as previously described, it can be adjusted by the following equation (1):

（1）

where res is enhancement layer image data, it can be understood that the res is adjusted to a positive integer by 2048. Base may be data that preprocesses a pre-Base layer image.

Furthermore, it is also possible to directly: and adjusting the residual error to obtain the residual error with positive integer data, and directly using the residual error as the image data of the enhancement layer.

The second preset encoding rule refers to an encoding rule for the enhancement layer image data, and is encoded accordingly. The encoding process may include: as shown in fig. 2, the image data is divided into slices (slice image data) as the divided slice image data in fig. 2, DCT (Discrete Cosine Transform), quantization, and calculation of Qp map (quantization parameter table), i.e., quantization parameters, coefficient scanning, entropy coding, and the like in fig. 2. Image data is encoded by quantizing a DCT transform coefficient obtained by DCT transform based on the Qp map. Whereas Qp map may be determined from the base layer image data.

Based on this, encoding enhancement layer image data based on a second preset encoding rule includes: the enhancement layer image data is encoded based on the base layer image data.

In one embodiment, JPEG may be used for encoding for the second preset encoding rule. In addition, the encoding rules corresponding to encoding algorithms such as High bit depth JPEG, JPEG2000, and HEVC (High Efficiency Video Coding) may be used instead. Accordingly, the corresponding encoding process may be replaced by a process corresponding to an encoding algorithm such as high bit depth JPEG, JPEG2000, HEVC, etc.

Specifically, encoding enhancement layer image data based on base layer image data includes: determining first information reflecting the compression condition of the image data of the enhancement layer according to the image data of the base layer; based on the first information, the enhancement layer image data is encoded.

The first information is information reflecting the compression condition of the enhancement layer image, such as a Qp map, where the larger the Qp in the Qp map is, the more the enhancement layer image at the corresponding position is compressed, and the smaller the Qp is, the more complete the information stored in the enhancement layer image at the corresponding position is. It should be understood that the compression also refers to the image data information being removed.

Specifically, the number of the bright and/or dark portions of the base layer image data may be determined, and then the size of Qp may be determined by a number threshold. Thereby performing encoding.

Wherein, according to the image data of the base layer, determining first information reflecting the compression condition of the image data of the enhancement layer comprises the following steps: according to the brightness distribution condition of the image data of the base layer, first information reflecting the condition that the image data of the enhancement layer is compressed is determined.

The brightness distribution of the base layer image data refers to the distribution of bright (also referred to as highlight) and dark in the base layer image data. If the number of corresponding bright and/or dark places in a certain image data area is larger, the more complete it is saved, the smaller the Qp. Otherwise, Qp is larger.

More specifically, the number of highlight pixel points is determined according to the brightness condition in each preset image data area in the image; determining the number of dark pixel points according to the brightness condition in each preset image data area; and determining the corresponding brightness distribution condition according to the number of the high-brightness pixel points and/or the number of the dark pixel points.

The number of the dark pixel points and the number of the highlight pixel points can be determined through the color information of the corresponding image data.

Specifically, a YUV format of a base layer image is converted into an RGB format, and the number of highlight pixel points is determined according to the brightness condition in each preset image data area in the RGB format image; determining the number of dark pixel points according to the brightness condition in each preset image data area in the YUV format; and determining the corresponding brightness distribution condition according to the number of the high-brightness pixel points and/or the number of the dark pixel points.

For example, as described above, the encoder in the mobile phone first converts the input YUV (color coding method, "Y" represents brightness (Luma) and is also a gray scale value, and "U" and "V" represent Chroma (Chroma) and is used to describe the color and saturation of the image for specifying the color of the pixel) base layer image data into image data in RGB (RGB color mode) image format. The number sum of the pixel points at the highlight and the dark can be counted for each area based on the fact that the base layer image data is divided into a plurality of image data areas in advance. And determining according to the counted number sum of the pixels at the highlight and the dark in each region, if the number sum of the pixels at the highlight and the dark is more than 8% of the number of the pixels in the corresponding region (for example, if the number sum of the pixels in the region is 16x 32, the number sum of the pixels in the region is more than 16x 32 x 8%), setting the quantization parameter qIdx in the region to be 2, otherwise, setting the quantization parameter to be 8 until the quantization parameter is set in each region, and obtaining the Qp map for guiding the quantization process corresponding to the image data of the enhancement layer. It should be understood that each region shares the same set quantization parameter.

The division of the base layer image data into a plurality of regions may be performed in a manner of the above-described slice (slice image data, each 8 MBs is divided into one slice image data MB (Macroblock, MB size is 16 × 16 pixels)), or may be performed in a manner of dividing the base layer image data into 16 × 32 Quantization Blocks (QB), where the same Quantization parameter is used in the same QB and/or the same slice. It should be understood that the image data may be divided into regions according to the requirements, and is not limited herein.

For a quantized block, it may determine the first information as:

specifically, determining first information reflecting the compression condition of the image data of the enhancement layer according to the brightness distribution condition of the image data of the base layer comprises: dividing base layer image data into quantization blocks; determining corresponding brightness distribution conditions for each quantization block; and determining the quantization parameter of each quantization block according to the brightness distribution condition.

Since similar embodiments have been described above, they will not be described in detail here.

When aiming at slice, the determination of the brightness distribution condition may specifically be: for each slice image data, determining a corresponding brightness distribution, including: for each slice image data, determining a corresponding brightness distribution, including: determining the number of highlight pixel points according to the brightness condition in each slice image data; determining the number of dark pixel points according to the brightness condition in each slice image data; and determining the corresponding brightness distribution condition according to the number of the high-brightness pixel points and/or the number of the dark pixel points.

More specifically, determining the number of highlight pixel points according to the brightness condition in each slice image data includes: converting the YUV format of the image data of the basic layer into an RGB format, and determining the number of highlight pixel points according to the brightness condition in each slice image data in the RGB format; determining the number of dark pixel points according to the brightness condition in each slice image data, comprising: determining the number of dark pixel points according to the brightness condition in each slice image data in the YUV format; and determining the corresponding brightness distribution condition according to the number of the high-brightness pixel points and/or the number of the dark pixel points.

Since the foregoing has been set forth, further description is omitted herein.

For the case of according to the brightness distribution, the quantization parameter for determining slice may be: according to the brightness distribution condition of the image data of the base layer, determining first information reflecting the compression condition of the image data of the enhancement layer, comprising the following steps: dividing the base layer image data into slice image data; determining corresponding brightness distribution conditions for each slice image data; and determining the quantization parameter of each slice of image data according to the brightness distribution. Since the foregoing has been set forth, further description is omitted herein.

And counting the number of pixel points at the highlight position and/or the dark position in each area, wherein the pixel at the highlight position is determined if one channel value of the three RGB channels is more than 220. The pixel determination condition in the dark is that the value of the Y channel in the YUV is less than 64, and the pixel is in the dark. Thereby performing statistics.

It should be understood that, for any area, if only the dark pixels or the highlight pixels are counted, the determination may be made only according to the number of the dark pixels or the highlight pixels, and if the area has both the dark pixels and the highlight pixels, the determination may be made according to the sum of the number of the dark pixels and the highlight pixels. And will not be described in detail herein.

And for slice, the quantization parameter may be determined as:

specifically, determining the quantization parameter of each slice of image data according to the brightness distribution includes: and determining the quantization parameter of the corresponding slice image data according to the number of the high-brightness pixel points and/or the number of the dark pixel points in the brightness distribution condition.

Since the foregoing has been set forth, further description is omitted herein.

The above is a process of adaptively setting the quantization parameter according to the importance of the image data content. Considering that a user can make large adjustments to both ends of a dynamic range of image data, namely, a bright and dark portion of the image data at a post-editing end, in the embodiment of the present application, relatively small quantization parameters are used in the portions of the image data, so that the encoding quality of the portions of the image data is high, and a post-editing space of the user image is ensured.

Besides, the first information may be determined according to the luminance distribution of the HDR image data (e.g., the 12-bit image data to be processed described above), and the implementation process of the determination is similar to that described above, and thus is not described again. Only the description is as follows: and replacing the image data of the base layer with the image data to be processed.

In addition, the encoding process for the enhancement layer image data shown in fig. 2 can be implemented in a modular manner. For example, each sub-module is divided into slice image data, DCT (Discrete Cosine Transform), quantization, Qp map (quantization parameter table) calculation, coefficient scanning, entropy coding, and the like to implement its corresponding function. Each submodule can obtain an output result through input information. For the Qp map sub-module, the input can be the base layer image data or the image data to be processed, and the output is Qp map.

After the first information is determined, quantization may be performed.

Specifically, the encoding of the enhancement layer image data based on the first information includes: determining second information dividing image data frequencies of the enhanced layer image data; quantizing the second information based on the first information; and encoding the enhancement layer image data according to the quantization result.

Wherein the second information refers to information for dividing image data frequencies of the enhanced layer image data. Such as DCT transform coefficients, which divide the image data frequency of the enhancement layer image data into high and low frequencies. The low frequency is at the upper left corner of the DCT coefficient matrix, and the high frequency is at the lower right corner of the DCT coefficient matrix, so that subsequent scanning coding is facilitated, the high-frequency information of the image data is compressed, and the low-frequency information of the image data is reserved.

Wherein determining second information dividing image data frequencies of the enhanced layer image data may include:

the image data of the enhancement layer can be divided into slices, each slice is divided into block sub-blocks of 8x8 pixels, and each block is subjected to 2D-DCT (two-dimensional DCT) to obtain DCT (discrete cosine transform) coefficients of each block.

The block can be divided in a specific manner as follows:

dividing the image data of the enhancement layer to obtain a plurality of groups of macro blocks, and dividing the macro blocks into subblocks; and determining second information corresponding to the sub-blocks aiming at the sub-blocks.

The method comprises the steps of dividing image data of an enhancement layer to obtain a plurality of groups of macro blocks, and dividing the macro blocks into subblocks; based on the groups of macroblocks, slice image data is generated in combination.

For example, as described above, after the encoder in the mobile phone obtains the enhancement layer image data, the encoder may first divide the input enhancement layer image data into slices for subsequent smooth encoding. First, enhancement layer image data is divided into macroblocks MB, the size of which is 16x16 pixels. The MB may then be further divided into 4 blocks, each block having a size of 8 × 8 pixels, which may be the basic unit for subsequent processing.

When the resolution of the enhancement layer image data does not satisfy the multiple of 16, padding may be performed on the enhancement layer image data until the resolution satisfies the multiple of 16. Finally, every 8 MBs of the enhancement layer image data are divided into a set, called slice.

Preferably, MBs of a slice are located in the same MB line, and when the number of remaining MBs of a line is less than 8 MBs, the number of MBs in the slice may be 4, and when the number of remaining MBs of a line is less than 4 MBs, the number of MBs in the slice may be 2. And so on until there are only 1 MB in slice.

A schematic diagram of the division into slices is given in fig. 3. Wherein, each row has 8 slices, 19 slices rows in total, and 152 slices in total. Each slice can encode and decode in parallel. Wherein, slice301 has 8 MBs, slice302 has 4 MBs, slice303 has 2 MBs, and slice304 has one MB. The sub-module corresponding to the division may have an input of the enhancement layer image data and an output of the sub-module is the divided image data. In addition, the 2D-DCT transform calculation formula (2) is as follows: 2D-DCT transformation can be determined for each block to obtain a variation coefficient corresponding to each block.

（2）

Where S (u, v) is a DCT transform coefficient, x and y are pixels, 0-7, u is a frequency in a horizontal direction and v is a frequency in a vertical direction. s (x, y) are coefficient values in the spatial domain.

The input of the sub-module corresponding to the DCT transform may be the divided image data, and the output may be a DCT transform coefficient.

It is also worth mentioning that the 2D-DCT transform calculation can also be obtained by a 1D-DCT one-dimensional DCT transform, on the basis of which 2D-DCT transform coefficients are obtained. Specifically, the two-dimensional DCT transform may be decomposed into two times of one-dimensional DCT transform, where the one-dimensional DCT transform formula (3) is:

（3）

the formula may also determine 1D-DCT transformation for each block to obtain a variation coefficient corresponding to each block. X (k) is a one-dimensional DCT transform coefficient of variation, k is 0,1, 2 … … 7, x (i) is an image data signal corresponding to block, and i is a corresponding pixel.

Wherein, the transformation matrix can be planned as the following form:

wherein:

wherein, based on the first information, quantizing the second information may be: the method is realized by the following formula:

the quantization formula for the DC component is:

（4）

further, in some embodiments, one may use:

（5）

where 2^ (bitdepth +2), when bitdepth is 12, the value is 16384. The code can be directly brought into the value for processing in specific processing.

The quantization formula for the AC component is:

（6）

wherein, TF [ i ] [ j ] is DCT transform coefficient after DCT transform, bitdepth is bit width of DCT transform coefficient, QF [ i ] [ j ] is DCT transform coefficient after quantization, qMat [ i ] [ j ] is quantization matrix, which is preset. qIdx is a quantization parameter, obtained from the corresponding position of the input QP map, floor is a down-rounding function, and i, j represents the coordinates in 8x8 block.

For the DC component, since the value is larger than the AC component, in order to reduce the occupation of the AC component, save the storage resource and reduce the subsequent calculation amount, the DC component is reduced by adjusting 2^ (bitdepth +2) or 16384, i.e. TF [ i ] [ j ] -2^ (bitdepth +2) is used to prevent the DC component from being too large.

If the resources are sufficient, the subsequent calculation can be carried out quickly, or the adjustment is not needed, the corresponding item can be removed from the formula directly.

Each quantization may be performed in blocks of 8 by 8. In the embodiment of the present application, the dividing relationship among block, MB, QB, and slice is as shown in fig. 4. In fig. 4, each tile 401 represents a block of DCT transformed and quantized basic structure 8 × 8, each 4 blocks constitute an MB, and the order of arrangement of 4 blocks within 1 MB is as shown in fig. 4. Each 8 blocks constitute a QB (16 x 32), i.e. the same qIdx is used for quantization of each 8 blocks (i.e. the subblocks in the same slice image data share the same quantization parameter), which value is indexed from the Qp map. The whole figure 4 shows 16 × 8 as the size of one slice.

In addition, the texture boundaries of the image data require special consideration.

At this time, if a slice still uses the same qIdx, but since the slice is located at the boundary of the texture of the image data, it is necessary to use a smaller qIdx, for example 2, instead of the slice, which makes the code stream more wasteful, and it is necessary to use a larger qIdx, for example 8, instead of the slice, which makes the encoding quality of the partial image data worse. Therefore, such slice may be continuously divided into smaller blocks, such as QB, and then, the number information of the pixel points at the highlight and/or dark positions in each QB is counted again according to the above manner, and a corresponding qIdx is allocated to each QB, that is, at this time, one QB shares one quantization parameter, and each block is quantized to complete quantization.

It will be appreciated that the same quantisation parameter is shared for one QB. Then, as can be seen from the foregoing, starting with the partitioning of the enhancement layer image data, partitioning can be into QBs, each of which contains multiple blocks. And when DCT is carried out, block in the DCT is transformed. Then, quantization is performed for each block, and the quantization parameter used in quantization is one quantization parameter shared by one QB. Certainly, QB may also continue to be divided into smaller image data area units, and repeat the QP map and DCT transform to achieve final quantization, which is not described herein again.

It should be noted that the same quantization parameter may also be used for slice. That is, as can be seen from the foregoing, the quantization parameter corresponding to each slice is determined and quantized, and then only for the texture boundary of the image data, the same quantization parameter is shared by one QB, and quantization is performed. And will not be described in detail herein.

The input of the sub-module corresponding to the quantization function is the DCT transform coefficient and the QP map, and the output is the quantization coefficient.

After the quantization result is obtained, the quantization result, i.e., the quantization coefficient, may be scanned and encoded to generate a code stream.

Specifically, encoding the enhancement layer image data according to the quantization result includes: scanning the quantization result and encoding the scanned quantization result; obtaining the enhancement layer image code stream includes: and generating an enhancement layer image data code stream based on the encoding result.

Wherein, the scanning mode can be as follows: scanning the quantization results, including: scanning the quantization result in each sub-block according to a preset scanning mode to obtain a first scanned quantization result; and for the slice image data, sequentially scanning the quantization results in each sub-block in the slice image data according to the quantization result after the first scanning to obtain a quantization result after the second scanning, which is used as a final quantization result after the scanning.

The preset scanning mode may be a progressive scanning mode.

Before scanning, scanning is carried out based on subblocks in slice, so that image data of an enhancement layer is divided to obtain a plurality of groups of macro blocks, and the macro blocks are divided into subblocks; based on the groups of macroblocks, slice image data is generated in combination.

For example, as described above, as shown in fig. 5, the encoder in the mobile phone performs scanning in a progressive scanning manner, that is, each block is scanned separately, and a scanning result, that is, a reordered block result is obtained in fig. 5. In fig. 5, the left graph is a block before scanning, the right graph is a block scanned by a progressive scanning mode, and the arrow direction represents the scanning order. Then, after the 8x8 blocks are scanned, all the 8x8 blocks in the slice are scanned, frequency components at the same position in each block are scanned first, and then the next frequency component in each block is scanned, wherein the scanning sequence is from low frequency to high frequency. As shown in the direction of the arrows in fig. 6. In fig. 6, scanning is performed in order of frequency as scanning is started from the 0 position. In one embodiment, the frequency components may be swept from low to high. The frequency components at the same position, i.e. the small blocks (pixels) of the lowest frequency component, are scanned first, and then the small blocks (pixels) of the frequency components at the next position are scanned sequentially, i.e. the next frequency component is scanned until the end.

It should be noted that, for the sub-module corresponding to the scanning function, the input is the quantized coefficient, and the output is the quantized coefficient reordered after scanning.

After the scan is completed, the quantized coefficients reordered after the scan are encoded.

For example, an encoder in the handset entropy encodes the quantized coefficients reordered after scanning, as described above. In the entropy coding process of the embodiment of the present application, Rice-golomb (Rice-golomb coding, Rice-glomb coding) mixed entropy coding is adopted.

The input of the sub-module of the above entropy coding function is the quantized coefficients reordered after scanning, and the output is the binary sequence after entropy coding.

The preset fields can be selected according to different encoding rules. For example, for the coding rule corresponding to JPEG, it may be the position immediately following the APP0 application segment.

For example, as described above, the encoder in the mobile phone finds the position of the APP0 application segment (stream flag character FF E0) of the base layer image stream, then reads the length of the application segment, moves the pointer backward by the length, which is the insertion position of the enhancement layer image stream, inserts the enhancement layer image stream into the base layer image stream, and outputs the final image stream.

If the APP0 application segment does not exist in the base layer image code stream, the enhancement layer image code stream is inserted into the last or other self-defined position of the base layer image code stream.

As shown in fig. 7, the structure of the enhancement layer image code stream is organized. Wherein, the slice header (slice header information) stores qIdx values of different QBs in the slice, the qIdx of each QB is stored by 1bit, 1 represents that qIdx is 8, and 0 represents that qIdx is 2.

In addition, before the enhancement layer image code stream is inserted, the enhancement layer image code stream can be packaged. For example, an encoder in a mobile phone encapsulates an enhancement layer image code stream by using an APP7 (code stream flag character FF E7) application segment, and since JPEG specifies that the length of the application segment cannot exceed 0XFFFF, the enhancement layer image code stream needs to be cut and then encapsulated. The above encapsulation field is only an example, and when implemented, other APP fields may be used.

Specifically, the method 100 further includes: dividing the enhancement layer image code stream, and packaging the divided code stream; writing the enhancement layer image code stream into a preset field in the base layer image code stream, wherein the method comprises the following steps: and writing the packaged code stream into a preset field, wherein the packaged code stream has corresponding code stream identification information.

An encoder in the mobile phone divides the enhancement layer image code stream, packages the divided code stream, and inserts the packaged code stream into an adding position, wherein the enhancement layer image code stream and the divided code stream both have corresponding code stream identification information.

After the enhancement layer image code stream is cut into multiple segments, each segment needs a certain piece of header information for marking, the content of the header information is shown in fig. 8, the segment of header information is used for describing the total information of the enhancement layer image code stream, and the enhancement layer image code stream before being cut is only recorded once.

Fig. 9 shows header information of each enhancement layer image code stream after being cut, and each enhancement layer image code stream has the header information.

By arranging the enhancement layer image code stream in the base layer image code stream according to the embodiment of the application, the base layer image of the image can be obtained by using a traditional picture browser, such as a JPEG (joint photographic experts group) picture browser, without performing post-editing on the image or when browsing by using a low-bit-depth display, so that the spreading property of the HDR image is greatly improved.

When the image needs to be edited, the HDR image can be obtained by using a proprietary image browser, and the image is visually similar to a base layer image directly obtained by a traditional browser, such as a JPEG picture browser, so that the editing operation of a user is greatly simplified.

Because the enhancement layer image is encoded by using the encoding mode in the embodiment of the application, the HDR image after subsequent decoding has information approximate to the original image, so that a larger post-editing space is provided, and more authoring space and more degrees of freedom are provided for a user.

Specifically, if the decoding end can decode all information in the target image code stream, an image with a high dynamic range close to the image to be processed can be obtained. The image data to be processed is close to the information content of the original image, so that the post-editing space is large.

An application scenario of the embodiment of the present application is described below as an example: the user can take a picture through the mobile phone and take a next landscape picture. The landscape is an image of a 16bit linear space. The mobile phone sends the acquired 16-bit image to an encoder in the mobile phone, so that the encoder acquires the 16-bit image. The HDR image data in the non-linear space is derived from the 16-bit image data, either autonomously by the encoder or by user selection: 12bit image data as image data to be processed. The encoder obtains 8-bit image data from the 12-bit image data: base layer image data. And then compensating and adjusting the residual error between the 12-bit image data and the 8-bit image data to obtain the compensated and adjusted positive integer image data as the enhancement layer image data.

And the encoder encodes the basic layer image data according to the JPEG encoding rule to obtain a JPEG code stream. And meanwhile, encoding the enhancement layer image data by using the encoding rule of the enhancement layer image, wherein the enhancement layer image code stream is obtained. And packaging the enhancement layer image code stream into an APP7 application segment in the JPEG code stream to obtain the code stream of the HDR image data. The handset can store the code stream.

The user can choose to send the 16-bit image in the first user equipment to the second user equipment for post-editing. And the first user equipment sends the finally obtained code stream to the second user equipment. And the second user equipment decodes the code stream through a decoder arranged on the second user equipment.

When the second user equipment only supports a common JPEG decoder, only the base layer data can be decoded, the computer only acquires the JPEG code stream, and the JPEG code stream is decoded and displayed, and the displayed image is the base layer image data.

When the second user equipment has not only a common JPEG decoder but also a decoder with a decoding rule corresponding to the encoding rule of the enhancement layer, the second user equipment can take out the image data code stream of the enhancement layer from the final code stream, decode the image data code stream according to the corresponding decoding rule, finally obtain the decoded image data of the base layer and the decoded image data of the enhancement layer, and combine the decoded image data of the base layer and the decoded image data of the enhancement layer to obtain HDR image data for display so as to be used for a user to perform later editing.

Wherein, above-mentioned user equipment can be cell-phone, cloud platform camera, intelligent wearing equipment, unmanned aerial vehicle user equipment. The first device may be a device with a photographing function, and after photographing, the first device performs encoding processing based on the method shown in fig. 1 to obtain a target image code stream.

The target image code stream can be transmitted from the first user equipment to the second user equipment directly or through a plurality of pieces of equipment in a wired or wireless mode. For example, the first user device may transmit a file corresponding to the target image code stream to an internet server, and the second user device downloads the corresponding file from the internet server.

It should be noted that, the steps involved in the method 100 are not limited to the order, and may not be performed according to the sequence of the reference numbers and the writing order, and there may be a possibility of concurrent execution. Meanwhile, the method 100 may be performed by a plurality of processing components in the electronic device operating in parallel.

Based on the similar inventive concept, fig. 10 is a schematic flowchart of a data processing method according to an embodiment of the present invention; the method 1000 provided by the embodiment of the application can be executed by an electronic device with an encoder, the encoder can be arranged in the electronic device, the electronic device can be a camera, an unmanned aerial vehicle, a handheld cloud deck, a mobile phone, a tablet computer and the like, and the electronic device with the shooting or photographing function can be used. The method 1000 includes the steps of:

1001: and acquiring image data to be processed.

1002: and extracting base layer image data of a preset data amount from the image data to be processed.

1003: and calculating a residual error based on the image data to be processed and the base layer image data, and determining the enhancement layer image data.

1004: and coding the basic layer image data based on a first preset coding rule to obtain a basic layer image code stream.

1005: and coding the enhancement layer image data based on a second preset coding rule to obtain enhancement layer coded data.

1006: and writing the enhancement layer coded data into a preset field in the base layer image code stream to generate a target image code stream.

Since the detailed description of the steps 1001-1006 has already been provided above, the detailed description is omitted here.

In an embodiment, the image data to be processed may be obtained by performing a linear or non-linear transformation on the basis of the original image data.

In an embodiment, the image data to be processed is high dynamic range image data. The original image data to be processed comprises linear image data or nonlinear image data. The linear image data may be resultant image data acquired by an image sensor. The nonlinear image data may be high dynamic range image data generated based on a plurality of images, or may be obtained image data acquired by an image sensor.

Specifically, a plurality of images obtained by taking multiple exposures of a scene based on different shots may be collected by an image sensor in a short time, and the obtained plurality of images are combined into one image with a high dynamic range, which is used as the to-be-processed image data obtained in step 1001.

Since the foregoing has been set forth, further description is omitted herein.

In addition, the data volume of the image data to be processed is smaller than the original image data.

In addition, the first preset encoding rule is an encoding rule determined according to a JPEG encoding rule.

Wherein, 1002: extracting a preset amount of base layer image data from the image data to be processed may include: and carrying out linear transformation based on the image data to be processed to obtain the base layer image data with preset data volume.

Since the foregoing has been set forth, further description is omitted herein.

In addition, reference may be made to various steps in the method 100 described above for what the method 1000 may not describe in detail.

It should be noted that, the steps involved in the method 1000 are not limited in order, and may not be performed according to the sequence of the reference numbers and the writing order, and there may be a possibility of concurrent execution. Meanwhile, the method 1000 may be performed by a plurality of processing components in the electronic device working in parallel.

Based on the similar inventive concept, fig. 11 is a schematic flow chart of a data processing method according to an embodiment of the present invention; the method 1100 provided by the embodiment of the application can be executed by electronic equipment with a decoder, the decoder can be arranged in the electronic equipment, and the electronic equipment can be cameras, unmanned planes, handheld cloud platforms, mobile phones, computers and the like, and can be electronic equipment with camera shooting or photographing functions. The method 1100 includes the steps of:

1101: and acquiring a target image code stream.

1102: and decoding the target image code stream based on a first preset decoding rule to obtain base layer image data.

The first predetermined decoding rule corresponds to the first predetermined encoding rule, i.e., the first predetermined encoding rule is set as an encoding rule determined according to the JPEG encoding rule, and the first predetermined decoding rule may be a decoding rule determined according to the JPEG decoding rule.

For example, as described above, the computer receives the target image code stream sent by the user's mobile phone. And a decoder in the computer acquires the JPEG code stream in the code stream, and then decodes the JPEG code stream according to the decoding rule determined by the JPEG decoding rule to obtain the image data of the basic layer for display. It should be understood that, although the target image code stream may be in the form of a JPEG code stream, since an enhancement layer image data code stream is also included therein, the enhancement layer image data code stream is included in the target image code stream in addition to the JPEG code stream.

Further, the base layer image data is used to generate a base layer image.

For example, as described above, after the decoder in the computer acquires the base layer image data, the base layer image in the JPEG image format may be directly generated according to the data, and may be displayed on the display screen of the computer.

In addition, the method 1100 further includes: decoding the target image code stream based on a second preset decoding rule to obtain enhancement layer image data; the enhancement layer image data is used to generate a target image in cooperation with the base layer image data, and the dynamic range of the target image is larger than that of the base layer image.

The second predetermined decoding rule corresponds to the second predetermined encoding rule, i.e., the second predetermined encoding rule set above. The second preset decoding rule may be an inverse process of the second preset encoding rule.

For example, as described above, the decoder in the computer may also obtain the enhancement layer image data stream from the APP7 application segment in the target image stream. The segmented code stream may be spliced into a complete enhancement layer image data code stream. After the enhancement layer image data code stream is obtained, the decoder may perform decoding according to the inverse process of the enhancement layer image data coding rule (i.e., the second preset coding rule) described above, so as to obtain the enhancement layer image data. And finally combining the enhancement layer image data and the base layer image data to obtain final HDR image data, and generating and displaying the HDR image.

In addition, reference may be made to the steps of the method 100 described above for details of the method 1100.

It should be noted that the steps involved in the method 1100 are not limited in order, and may not be performed according to the order of reference numbers and writing order, and there may be a possibility of concurrent execution. Meanwhile, the method 1100 may be performed by a plurality of processing components in the electronic device operating in parallel.

Fig. 12 is a schematic structural diagram of a data processing apparatus according to an embodiment of the present invention; the apparatus 1200 may be applied to an electronic device that may have an encoder, for example, a camera, an unmanned aerial vehicle, a handheld cradle head, a mobile phone, a tablet computer, and the like. The apparatus 1200 includes: a transformation module 1201, an extraction module 1202, a first encoding module 1203, a determination module 1204, a second encoding module 1205, and a generation module 1206. The following detailed description is directed to the functions of the various modules:

the transformation module 1201 is configured to obtain original image data, and perform nonlinear image transformation on the original image data to obtain image data to be processed.

An extracting module 1202, configured to extract base layer image data of a preset data amount from image data to be processed or original image data.

The first encoding module 1203 is configured to encode the base layer image data based on a first preset encoding rule, so as to obtain a base layer image code stream.

A determining module 1204, configured to calculate a residual based on the image data to be processed and the base layer image data, and determine the enhancement layer image data based on the residual.

The second encoding module 1205 is configured to encode the enhancement layer image data based on a second preset encoding rule to obtain an enhancement layer image code stream.

The generating module 1206 is configured to write the enhancement layer image code stream into a preset field in the base layer image code stream to generate a target image code stream.

Wherein the original image data is image data with a high dynamic range. The raw image data includes linear image data or non-linear image data. The linear image data may be resultant image data acquired by an image sensor. The nonlinear image data may be high dynamic range image data generated based on a plurality of images, or may be obtained image data acquired by an image sensor.

Furthermore, the transformation module 1201 is further configured to: and performing linear image transformation processing on the original image data to obtain image data to be processed.

Specifically, the transformation module 1201 is specifically configured to: and carrying out at least one time of nonlinear image transformation processing on the original image data to obtain image data to be processed.

Specifically, the transformation module 1201 includes: a determination unit configured to determine a target of a nonlinear image transformation of the original image data based on the image accuracy loss; and the transformation unit is used for carrying out nonlinear image transformation processing on the original image based on the target of the nonlinear image transformation to obtain image data to be processed.

An extraction module 1202 to: and performing linear image transformation processing on the image data to be processed or the original image data to obtain the base layer image data with preset data volume.

Further, the apparatus 1200: the preprocessing module is used for preprocessing the basic layer image data to obtain preprocessed basic layer image data; a determining module 1204 for: a residual is calculated based on the image data to be processed and the pre-processed base layer image data.

Specifically, the preprocessing module is configured to: and decoding the base layer image code stream based on a first preset decoding rule corresponding to the first preset encoding rule, and using the decoded data as the preprocessed base layer image data.

Specifically, the determining module 1204 is configured to: and subtracting the preprocessed basic layer image data from the image data to be processed to obtain a residual error.

Specifically, the determining module 1204 is configured to: obtaining data to be reserved based on the residual error; and selecting reserved data from the data to be reserved as the enhancement layer image data.

Specifically, the determining module 1204 is configured to: and adjusting the reserved data to obtain the enhanced layer image data with positive integer data.

Specifically, the determining module 1204 includes: the alignment unit is used for aligning the dynamic range represented by the image data to be processed with the dynamic range represented by the preprocessed basic layer image data; the compensation unit is used for compensating the aligned image data to be processed according to the proportion of the dynamic range of the image data of the enhancement layer to the dynamic range of the image data of the basic layer; and the first determining unit is used for subtracting the preprocessed base layer image data from the compensated image data to be processed to obtain a residual error.

In addition, the apparatus 1200 further comprises: the adding module is used for adding random noise data into the base layer image data; a determining module 1204 for: residual errors are calculated based on the image data to be processed and the base layer image data to which random noise data is added.

And the adding module is also used for adding random noise data in the process of calculating the residual error so as to enable the obtained residual error to have the random noise data.

And the adding module is also used for adding random noise data into the residual after the residual is calculated.

The first preset encoding rule is an encoding rule determined according to a JPEG encoding rule.

In particular, the second encoding module 1205 is configured to encode the enhancement layer image data based on the base layer image data.

Specifically, the second encoding module 1205 is configured to determine, according to the base layer image data, first information reflecting a situation that the enhancement layer image data is compressed; based on the first information, the enhancement layer image data is encoded.

Specifically, the second encoding module 1205 is configured to determine the first information reflecting the compression condition of the enhancement layer image data according to the luminance distribution condition of the base layer image data.

Specifically, the second encoding module 1205 includes: a second determining unit for determining second information of dividing an image data frequency of the enhanced layer image data; a quantization unit configured to quantize the second information based on the first information; and the coding unit is used for coding the image data of the enhancement layer according to the quantization result.

Specifically, the encoding unit is configured to scan a quantization result and encode the scanned quantization result; obtaining the enhancement layer image code stream includes: and generating an enhancement layer image data code stream based on the encoding result.

In addition, the apparatus 1200 further comprises: the dividing module is used for dividing the image data of the enhancement layer to obtain a plurality of groups of macro blocks and dividing the macro blocks into subblocks; a determining module 1204, further configured to: and determining second information corresponding to the sub-blocks aiming at the sub-blocks.

In addition, the dividing module is further configured to divide the image data of the enhancement layer to obtain a plurality of groups of macroblocks, and divide the macroblocks into subblocks; the apparatus 1200 further comprises: the combination module is used for combining and generating slice image data based on a plurality of groups of macro blocks; wherein the encoding unit is configured to: scanning the quantization result in each sub-block according to a preset scanning mode to obtain a first scanned quantization result; and for the slice image data, sequentially scanning the quantization results in each sub-block in the slice image data according to the quantization result after the first scanning to obtain a quantization result after the second scanning, which is used as a final quantization result after the scanning.

Specifically, the second encoding module 1205 includes: a dividing unit configured to divide the base layer image data into slice image data; a third determining unit, configured to determine, for each slice image data, a corresponding luminance distribution; and determining the quantization parameter of each slice of image data according to the brightness distribution.

Specifically, the second encoding module 1205 includes: a dividing unit further configured to divide the base layer image data into quantization blocks; a third determining unit, configured to determine, for each quantized block, a corresponding luminance distribution; and determining the quantization parameter of each quantization block according to the brightness distribution condition.

Specifically, the third determining unit is configured to: determining the number of highlight pixel points according to the brightness condition in each slice image data; determining the number of dark pixel points according to the brightness condition in each slice image data; and determining the corresponding brightness distribution condition according to the number of the high-brightness pixel points and/or the number of the dark pixel points.

Specifically, the third determining unit is configured to: converting the YUV format of the image data of the basic layer into an RGB format, and determining the number of highlight pixel points according to the brightness condition in each slice image data in the RGB format; determining the number of dark pixel points according to the brightness condition in each slice image data, comprising: determining the number of dark pixel points according to the brightness condition in each slice image data in the YUV format; and determining the corresponding brightness distribution condition according to the number of the high-brightness pixel points and/or the number of the dark pixel points.

Specifically, the third determining unit is configured to: and determining the quantization parameter of the corresponding slice image data according to the number of the high-brightness pixel points and/or the number of the dark pixel points in the brightness distribution condition.

Wherein the subblocks in the same slice image data share the same quantization parameter.

In addition, the apparatus 1200 further comprises: the segmentation module is used for segmenting the enhancement layer image code stream and packaging the segmented code stream; a generating module 1206 for: and writing the packaged code stream into a preset field, wherein the packaged code stream has corresponding code stream identification information.

Fig. 13 is a schematic structural diagram of a data processing apparatus according to an embodiment of the present invention; the apparatus 1300 may be applied to an electronic device that may have an encoder, for example, a camera, an unmanned aerial vehicle, a handheld cradle head, a mobile phone, a tablet computer, and the like. The apparatus 1300 includes: an obtaining module 1301, an extracting module 1302, a determining module 1303, a first encoding module 1304, a second encoding module 1305, and a generating module 1306. The following detailed description is directed to the functions of the various modules:

an obtaining module 1301, configured to obtain image data to be processed.

An extracting module 1302, configured to extract base layer image data of a preset data amount from the image data to be processed.

A determining module 1303, configured to calculate a residual based on the image data to be processed and the base layer image data, and determine enhancement layer image data.

And a first encoding module 1304, configured to encode the base layer image data based on a first preset encoding rule, so as to obtain a base layer image code stream.

The second encoding module 1305 is configured to encode the enhancement layer image data based on a second preset encoding rule to obtain enhancement layer encoded data.

A generating module 1306, configured to write the enhancement layer encoded data into a preset field in the base layer image code stream, so as to generate a target image code stream.

The image data to be processed is obtained by performing linear or nonlinear transformation on the basis of original image data.

Wherein, the extracting module 1302 is configured to: and carrying out linear transformation based on the image data to be processed to obtain the base layer image data with preset data volume.

It should be noted that, for parts not described in detail in this embodiment, reference may be made to the related description of the embodiment shown in the data processing apparatus 1200, and details are not described herein again.

Fig. 14 is a schematic structural diagram of a data processing apparatus according to an embodiment of the present invention; the apparatus 1400 may be applied to an electronic device that may have a decoder, for example, a camera, an unmanned aerial vehicle, a handheld cradle head, a mobile phone, a tablet computer, and the like. The apparatus 1400 comprises: an obtaining module 1401 and a first decoding module 1402. The following detailed description is directed to the functions of the various modules:

and an obtaining module 1401, configured to obtain a target image code stream.

The first decoding module 1402 is configured to decode the target image code stream based on a first preset decoding rule to obtain base layer image data.

Wherein the base layer image data is used to generate a base layer image.

In addition, the apparatus 1400 further comprises: the second decoding module is used for decoding the target image code stream based on a second preset decoding rule to obtain enhancement layer image data; the enhancement layer image data is used to generate a target image in cooperation with the base layer image data, and the dynamic range of the target image is larger than that of the base layer image.

In one possible design, the structure of the data processing apparatus 1200 shown in fig. 12 may be implemented as an electronic device, which may be various electronic devices with a photographing or shooting function, such as a camera, an unmanned aerial vehicle, a handheld cradle head, a mobile phone, a tablet computer, and the like. As shown in fig. 15, the electronic device 1500 may include: one or more processors 1501, one or more memories 1502, and an image acquisition device 1503. The memory 1502 is used for storing programs for supporting the electronic device to execute the data processing method provided in the embodiments shown in fig. 1-9. The image acquisition device 1503 is used for acquiring original image data; the processor 1501 is configured to execute programs stored in the memory 1502. In particular, the program comprises one or more computer instructions, wherein the one or more computer instructions, when executed by the processor 1501, enable the following steps to be performed:

the processor 1501 runs the computer program stored in the memory 1502 to implement:

carrying out nonlinear image transformation processing on the original image data to obtain image data to be processed; extracting base layer image data with preset data volume from image data to be processed or original image data; coding the base layer image data based on a first preset coding rule to obtain a base layer image code stream; calculating a residual error based on the image data to be processed and the base layer image data, and determining enhancement layer image data based on the residual error; encoding the image data of the enhancement layer based on a second preset encoding rule to obtain an image code stream of the enhancement layer; and writing the enhancement layer image code stream into a preset field in the base layer image code stream to generate a target image code stream.

Specifically, the processor 1501 is specifically configured to: and carrying out at least one time of nonlinear image transformation processing on the original image data to obtain image data to be processed.

Further, the processor 1501 is further configured to: and performing linear image transformation processing on the original image data to obtain image data to be processed.

Specifically, the processor 1501 is specifically configured to: determining a target of a non-linear image transformation of the original image data based on the image precision loss; and performing nonlinear image transformation processing on the original image based on the target of the nonlinear image transformation to obtain image data to be processed.

Specifically, the processor 1501 is specifically configured to: and performing linear image transformation processing on the image data to be processed or the original image data to obtain the base layer image data with preset data volume.

Specifically, the processor 1501 is specifically configured to: preprocessing the base layer image data to obtain preprocessed base layer image data; calculating a residual based on the image data to be processed and the base layer image data, comprising: the residual is calculated based on the image data to be processed and the pre-processed base layer image data.

Specifically, the processor 1501 is specifically configured to: and decoding the base layer image code stream based on a first preset decoding rule corresponding to the first preset encoding rule, and taking the decoded data as preprocessed base layer image data.

Specifically, the processor 1501 is specifically configured to: and subtracting the preprocessed basic layer image data from the image data to be processed to obtain the residual error.

Specifically, the processor 1501 is specifically configured to: obtaining data to be reserved based on the residual error; the retention data is selected from the data to be retained and is used as the enhancement layer image data.

Specifically, the processor 1501 is specifically configured to: and adjusting the reserved data to obtain the enhanced layer image data with positive integer data.

Specifically, the processor 1501 is specifically configured to: aligning the dynamic range represented by the image data to be processed with the dynamic range represented by the preprocessed base layer image data; compensating the aligned image data to be processed according to the proportion of the dynamic range of the image data of the enhancement layer to the dynamic range of the image data of the basic layer; and subtracting the preprocessed base layer image data from the compensated image data to be processed to obtain a residual error.

Further, the processor 1501 is further configured to: adding random noise data to the base layer image data; the processor 1501 is specifically configured to: residual errors are calculated based on the image data to be processed and the base layer image data to which random noise data is added.

Further, the processor 1501 is further configured to: in the process of calculating the residual, random noise data is added so that the resulting residual has random noise data.

Specifically, the processor 1501 is specifically configured to: after the residual is calculated, random noise data is added to the residual.

Specifically, the processor 1501 is specifically configured to: the enhancement layer image data is encoded based on the base layer image data.

Specifically, the processor 1501 is specifically configured to: determining first information reflecting the compression condition of the image data of the enhancement layer according to the image data of the base layer; based on the first information, the enhancement layer image data is encoded.

Specifically, the processor 1501 is specifically configured to: according to the brightness distribution condition of the image data of the base layer, first information reflecting the condition that the image data of the enhancement layer is compressed is determined.

Specifically, the processor 1501 is specifically configured to: determining second information dividing image data frequencies of the enhanced layer image data; quantizing the second information based on the first information; and encoding the enhancement layer image data according to the quantization result.

Specifically, the processor 1501 is specifically configured to: scanning the quantization result and encoding the scanned quantization result; obtaining the enhancement layer image code stream includes: and generating the enhancement layer image data code stream based on the encoding result.

Further, the processor 1501 is further configured to: dividing the image data of the enhancement layer to obtain a plurality of groups of macro blocks, and dividing the macro blocks into subblocks; and determining second information corresponding to the sub-blocks aiming at the sub-blocks.

Further, the processor 1501 is further configured to: dividing the image data of the enhancement layer to obtain a plurality of groups of macro blocks, and dividing the macro blocks into subblocks; generating slice image data based on the groups of macro blocks in a combined manner; the processor 1501 is specifically configured to: scanning the quantization result in each sub-block according to a preset scanning mode to obtain a first scanned quantization result; and for the slice image data, sequentially scanning the quantization results in each sub-block in the slice image data according to the quantization result after the first scanning to obtain a quantization result after the second scanning, which is used as a final quantization result after the scanning.

Specifically, the processor 1501 is specifically configured to: dividing the base layer image data into slice image data; determining corresponding brightness distribution conditions for each slice image data; and determining the quantization parameter of each slice of image data according to the brightness distribution.

Specifically, the processor 1501 is specifically configured to: dividing base layer image data into quantization blocks; determining corresponding brightness distribution conditions for each quantization block; and determining the quantization parameter of each quantization block according to the brightness distribution condition.

Specifically, the processor 1501 is specifically configured to: determining the number of highlight pixel points according to the brightness condition in each slice image data; determining the number of dark pixel points according to the brightness condition in each slice image data; and determining the corresponding brightness distribution condition according to the number of the high-brightness pixel points and/or the number of the dark pixel points.

Specifically, the processor 1501 is specifically configured to: converting the YUV format of the image data of the basic layer into an RGB format, and determining the number of highlight pixel points according to the brightness condition in each slice image data in the RGB format; determining the number of dark pixel points according to the brightness condition in each slice image data, comprising: determining the number of dark pixel points according to the brightness condition in each slice image data in the YUV format; and determining the corresponding brightness distribution condition according to the number of the high-brightness pixel points and/or the number of the dark pixel points.

Specifically, the processor 1501 is specifically configured to: and determining the quantization parameter of the corresponding slice image data according to the number of the high-brightness pixel points and/or the number of the dark pixel points in the brightness distribution condition.

Further, the processor 1501 is further configured to: dividing the enhancement layer image code stream, and packaging the divided code stream; writing the enhancement layer image code stream into a preset field in the base layer image code stream, wherein the method comprises the following steps: and writing the packaged code stream into a preset field, wherein the packaged code stream has corresponding code stream identification information.

In addition, an embodiment of the present invention provides a computer-readable storage medium, where the storage medium is a computer-readable storage medium, and program instructions are stored in the computer-readable storage medium, where the program instructions are used to implement the methods in fig. 1 to 9.

In one possible design, the structure of the data processing apparatus 1300 shown in fig. 13 may be implemented as an electronic device, which may be various electronic devices with a photographing or shooting function, such as a camera, an unmanned aerial vehicle, a handheld cradle head, a mobile phone, a tablet computer, and the like. As shown in fig. 16, the electronic device 1600 may include: one or more processors 1601, one or more memories 1602. The memory 1602 is used for storing a program for supporting the electronic device to execute the data processing method provided in the embodiment shown in fig. 10. Wherein the processor 1601 is configured to execute programs stored in the memory 1602. In particular, the program includes one or more computer instructions, where the one or more computer instructions, when executed by the processor 1601, enable the following steps to be performed:

the processor 1601 runs a computer program stored in the memory 1602 to implement: acquiring image data to be processed; extracting base layer image data with preset data volume from the image data to be processed; calculating a residual error based on the image data to be processed and the base layer image data, and determining enhancement layer image data; coding the basic layer image data based on a first preset coding rule to obtain a basic layer image code stream; encoding the enhancement layer image data based on a second preset encoding rule to obtain enhancement layer encoded data; and writing the enhancement layer coded data into a preset field in the base layer image code stream to generate a target image code stream.

The processor 1601 is specifically configured to: and carrying out linear transformation based on the image data to be processed to obtain the base layer image data with preset data volume.

It should be noted that, for parts not described in detail in this embodiment, reference may be made to the related description of the embodiment shown in the electronic device 1500, and details are not described herein again.

In addition, an embodiment of the present invention provides a computer-readable storage medium, where the storage medium is a computer-readable storage medium, and program instructions are stored in the computer-readable storage medium, where the program instructions are used to implement the method in fig. 10.

In one possible design, the structure of the data processing apparatus 1400 shown in fig. 14 may be implemented as an electronic device, which may be various electronic devices with a photographing or shooting function, such as a camera, an unmanned aerial vehicle, a handheld cradle head, a mobile phone, a tablet computer, and so on. As shown in fig. 17, the electronic device 1700 may include: one or more processors 1701, one or more memories 1702, and a communications component 1703. The memory 1702 is used for storing a program that supports the electronic device to execute the data processing method provided in the embodiment shown in fig. 11. The communication component 1703 is configured to obtain a target image code stream; the processor 1701 is configured to execute programs stored in the memory 1702. In particular, the program includes one or more computer instructions which, when executed by the processor 1701, enable the following steps to be performed:

the processor 1701 runs the computer program stored in the memory 1702 to implement:

and decoding the target image code stream based on a first preset decoding rule to obtain base layer image data.

Wherein the base layer image data is used to generate a base layer image.

Further, the processor 1701 is further configured to: decoding the target image code stream based on a second preset decoding rule to obtain enhancement layer image data; the enhancement layer image data is used to generate a target image in cooperation with the base layer image data, and the dynamic range of the target image is larger than that of the base layer image.

In addition, an embodiment of the present invention provides a computer-readable storage medium, where the storage medium is a computer-readable storage medium, and program instructions are stored in the computer-readable storage medium, where the program instructions are used to implement the method in fig. 11.

The technical solutions and the technical features in the above embodiments may be used alone or in combination in case of conflict with the present disclosure, and all embodiments that fall within the scope of protection of the present disclosure are intended to be equivalent embodiments as long as they do not exceed the scope of recognition of those skilled in the art.

In the embodiments provided in the present invention, it should be understood that the disclosed correlation detection apparatus (e.g., IMU) and method may be implemented in other ways. For example, the above-described remote control device embodiments are merely illustrative, and for example, the division of the modules or units is only one logical division, and other divisions may be realized in practice, for example, a plurality of units or components may be combined or integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, remote control devices or units, and may be in an electrical, mechanical or other form.

The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.

In addition, functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit can be realized in a form of hardware, and can also be realized in a form of a software functional unit.

The integrated unit, if implemented in the form of a software functional unit and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present invention may be embodied in the form of a software product, which is stored in a storage medium and includes instructions for causing a computer processor (processor) to execute all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and other various media capable of storing program codes.

The above description is only an embodiment of the present invention, and not intended to limit the scope of the present invention, and all modifications of equivalent structures and equivalent processes performed by the present specification and drawings, or directly or indirectly applied to other related technical fields, are included in the scope of the present invention.

Finally, it should be noted that: the above embodiments are only used to illustrate the technical solution of the present invention, and not to limit the same; while the invention has been described in detail and with reference to the foregoing embodiments, it will be understood by those skilled in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some or all of the technical features may be equivalently replaced; and the modifications or the substitutions do not make the essence of the corresponding technical solutions depart from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A data processing method, comprising:

acquiring original image data, and performing nonlinear image transformation processing on the original image data to obtain image data to be processed;

extracting base layer image data with preset data volume from the image data to be processed or the original image data;

coding the basic layer image data based on a first preset coding rule to obtain a basic layer image code stream;

calculating a residual error based on the image data to be processed and the base layer image data, and determining enhancement layer image data based on the residual error;

coding the enhancement layer image data based on a second preset coding rule to obtain an enhancement layer image code stream;

and writing the enhancement layer image code stream into a preset field in the base layer image code stream to generate a target image code stream.

2. The method of claim 1, wherein the raw image data is high dynamic range image data.

3. The method of claim 1, wherein the raw image data comprises linear image data or non-linear image data.

4. The method according to claim 1, wherein the performing the non-linear image transformation on the original image data to obtain image data to be processed comprises:

and performing at least one time of nonlinear image transformation processing on the original image data to obtain image data to be processed.

5. The method according to claim 1, wherein the performing the non-linear image transformation on the original image data to obtain image data to be processed comprises:

determining a target of a non-linear image transformation of the original image data based on the image precision loss;

and performing nonlinear image transformation processing on the original image based on the target of the nonlinear image transformation to obtain image data to be processed.

6. The method according to claim 1, wherein the extracting a preset amount of base layer image data from the image data to be processed or the original image data comprises:

and performing linear image transformation processing on the image data to be processed or the original image data to obtain base layer image data with preset data volume.

7. The method of claim 1, further comprising:

preprocessing the basic layer image data to obtain preprocessed basic layer image data;

the computing a residual based on the image data to be processed and the base layer image data comprises:

calculating the residual based on the image data to be processed and the pre-processed base layer image data.

8. The method of claim 7, wherein preprocessing the base layer image data to obtain the preprocessed base layer image data comprises:

and decoding the base layer image code stream based on a first preset decoding rule corresponding to the first preset encoding rule, and taking data obtained by decoding as the preprocessed base layer image data.

9. The method of claim 7, wherein computing the residual based on the image data to be processed and the pre-processed base layer image data comprises:

and subtracting the preprocessed base layer image data from the image data to be processed to obtain the residual error.

10. The method of claim 1, wherein determining enhancement layer image data based on the residual comprises:

obtaining data to be reserved based on the residual error;

and selecting reserved data from the data to be reserved as the enhancement layer image data.

11. The method of claim 10, wherein selecting the retained data from the data to be retained as the enhancement layer image data comprises:

and adjusting the reserved data to obtain the enhancement layer image data with positive integer data.

12. The method of claim 7, wherein computing the residual based on the image data to be processed and the pre-processed base layer image data comprises:

aligning a dynamic range represented by the image data to be processed with a dynamic range represented by the preprocessed base layer image data;

compensating the aligned image data to be processed according to the proportion of the dynamic range of the enhancement layer image data to the dynamic range of the base layer image data;

and subtracting the preprocessed base layer image data from the compensated image data to be processed to obtain a residual error.

13. The method of claim 1, further comprising: adding random noise data to the base layer image data;

and calculating residual errors based on the image data to be processed and the base layer image data added with random noise data.

14. The method of claim 1, further comprising: and adding random noise data in the process of calculating the residual error so as to enable the obtained residual error to have random noise data.

15. The method of claim 1, further comprising: after the residual is calculated, random noise data is added to the residual.

16. The method according to claim 1, wherein the first predetermined encoding rule is an encoding rule determined according to a JPEG encoding rule.

17. The method of claim 1, wherein the encoding the enhancement layer image data based on the second preset encoding rule comprises:

encoding the enhancement layer image data based on the base layer image data.

18. The method of claim 17, wherein encoding the enhancement layer image data based on the base layer image data comprises:

determining first information reflecting the compression condition of the enhancement layer image data according to the base layer image data;

encoding the enhancement layer image data based on the first information.

19. The method of claim 17, wherein determining the first information reflecting the compression of the enhancement layer image data according to the base layer image data comprises:

and determining first information reflecting the compression condition of the enhancement layer image data according to the brightness distribution condition of the base layer image data.

20. The method of claim 18, wherein said encoding the enhancement layer image data based on the first information comprises:

determining second information dividing image data frequencies of the enhancement layer image data;

quantizing the second information based on the first information;

and encoding the enhancement layer image data according to the quantization result.

21. The method of claim 20, wherein encoding the enhancement layer image data according to the quantization result comprises:

scanning the quantization result and encoding the scanned quantization result;

obtaining the enhancement layer image code stream includes: and generating the enhancement layer image data code stream based on the encoding result.

22. The method of claim 20, further comprising:

dividing the enhancement layer image data to obtain a plurality of groups of macro blocks, and dividing the macro blocks into subblocks;

and aiming at the sub-blocks, determining the second information corresponding to the sub-blocks.

23. The method of claim 21, further comprising:

generating slice image data based on the groups of macro blocks in a combined manner;

wherein the scanning the quantization result comprises:

scanning the quantization result in each sub-block according to a preset scanning mode to obtain a first scanned quantization result;

and for the slice image data, sequentially scanning the quantization results in each sub-block in the slice image data according to the quantization result after the first scanning to obtain a quantization result after the second scanning, which is used as a final quantization result after the scanning.

24. The method of claim 19, wherein determining the first information reflecting the compression of the enhancement layer image data according to the luminance distribution of the base layer image data comprises:

dividing the base layer image data into slice image data;

determining corresponding brightness distribution conditions for each slice image data;

and determining the quantization parameter of each slice of image data according to the brightness distribution condition.

25. The method of claim 19, wherein determining the first information reflecting the compression of the enhancement layer image data according to the luminance distribution of the base layer image data comprises:

dividing the base layer image data into quantized blocks;

determining corresponding brightness distribution conditions for each quantization block;

and determining the quantization parameter of each quantization block according to the brightness distribution condition.

26. The method of claim 24, wherein determining a corresponding intensity profile for each slice of image data comprises:

determining the number of highlight pixel points according to the brightness condition in each slice image data;

determining the number of dark pixel points according to the brightness condition in each slice image data;

and determining the corresponding brightness distribution condition according to the number of the high-brightness pixel points and/or the number of the dark pixel points.

27. The method of claim 26, wherein said determining the number of highlighted pixels according to the brightness in each slice of image data comprises:

converting the YUV format of the image data of the basic layer into an RGB format, and determining the number of highlight pixel points according to the brightness condition in each slice image data in the RGB format;

determining the number of dark pixel points according to the brightness condition in each slice image data, comprising:

determining the number of dark pixel points according to the brightness condition in each slice image data in the YUV format;

28. The method of claim 24, wherein determining the quantization parameter for each slice of image data according to the intensity distribution comprises:

and determining the quantization parameter of the corresponding slice image data according to the number of the high-brightness pixel points and/or the number of the dark pixel points in the brightness distribution condition.

29. The method of claim 21, wherein sub-blocks in the same slice of image data share the same quantization parameter.

30. The method of claim 1, further comprising:

dividing the enhancement layer image code stream, and packaging the divided code stream;

writing the enhancement layer image code stream into a preset field in the base layer image code stream, wherein the method comprises the following steps:

and writing the packaged code stream into the preset field, wherein the packaged code stream has corresponding code stream identification information.

31. A data processing method, comprising:

acquiring a target image code stream;

32. The method of claim 31, wherein the base layer image data is used to generate a base layer image.

33. The method of claim 31, further comprising:

decoding the target image code stream based on a second preset decoding rule to obtain enhancement layer image data;

the enhancement layer image data is used for generating a target image in cooperation with the base layer image data, and the dynamic range of the target image is larger than that of the base layer image.

34. An electronic device, comprising: the device comprises a memory, a processor and an image acquisition device;

the memory for storing a computer program;

the image acquisition device is used for acquiring original image data;

the processor calls the computer program to realize the following steps:

carrying out nonlinear image transformation processing on the original image data to obtain image data to be processed;

35. The apparatus of claim 34, wherein the raw image data is high dynamic range image data.

36. The apparatus of claim 34, wherein the raw image data comprises linear image data or non-linear image data.

37. The apparatus of claim 34, wherein the processor is further configured to: and performing linear image transformation processing on the original image data to obtain image data to be processed.

38. The device of claim 34, wherein the processor is specifically configured to:

39. The device of claim 34, wherein the processor is specifically configured to: and performing linear image transformation processing on the image data to be processed or the original image data to obtain base layer image data with preset data volume.

40. The device of claim 34, wherein the processor is specifically configured to: preprocessing the basic layer image data to obtain preprocessed basic layer image data;

41. The device of claim 40, wherein the processor is specifically configured to: and decoding the base layer image code stream based on a first preset decoding rule corresponding to the first preset encoding rule, and taking data obtained by decoding as the preprocessed base layer image data.

42. The device of claim 40, wherein the processor is specifically configured to: and subtracting the preprocessed base layer image data from the image data to be processed to obtain the residual error.

43. The device of claim 34, wherein the processor is specifically configured to:

obtaining data to be reserved based on the residual error;

44. The device of claim 43, wherein the processor is specifically configured to: and adjusting the reserved data to obtain the enhancement layer image data with positive integer data.

45. The device of claim 40, wherein the processor is specifically configured to:

46. The apparatus of claim 34, wherein the processor is further configured to:

adding random noise data to the base layer image data;

the processor is specifically configured to:

47. The apparatus of claim 34, wherein the processor is further configured to:

and adding random noise data in the process of calculating the residual error so as to enable the obtained residual error to have random noise data.

48. The apparatus of claim 34, wherein the processor is further configured to:

after the residual is calculated, random noise data is added to the residual.

49. The apparatus of claim 34, wherein the first preset encoding rule is an encoding rule determined according to a JPEG encoding rule.

50. The device of claim 34, wherein the processor is specifically configured to:

encoding the enhancement layer image data based on the base layer image data.

51. The device of claim 50, wherein the processor is specifically configured to:

encoding the enhancement layer image data based on the first information.

52. The device of claim 50, wherein the processor is specifically configured to:

53. The device of claim 51, wherein the processor is specifically configured to: determining second information dividing image data frequencies of the enhancement layer image data;

quantizing the second information based on the first information;

54. The device of claim 53, wherein the processor is specifically configured to:

scanning the quantization result and encoding the scanned quantization result;

55. The device of claim 53, wherein the processor is further configured to:

56. The device of claim 54, wherein the processor is further configured to: dividing the enhancement layer image data to obtain a plurality of groups of macro blocks, and dividing the macro blocks into subblocks;

the processor is specifically configured to: scanning the quantization result in each sub-block according to a preset scanning mode to obtain a first scanned quantization result;

57. The device of claim 52, wherein the processor is specifically configured to:

dividing the base layer image data into slice image data;

58. The device of claim 56, wherein the processor is specifically configured to:

dividing the base layer image data into quantized blocks;

59. The device of claim 57, wherein the processor is specifically configured to: determining the number of highlight pixel points according to the brightness condition in each slice image data;

60. The device according to claim 59, wherein the processor is specifically configured to: converting the YUV format of the image data of the basic layer into an RGB format, and determining the number of highlight pixel points according to the brightness condition in each slice image data in the RGB format;

61. The device of claim 57, wherein the processor is specifically configured to: and determining the quantization parameter of the corresponding slice image data according to the number of the high-brightness pixel points and/or the number of the dark pixel points in the brightness distribution condition.

62. The apparatus of claim 54, wherein sub-blocks in the same slice of image data share the same quantization parameter.

63. The apparatus of claim 34, wherein the processor is further configured to:

64. An electronic device, comprising: a memory, a processor, and a communication component;

the memory for storing a computer program;

the communication component is used for acquiring a target image code stream;

the processor calls the computer program to realize the following steps:

65. The device of claim 64, wherein the base layer image data is used to generate a base layer image.

66. The device of claim 64, wherein the processor is further configured to:

67. A computer-readable storage medium, characterized in that the storage medium is a computer-readable storage medium having stored therein program instructions for implementing the method of any one of claims 1-33.