WO2022222112A1 - Data processing method, image sensor, image processor and electronic device - Google Patents

Abstract

A data processing method, an image sensor, an image processor and an electronic device. The data processing method comprises: an image sensor obtaining at least two pieces of image data for the same pixel position, wherein the at least two pieces of image data comprise first image data and second image data (S110); on the basis of a preset mapping relationship between the second image data and the first image data, determining a residual error corresponding to the second image data (S120); performing compressed encoding on the residual error to obtain compressed data (S130); and taking both the first image data and the compressed data as pixel data at the pixel position, and transmitting same to an image processor (S140). By means of the method, there is no need to transmit different original image data at the same pixel position to an image processor to synthesize a final pixel value at the pixel position by the image processor, but rather some of the original image data and residual error compressed data of the remaining original image data are transmitted, such that a data transmission amount can be significantly reduced.

Description

Data processing method, image sensor, image processor and electronic device

manual

technical field

The present application generally relates to the technical field of image acquisition and processing, and more particularly to a data processing method, an image sensor, an image processor and an electronic device.

Background technique

Dynamic range is an important metric for image sensors. Dynamic range is generally defined as the ratio of the brightness when the image sensor reaches its maximum signal value and the brightness when the amplitude of the signal value output by the image sensor equals the amplitude of the noise signal. An image sensor with a large dynamic range can accurately record both bright and dark subjects at the same time.

There are many types of high dynamic range sensor technologies, such as Digital Overlap (DOL for short) technology and Dual Conversion Gain (Dual Conversion Gain, DCG for short) technology. Among them, for the image sensor using DOL technology, each pixel is continuously exposed twice, which are recorded as long-exposure L signal and short-exposure S signal, which are output successively; the image processor that receives the L signal and the S signal uses an image processing algorithm to The signals are combined into a pixel value. Image sensors using DCG technology, each pixel performs two analog-to-digital conversions, and the gains of the two conversions are different. For example, the final pixel value is HG when the gain is high, and the final pixel value is LG when the gain is low; image sensor The HG signal and the LG signal are output successively. The image processor that receives the two signals uses an image processing algorithm to combine the two signals into a single pixel value. The problems brought about by the above two technologies have a common problem, that is, the amount of data sent is doubled. Under the trend of more and more pixels and higher frame rate, it is difficult to bear the doubling of the amount of data. Another method is to synthesize the two signals into a single pixel signal inside the image sensor and then send it. This method increases the complexity and power consumption of the image sensor, which cannot be realized in many cases.

SUMMARY OF THE INVENTION

In order to solve the above problems, the present application provides a data processing solution. The data processing solution proposed by the present application is briefly described below, and more details will be described in the specific embodiments in conjunction with the accompanying drawings in the following.

According to an aspect of the present application, there is provided a data processing method, the method comprising: obtaining at least two image data for the same pixel position by an image sensor, the at least two image data including first image data and second image data ; determine the residual corresponding to the second image data based on the preset mapping relationship between the second image data and the first image data; perform compression coding on the residual to obtain compressed data; Both the image data and the compressed data are transmitted to the image processor as pixel data at the pixel locations.

According to another aspect of the present application, a data processing method is provided, the method includes receiving at least two image data output by an image sensor for the same pixel position, the at least two image data including a first image data and a second image compressed data corresponding to the data; decode the compressed data to obtain decoded data; determine based on the preset mapping relationship between the first image data, the second image data and the first image data, and the decoded data the second image data; fusing the first image data and the second image data to obtain the pixel value of the pixel position.

According to yet another aspect of the present application, an image sensor is provided, the image sensor includes an optoelectronic device, a storage device, an encoding device and an interface device, wherein: the optoelectronic device is used to obtain at least two image data for the same pixel position, The at least two image data includes first image data and second image data; the storage device is used for storing a computer program executed by the encoding device, the computer program executing the following steps when executed by the encoding device : determine the residual corresponding to the second image data based on the preset mapping relationship between the second image data and the first image data, and perform compression coding on the residual to obtain compressed data; the interface device for transmitting both the first image data and the compressed data to an image processor as pixel data at the pixel location.

According to yet another aspect of the present application, an image processor is provided, the image processor includes an interface device, a storage device and a decoding device, wherein: the interface device is configured to receive at least two output from an image sensor for the same pixel position image data, the at least two image data include compressed data corresponding to the first image data and the second image data; the storage device is used for storing a computer program executed by the decoding device, and the computer program is executed by the decoding device. When the decoding device is running, the following steps are performed: decoding the compressed data to obtain decoded data; based on the first image data, the preset mapping relationship between the second image data and the first image data, and the decoding The data determines the second image data; the first image data and the second image data are fused to obtain the pixel value of the pixel position.

According to yet another aspect of the present application, a data processing method is provided, the method comprising: an image sensor generating at least two pixel data for the same pixel position; wherein the at least two pixel data includes first image data, and Compressed data obtained by compressing the residual corresponding to the second image data; the residual is determined based on the preset mapping relationship between the second image data and the first image data; the image sensor sends the image processor The at least two pixel data are sent.

According to yet another aspect of the present application, there is provided a data processing method, the method comprising: an image processor receiving at least two pixel data output from an image sensor for the same pixel position; wherein the at least two pixel data includes the first image data, and compressed data obtained by compressing the residual corresponding to the second image data; the residual is determined based on the preset mapping relationship between the second image data and the first image data; the The image processor obtains pixel values of the pixel positions based on the first image data and the compressed data.

According to still another aspect of the present application, an image sensor is provided, the image sensor includes a storage device, a processing device and an interface device, wherein: the storage device is used for storing a computer program executed by the processing device, the computer When the program is run by the processing device, the following steps are performed: generating at least two pixel data for the same pixel position; wherein the at least two pixel data includes the first image data, and performing the following steps on the residual corresponding to the second image data compressed data obtained by compression; the residual is determined based on the preset mapping relationship between the second image data and the first image data; the interface device is configured to send the at least two pixels to the image processor data.

According to yet another aspect of the present application, an image processor is provided, the image processor includes an interface device, a storage device and a processing device, wherein: the interface device is configured to receive at least an output image sent by an image sensor for the same pixel position Two pixel data; wherein, the at least two pixel data include first image data and compressed data obtained by compressing residuals corresponding to the second image data; the residuals are based on the second image data and the The preset mapping relationship of the first image data is determined; the storage device is configured to store a computer program executed by the processing device, and the computer program executes the following steps when executed by the processing device: based on the The first image data and the compressed data obtain pixel values at the pixel locations.

According to yet another aspect of the present application, an electronic device is provided, the electronic device includes an image sensor and an image processor, wherein: the image sensor includes the above-mentioned image sensor; and the image processor includes the above-mentioned image processor.

According to yet another aspect of the present application, a computer-readable storage medium is provided, the computer-readable storage medium comprising instructions, when executed on a computer, cause the computer to execute the above data processing method.

The data processing method, image sensor, image processor and electronic device according to the embodiments of the present application do not need to transmit different original image data of the same pixel position to the image processor to synthesize the final pixel value of the pixel position by the image processor, but It is the residual compressed data that transmits part of the original image data and the rest of the original image data, which can greatly reduce the amount of data transmission; moreover, because different image data at the same pixel position is still transmitted to the image processor, there is no need for the image sensor side. Perform pixel synthesis without adding complexity and power consumption to the image sensor.

Description of drawings

FIG. 1 shows a schematic flowchart of a data processing method according to an embodiment of the present application.

FIG. 2 shows a schematic flowchart of a data processing method according to another embodiment of the present application.

FIG. 3 shows a schematic structural block diagram of an image sensor according to an embodiment of the present application.

FIG. 4 shows a schematic structural block diagram of an image processor according to an embodiment of the present application.

FIG. 5 shows a schematic structural block diagram of an electronic device according to an embodiment of the present application.

FIG. 6 shows a schematic flowchart of a data processing method according to still another embodiment of the present application.

FIG. 7 shows a schematic flowchart of a data processing method according to yet another embodiment of the present application.

FIG. 8 shows a schematic structural block diagram of an image sensor according to another embodiment of the present application.

FIG. 9 shows a schematic structural block diagram of an image processor according to another embodiment of the present application.

Detailed ways

In order to make the objectives, technical solutions and advantages of the present application more apparent, the exemplary embodiments according to the present application will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application, and it should be understood that the present application is not limited by the example embodiments described herein.

In the following description, numerous specific details are set forth in order to provide a more thorough understanding of the present application. It will be apparent, however, to one skilled in the art that the present application may be practiced without one or more of these details. In other instances, some technical features known in the art have not been described in order to avoid confusion with the present application.

It should be understood that the application may be embodied in different forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of this application to those skilled in the art.

The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. As used herein, the singular forms "a," "an," and "the/the" are intended to include the plural forms as well, unless the context clearly dictates otherwise. It should also be understood that the terms "compose" and/or "include", when used in this specification, identify the presence of stated features, integers, steps, operations, elements and/or components, but do not exclude one or more other The presence or addition of features, integers, steps, operations, elements, parts and/or groups. As used herein, the term "and/or" includes any and all combinations of the associated listed items.

For a thorough understanding of the present application, detailed steps and detailed structures will be presented in the following description, so as to explain the technical solutions proposed by the present application. The preferred embodiments of the present application are described in detail below, however, the present application may have other embodiments in addition to these detailed descriptions.

FIG. 1 shows a schematic flowchart of a data processing method 100 according to an embodiment of the present application. As shown in FIG. 1 , the data processing method 100 according to the embodiment of the present application may include the following steps:

In step S110, at least two image data including the first image data and the second image data are obtained by the image sensor for the same pixel position.

In step S120, a residual corresponding to the second image data is determined based on a preset mapping relationship between the second image data and the first image data.

In step S130, compression coding is performed on the residual to obtain compressed data;

In step S140, both the first image data and the compressed data are transmitted to an image processor as pixel data at the pixel positions.

In the embodiment of the present application, the image sensor obtains at least two image data for the same pixel position, that is, the first image data and the second image data, and transmits the residual compressed data of the first image data and the second image data to the image The processor is configured to determine the final pixel value of the pixel position in combination with the first image data after the second image data is recovered by the image processor. In the embodiment of the present application, the data transmitted to the image processor includes two types: one is the original image data (ie the first image data), and the other is the processed image data (ie the residual of the second image data) The data obtained after being compressed); wherein, the residual of the second image data is determined according to the preset mapping relationship between the second image data and the first image data, and the compressed data is obtained by compressing and encoding the residual. ; After the image processor receives the two types of data, it can decode and obtain the second image data according to the preset mapping relationship and the corresponding decoding method, so as to determine the final pixel value of the pixel position in combination with the first image data. Therefore, the data processing method according to the embodiment of the present application does not need to transmit different original image data of the same pixel position to the image processor to synthesize the final pixel value of the pixel position by the image processor, but transmits part of the original image data and the rest Residual compressed data of the original image data, which can greatly reduce the amount of data transmission; moreover, because different image data at the same pixel position is still transmitted to the image processor, there is no need to perform pixel synthesis on the image sensor side, avoiding adding image sensors. complexity and power consumption.

In the embodiment of the present application, the image sensor may be exposed at least twice for the same pixel position to obtain at least two image data, and the time of each exposure is different. In another embodiment, the image sensor may obtain exposure data by exposing the same pixel position once, and perform at least two analog-to-digital conversions on the exposure data to obtain at least two image data, and the gain of each analog-to-digital conversion is different. In other embodiments, the image sensor may also obtain at least two image data for the same pixel position in other suitable manners. In the embodiments of the present application, a pixel position may refer to a pixel unit, or a pixel row, or other positions related to pixels. For example, the above exposure can be performed in row units, for example, long exposure and short exposure are successively performed for each row of pixels, and then sequentially sent to the image processor in row units to meet the needs of timing control. Similarly, the image data obtained after analog-to-digital conversion can also be output to the image processor in units of lines or in units of frames to meet the needs of timing control.

The at least two image data obtained for the same pixel position are described below by taking at least two exposures for the same pixel position as an example. In the embodiment of the present application, when the same pixel position is exposed twice, two image data are obtained, namely the first image data and the second image data, wherein the first image data is the image obtained after the first exposure data, the second image data is the image data obtained after the second exposure. For example, if the time of the first exposure is longer than the time of the second exposure, the first image data is the data obtained by the long exposure, and the second image data is the data obtained by the short exposure; conversely, if the time of the first exposure is If the time of the second exposure is shorter than that of the second exposure, the first image data is the data obtained by the short exposure, and the second image data is the data obtained by the long exposure.

In the embodiment of the present application, when the same pixel position is exposed more than twice, the first image data may be the output data obtained after the first exposure, and the second image data may be the output data obtained after the remaining exposures. In this embodiment, except the data obtained after the first exposure is used as the first image data, the data obtained from other exposures are used as the second image data, which means that the residual data corresponding to the second image data is determined in step S120 The difference time depends on the data obtained after the first exposure. For example, when exposing the same pixel position three times, the first image data is the output data obtained after the first exposure, such as the A data, the data obtained after the second exposure and the data obtained after the third exposure are both For the second image data, in order to distinguish from each other, the data obtained after the second exposure and the data obtained after the third exposure may be referred to as B data and C data, respectively. Then, the residual of the B data may be determined according to the preset mapping relationship between the A data and the B data; the residual of the C data may be determined according to the preset mapping relationship between the A data and the C data. The preset mapping relationship will be further described below.

In another embodiment of the present application, when the same pixel position is exposed more than twice, the first image data may also be part of all output data obtained after each exposure, and the second image data may be other part of the data . In this embodiment, among the data obtained by multiple exposures, part of the data is used as the first image data, and the remaining part of the data is used as the second image data. For example, when exposures are made for the same pixel position four times, the first image data is, for example, the output data obtained after the first exposure and the second exposure, respectively. The data obtained after the first exposure and the data obtained after the fourth exposure are both the second image data. In order to distinguish from each other, the data obtained after the third exposure and the data obtained after the fourth exposure can be called C data respectively. and D data. Then, the residual of the C data can be determined according to the preset mapping relationship between the A data and the C data, or can be determined according to the preset mapping relationship between the B data and the C data; the residual of the D data can be determined according to the A data The preset mapping relationship with the D data is determined, or may be determined according to the preset mapping relationship between the B data and the D data.

In the embodiments of the present application, whether multiple exposures (with different exposure times for each time) are used to obtain multiple image data for the same pixel position, or multiple analog-to-digital conversions are used for one exposure (each time the gain of analog-to-digital conversion is different) To obtain multiple image data for the same pixel position, there can be a definite mapping relationship between the finally obtained image data. For example, when using DOL, DCG and other high dynamic range sensor technologies that output multiple signals of different amplitudes for the same pixel position, record the current pixel position and output two different pixel values (ie image data), which are S and L respectively. When neither S nor L reaches 0 or the maximum value, the numerical relationship between the two can be determined. This relationship may be determined by image sensor hardware design (such as analog-to-digital conversion gain in DCG mode), or by image sensor exposure parameters specified by the control program (such as exposure time in DOL mode). Usually the two have a roughly linear relationship, such as L=4*S (in this expression, the ratio between two exposure times is 4 as an example to describe, that is, the long exposure time is 4 times the short exposure time). There are some deviations in the case. In practical applications, the mapping relationship L=M(S) can be obtained by traversing the L signals corresponding to different S signals. Therefore, in the embodiments of the present application, the manner of obtaining multiple image data for the same pixel position may be predetermined, and accordingly, the mapping relationship between the obtained image data may be further determined, as described in the previous example.

Based on this, determining the residual corresponding to the second image data based on the preset mapping relationship between the second image data and the first image data in step S120 may include: mapping the second image data based on the preset mapping relationship The image data is expressed as a function of the first image data to obtain derived image data; the difference between the second image data and the derived image data is calculated to obtain a residual corresponding to the second image data. In the foregoing example, the first image data is S, and the second image data is L. According to the mapping relationship M, L can be expressed as M(S) (ie, the derived image data), and then the residual D=L-M is calculated. (S), which is the difference between the L signal and M(S).

In the embodiments of the present application, the above-mentioned mapping relationship is generally established when both the first image data and the second image data are not zero or have not reached the limit maximum value. When both of the image data are not zero or have not reached the limit maximum value, the value of the D=L-M(S) signal is very close to 0, which is very beneficial for compression. When either of the first image data and the second image data is zero or reaches the limit maximum value, the value of the D=L-M(S) signal is not close to 0, and the compressed data will be more complicated. Overall, since L=M(S) can be satisfied in most cases, a higher compression ratio can be achieved overall, as described below.

In the embodiment of the present application, after the residual is obtained in step S120, the residual is encoded to obtain compressed data, and then the first image data and the compressed data are transmitted to the image processor as pixel data at the pixel position, as mentioned earlier. In the previous example, the residual D=L-M(S), since in most cases L=M(S), ie D=L-M(S), the signal value is very close to 0, so the magnitude of D is compared with the L signal. Much smaller and more concentrated for compression. Sending the compressed data to the image processor, rather than sending the data L directly, can greatly reduce the amount of data transferred. In addition, the first image data and the compressed data can be sent to the image processor together, or after the first image data is obtained, the first image data can be sent to the image processor, and then the compressed data can be calculated and sent to the image processor. device.

In one example, encoding the residual may include: performing difference encoding on the residual (ie, Differential Pulse code modulation, DPCM encoding for short). Specifically, performing difference encoding on the residual to obtain compressed data may include: using the true value of the previous residual corresponding to one or more pixel positions before the pixel position corresponding to the current residual as the prediction of the current residual Calculate the difference between the true value of the current residual and the predicted value as error data; perform positive mapping on the error data to obtain positive data; quantify the positive data to obtain a quantized data; entropy encoding is performed on the quantized data to obtain the compressed data. In one example, the entropy encoding of the quantized data may include: Huffman encoding or Gromb encoding of the quantized data.

The above encoding process is described below with a more detailed example. Taking N (eg, N=8) adjacent pixels as a unit, for each pixel, the current pixel is predicted by the numerical value of the first m (eg, m=4) pixels, as expressed by the following formula:

in,

is the predicted value of the current residual, x ₀ , x ₁ , x _m-1 are the real values of the previous residual corresponding to m pixel positions before the pixel position corresponding to the current residual, a ₀ , a ₁ , ..., a _m-1 is a predefined coefficient (eg: a ₀ =a ₁ =a ₂ =a ₃ =0.25).

The error data _em is the difference between the actual value of the current residual and the predicted value, namely:

The positive value mapping is performed on the error data to obtain the positive value data e' _m . The positive value mapping process is:

Quantize the positive data _e'm , that is:

Wherein, q is a predefined quantization coefficient, for example, q=2.

Finally, entropy encoding is performed on e _q to obtain compressed data.

In the embodiments of the present application, Huffman coding is used for entropy coding, that is, a variable-length coding table is used to encode numerical values by using a table look-up method. In other embodiments, other entropy coding techniques, such as Glenbow coding, may also be used. In addition, the aforementioned predefined parameters, such as the mapping relationship L=M(S), the DPCM coding scheme, the number N of pixels included in the adjacent pixel group, the number m of previous pixels used to predict the current pixel value, the quantization coefficient q, the entropy coding The code table (such as the Huffman code table used by Huffman coding) can be specified and stored in the image sensor by the image sensor control program. At the same time, the image sensor control program can notify the image processor of the same parameters to reconstruct the L signal before compression (ie, the second image data).

Based on the above description, the data processing method according to the embodiment of the present application does not need to transmit different original image data of the same pixel position to the image processor to synthesize the final pixel value of the pixel position by the image processor, but transmits part of the original image data and the residual compressed data of the rest of the original image data, which can greatly reduce the amount of data transmission; moreover, because different image data at the same pixel position is still transmitted to the image processor, there is no need to perform pixel synthesis on the image sensor side, avoiding Increase the complexity and power consumption of the image sensor.

The above exemplarily shows the data processing method 100 according to an embodiment of the present application, which is described from the perspective of an image sensor, and can be executed by the image sensor. The following describes a data processing method 200 according to another embodiment of the present application with reference to FIG. 2 , which is described from the perspective of an image processor and can be executed by an image processor, and the data processing method 200 is the same as the data processing method 100 Corresponding. As shown in FIG. 2, the data processing method 200 may include the following steps:

In step S210, at least two image data output by the image sensor for the same pixel position are received, where the at least two image data include compressed data corresponding to the first image data and the second image data.

In step S220, the compressed data is decoded to obtain decoded data;

In step S230, the second image data is determined based on the first image data, a preset mapping relationship between the second image data and the first image data, and the decoded data;

In step S240, the first image data and the second image data are fused to obtain the pixel value of the pixel position.

In the embodiment of the present application, the data processing method 200 is the data processing performed by the image processor after receiving different image data of the same pixel position obtained by the image sensor after passing through the data processing method 100 . Specifically, after receiving the compressed data corresponding to the first image data and the second image data (the first image data and the compressed data may be received from the image sensor together, or the first image data may be received first, and then received Compressed data, depending on the sending situation of the image sensor), decode the compressed data (the decoding method corresponds to the encoding method, and the parameters related to decoding can be sent by the image sensor to the image processor, or can be stored in the image processor in advance); The decoded data obtained after decoding is the residual data in the data processing method 100 (such as D in the foregoing example); then, based on the preset mapping relationship (such as M in the foregoing example), the first image data (such as S) in the second image data L, that is, the derived image data of the second image data (such as M(S) in the previous example), the sum of the derived image data and the residual data (ie the decoded data) is the first Two image data (L=D+M(S)); after obtaining the second image data itself, fuse both the first image data and the second image data (such as averaging, weighted averaging, or other image processing algorithm), you can get the pixel value of the pixel position.

Based on the above description, the data processing method according to the embodiment of the present application receives part of the original image data for the same pixel position and residual compressed data of the remaining original image data from the image sensor, which can greatly reduce the amount of data transmission.

The data processing method according to the embodiment of the present application has been exemplarily described above, and the apparatus for executing the above method will be described below with reference to FIG. 3 and FIG. 4 .

FIG. 3 shows a schematic structural block diagram of an image sensor 300 according to an embodiment of the present application. As shown in FIG. 3 , the image sensor 300 includes an optoelectronic device 310, a storage device 320, an encoding device 330 and an interface device 340, wherein: the optoelectronic device 310 is used to obtain at least two image data for the same pixel position, the at least two The pieces of image data include first image data and second image data; the storage device 320 is configured to store a computer program executed by the encoding device 330, and the computer program executes the following steps when executed by the encoding device 330: The residual corresponding to the second image data is determined based on the preset mapping relationship between the second image data and the first image data, and the residual is compressed and encoded to obtain compressed data; the interface device 340 for transmitting both the first image data and the compressed data to an image processor as pixel data at the pixel location. The image sensor 300 according to the embodiment of the present application can be used to execute the data processing method 100 described above. Those skilled in the art can understand the operations performed by the components of the image sensor 300 in combination with the above description. For brevity, only some main operations are described here. , the specific details will not be repeated.

In the embodiment of the present application, the optoelectronic device 310 is used for exposing the same pixel position at least twice to obtain at least two image data, and the time of each exposure is different; or the optoelectronic device 310 is used for the same pixel position Exposure is performed once to obtain exposure data, and at least two analog-to-digital conversions are performed on the exposure data to obtain at least two image data, and the gain of each analog-to-digital conversion is different.

In the embodiment of the present application, the computer program is executed when the encoding device 330 is executed to determine the second image based on the preset mapping relationship between the second image data and the first image data The residual error corresponding to the data includes: representing the second image data as a function of the first image data based on a preset mapping relationship to obtain derived image data; calculating the difference between the second image data and the derived image data The residual difference corresponding to the second image data is obtained; wherein, the preset mapping relationship is determined based on the ratio between the exposure times used in different exposures and the linearity of the image sensor , or, the preset mapping relationship is determined based on the ratio between the respective gains of different analog-to-digital conversions and the linearity of the image sensor.

In the embodiment of the present application, when both the first image data and the second image data are not zero or reach a limit maximum value, the residual is close to zero.

In the embodiment of the present application, after the optoelectronic device 310 outputs the first image data, the interface device 340 first transmits the first image data to the image processor, and then transmits the first image data from the encoding device 330 obtains the compressed data and transmits the compressed data to the image processor.

In the embodiment of the present application, the photoelectric device 310 is exposed twice for the same pixel position, and if the time of the first exposure is longer than the time of the second exposure, the first image data is obtained by long exposure If the time of the first exposure is shorter than the time of the second exposure, the first image data is the data obtained by the short exposure, and the second image The data is the data obtained by long exposure.

In the embodiment of the present application, the photoelectric device 310 is exposed more than twice for the same pixel position, and: the first image data is the output data obtained after the first exposure, and the second image data is the rest or the first image data is part of all the output data obtained after each exposure, and the second image data is other part of the data.

In the embodiment of the present application, the encoding device 330 is configured to perform difference encoding on the residual.

In the embodiment of the present application, the encoding device 330 performs difference encoding on the residual to obtain compressed data, including: encoding previous residuals corresponding to one or more pixel positions before the pixel positions corresponding to the current residuals The true value of the current residual is used as the predicted value of the current residual; the difference between the true value of the current residual and the predicted value is calculated as the error data; the positive value mapping is performed on the error data to obtain positive value data; The positive value data is quantized to obtain quantized data; the quantized data is entropy encoded to obtain the compressed data.

In the embodiment of the present application, the entropy encoding performed by the encoding device 330 on the quantized data includes Huffman encoding or Gromb encoding.

Based on the above description, the image sensor according to the embodiment of the present application does not need to transmit different original image data of the same pixel position to the image processor to synthesize the final pixel value of the pixel position by the image processor, but transmits part of the original image data and the residual compressed data of the rest of the original image data, which can greatly reduce the amount of data transmission; moreover, because different image data at the same pixel position is still transmitted to the image processor, there is no need to perform pixel synthesis on the image sensor side, avoiding increased Image sensor complexity and power consumption.

FIG. 4 shows a schematic structural block diagram of an image processor 400 according to an embodiment of the present application. As shown in FIG. 4 , the image processor 400 includes an interface device 410, a storage device 420 and a decoding device 430, wherein: the interface device 410 is configured to receive at least two image data output by the image sensor for the same pixel position, the at least The two image data include compressed data corresponding to the first image data and the second image data; the storage device 420 is used to store a computer program executed by the decoding device 430, and the computer program is executed by the decoding device 430. The following steps are performed when the compressed data is decoded to obtain decoded data; based on the preset mapping relationship between the first image data, the second image data and the first image data and the decoded data, determine the the second image data; the first image data and the second image data are fused to obtain the pixel value of the pixel position. The image processor 400 according to the embodiment of the present application can be used to execute the data processing method 200 described above. Those skilled in the art can understand the operations performed by the components of the image processor 400 in combination with the above description. For brevity, only some of the operations are described here. The main operation, the specific details will not be repeated.

In the embodiment of the present application, the preset mapping based on the first image data, the second image data and the first image data executed when the computer program is executed by the decoding device 430 and the decoded data to determine the second image data, including: mapping the first image data to derived data based on the preset mapping relationship, and calculating the difference between the derived data and the decoded data and obtain the second image data.

In the embodiment of the present application, the preset mapping relationship is received by the interface device 410 from the image sensor and transmitted to the decoding device 430 .

In the embodiment of the present application, the interface device 410 receives the first image data first, and then receives the compressed data.

Based on the above description, the image processor according to the embodiment of the present application receives part of the original image data for the same pixel position and residual compressed data of the remaining original image data from the image sensor, which can greatly reduce the amount of data transmission.

The electronic device provided according to still another aspect of the present application will be described below with reference to FIG. 5 . FIG. 5 shows a schematic structural block diagram of an electronic device 500 according to an embodiment of the present application. As shown in FIG. 5 , the electronic device 500 may include an image sensor 510 and an image processor 520, wherein: the image sensor 510 may be the image sensor 300 according to the embodiment of the present application described above, and the image processor 520 may be It is the aforementioned image processor 400 according to the embodiment of the present application. That is, the electronic device 500 according to the embodiment of the present application can simultaneously implement the above data processing methods 100 and 200 . Exemplarily, the electronic device 500 may be various high-, middle- and low-end cameras, video cameras, as well as various mobile phones, drones, etc. that use the camera device.

In yet another embodiment of the present application, a data processing method is also provided, which will be described below with reference to FIG. 6 . FIG. 6 shows a schematic flowchart of a data processing method 600 according to still another embodiment of the present application. As shown in FIG. 6, the data processing method 600 may include the following steps:

In step S610, the image sensor generates at least two pixel data for the same pixel position; wherein, the at least two pixel data includes the first image data and the compressed data obtained by compressing the residual corresponding to the second image data; The residual is determined based on a preset mapping relationship between the second image data and the first image data.

In step S620, the image sensor sends the at least two pixel data to the image processor.

In the embodiment of the present application, the data processing method 600 is substantially the same as the data processing method 100 shown in the foregoing in conjunction with FIG. 1 , except that the data processing method 100 is described from the perspective of the internal implementation of the image sensor, and the data processing method 600 is Described from the perspective of the code stream. Those skilled in the art can understand the specific process of the data processing method 600 in combination with the foregoing description. For the sake of brevity, the specific process of the data processing method 600 is not repeated here, and only some main steps are described.

In the embodiment of the present application, the first image data and the second image data in step S610 may be obtained by exposing the image sensor to the same pixel position at least twice, and each exposure time is different ; Or the first image data and the second image data in step S610 may be obtained by exposing the image sensor once for the same pixel position to obtain exposure data, and performing at least two analog-to-digital conversions on the exposure data to obtain exposure data. As a result, the gain of each analog-to-digital conversion is different.

In the embodiment of the present application, the residual in step S610 may be determined by the following method: expressing the second image data as a function of the first image data based on a preset mapping relationship to obtain a derived image data; calculating the difference between the second image data and the derived image data to obtain a residual corresponding to the second image data; wherein, the preset mapping relationship is based on exposures respectively adopted for different exposures The ratio between the time and the linearity of the image sensor is determined, or the preset mapping relationship is determined based on the ratio between the respective gains of different analog-to-digital conversions and the linearity of the image sensor. .

Based on the above description, the data processing method 600 according to the embodiment of the present application does not need to transmit different original image data of the same pixel position to the image processor to synthesize the final pixel value of the pixel position by the image processor, but transmits part of the Residual compressed data of the original image data and the rest of the original image data, which can greatly reduce the amount of data transfer; also, because different image data at the same pixel location is still sent to the image processor, there is no need to perform pixel synthesis on the image sensor side , to avoid increasing the complexity and power consumption of the image sensor.

In yet another embodiment of the present application, a data processing method is also provided, which will be described below with reference to FIG. 7 . FIG. 7 shows a schematic flowchart of a data processing method 700 according to yet another embodiment of the present application. As shown in FIG. 7, the data processing method 700 may include the following steps:

In step S710, the image processor receives at least two pixel data output from the image sensor for the same pixel position; wherein, the at least two pixel data includes the first image data, and the residuals corresponding to the second image data are processed compressed data obtained by compression; the residual is determined based on a preset mapping relationship between the second image data and the first image data.

In step S720, the image processor obtains the pixel value of the pixel position based on the first image data and the compressed data.

In the embodiment of the present application, the data processing method 700 is essentially the same as the data processing method 200 shown in the foregoing in conjunction with FIG. 2 , except that the data processing method 200 is described from the perspective of the internal implementation of the image processor, and the data processing method 700 is Described from the point of view of the code stream. Those skilled in the art can understand the specific process of the data processing method 700 in combination with the foregoing description. For the sake of brevity, the specific process of the data processing method 700 is not repeated here, and only some main steps are described.

In the embodiment of the present application, the image processor obtaining the pixel value of the pixel position based on the first image data and the compressed data may include: the image processor converts the pixel value based on the preset mapping relationship The first image data is mapped to derived data, the sum of the derived data and the decoded data of the compressed data is calculated to obtain the second image data, and the first image data and the second image data are combined. The image data is fused to obtain the pixel value of the pixel position.

Based on the above description, the data processing method according to the embodiment of the present application receives part of the original image data for the same pixel position and residual compressed data of the remaining original image data from the image sensor, which can greatly reduce the amount of data transmission.

The data processing methods 600 and 700 according to the embodiments of the present application are exemplarily described above, and the apparatuses for executing the above methods are described below with reference to FIG. 8 and FIG. 9 .

FIG. 8 shows a schematic structural block diagram of an image sensor 800 according to another embodiment of the present application. As shown in FIG. 8 , the image sensor 800 includes a storage device 810, a processing device 820, and an interface device 830, wherein: the storage device 810 is used for storing a computer program executed by the processing device 820, and the computer program is executed by the processing device 820. When the processing device 820 is running, the following steps are performed: at least two pixel data are generated for the same pixel position; wherein, the at least two pixel data include the first image data, and the residual error corresponding to the second image data is compressed. compressed data; the residual is determined based on a preset mapping relationship between the second image data and the first image data; the interface device 830 is configured to send the at least two pixel data to the image processor. The image sensor 800 according to the embodiment of the present application can be used to execute the data processing method 600 described above. Those skilled in the art can understand the operations performed by the components of the image sensor 800 in combination with the above description. For brevity, only some main operations are described here. , the specific details will not be repeated.

In the embodiment of the present application, the image sensor 800 further includes an optoelectronic device (not shown), the optoelectronic device is used to obtain the first image data and the second image data, wherein: the optoelectronic device Expose the same pixel position at least twice to obtain the first image data and the second image data, and the time for each exposure is different; or the optoelectronic device is exposed to the same pixel position once to obtain exposure data, and the The exposure data is subjected to at least two analog-to-digital conversions to obtain the first image data and the second image data, and the gain of each analog-to-digital conversion is different.

In the embodiment of the present application, the residual is determined by: representing the second image data as a function of the first image data based on a preset mapping relationship to obtain derived image data; calculating the The difference between the second image data and the derived image data obtains the residual corresponding to the second image data; wherein, the preset mapping relationship is based on the ratio between exposure times respectively adopted for different exposures and the linearity of the image sensor, or, the preset mapping relationship is determined based on the ratio between the respective gains of different analog-to-digital conversions and the linearity of the image sensor.

Based on the above description, the image sensor according to the embodiment of the present application does not need to transmit different original image data of the same pixel position to the image processor to synthesize the final pixel value of the pixel position by the image processor, but transmits part of the original image data and the residual compressed data of the rest of the original image data, which can greatly reduce the amount of data transmission; moreover, because different image data at the same pixel position is still transmitted to the image processor, there is no need to perform pixel synthesis on the image sensor side, avoiding increased Image sensor complexity and power consumption.

FIG. 9 shows a schematic structural block diagram of an image processor 900 according to another embodiment of the present application. As shown in FIG. 9 , the image processor 900 includes an interface device 910, a storage device 920 and a processing device 930, wherein: the interface device 910 is configured to receive at least two pixel data sent by the image sensor and output for the same pixel position; wherein , the at least two pixel data include first image data and compressed data obtained by compressing residuals corresponding to the second image data; the residuals are based on the second image data and the first image data The storage device 920 is configured to store a computer program executed by the processing device 930, and the computer program executes the following steps when executed by the processing device 930: based on the first image data and the compressed data to obtain pixel values for the pixel locations. The image processor 900 according to the embodiment of the present application can be used to execute the data processing method 700 described above. Those skilled in the art can understand the operations performed by the components of the image processor 900 in combination with the above description. For brevity, only some of the operations are described here. The main operation, the specific details will not be repeated.

In the embodiment of the present application, the computer program that is executed by the processing device to obtain the pixel value of the pixel position based on the first image data and the compressed data includes: based on the preset The mapping relationship maps the first image data to derived data, calculates the sum of the derived data and the decoded data of the compressed data to obtain the second image data, and combines the first image data with the derived data. The second image data is fused to obtain the pixel value of the pixel position.

Based on the above description, the image processor according to the embodiment of the present application receives part of the original image data for the same pixel position and residual compressed data of the remaining original image data from the image sensor, which can greatly reduce the amount of data transmission.

According to still another aspect of the present application, an electronic device is also provided, which may include an image sensor (such as the aforementioned image sensor 800 ) and an image processor (such as the aforementioned image processor 900 ), the structure of which may be as follows shown in Figure 5. That is to say, the electronic device according to the embodiment of the present application can simultaneously implement the above data processing methods 600 and 700 . Exemplarily, the electronic device can be various high, middle and low-end cameras, video cameras, as well as various mobile phones, drones, etc. that use camera devices.

In addition, according to an embodiment of the present application, a computer-readable storage medium is also provided, where program instructions are stored on the computer-readable storage medium, and the program instructions are used to execute the present application when the program instructions are run by a computer or a processor Corresponding steps of the data processing method of the embodiment. The computer-readable storage medium may include, for example, a memory card of a smartphone, a storage component of a tablet computer, a hard disk of a personal computer, a read only memory (ROM), an erasable programmable read only memory (EPROM), a portable compact disk Read only memory (CD-ROM), USB memory, or any combination of the above computer readable storage media. The computer-readable storage medium can be any combination of one or more computer-readable storage media.

Based on the above description, the data processing method, image sensor, image processor, and electronic device according to the embodiments of the present application do not need to transmit different original image data of the same pixel position to the image processor to synthesize the pixel position by the image processor. The final pixel value, but the residual compressed data of part of the original image data and the rest of the original image data is transmitted, which can greatly reduce the amount of data transmission; There is no need to perform pixel synthesis on the image sensor side, avoiding increasing the complexity and power consumption of the image sensor.

Although example embodiments have been described herein with reference to the accompanying drawings, it should be understood that the above-described example embodiments are by way of example only, and are not intended to limit the scope of the application thereto. Various changes and modifications may be made therein by those of ordinary skill in the art without departing from the scope and spirit of the present application. All such changes and modifications are intended to be included within the scope of this application as claimed in the appended claims.

Those of ordinary skill in the art can realize that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Skilled artisans may implement the described functionality using different methods for a particular application, but such implementations should not be considered beyond the scope of this application.

In the several embodiments provided in this application, it should be understood that the disclosed apparatus and method may be implemented in other manners. For example, the device embodiments described above are only illustrative. For example, the division of the units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components may be combined or May be integrated into another device, or some features may be omitted, or not implemented.

In the description provided herein, numerous specific details are set forth. It will be understood, however, that the embodiments of the present application may be practiced without these specific details. In some instances, well-known methods, structures and techniques have not been shown in detail in order not to obscure an understanding of this description.

Similarly, it is to be understood that in the description of the exemplary embodiments of the present application, various features of the present application are sometimes grouped together into a single embodiment, FIG. , or in its description. However, this method of application should not be construed as reflecting the intention that the claimed application claims more features than are expressly recited in the claims. Rather, as the corresponding claims reflect, the invention lies in the fact that the corresponding technical problem may be solved with less than all features of a single disclosed embodiment. Thus, the claims following the Detailed Description are hereby expressly incorporated into this Detailed Description, with the claims standing on their own as separate embodiments of this application.

It will be understood by those skilled in the art that all features disclosed in this specification (including the accompanying claims, abstract and drawings) and any method or apparatus so disclosed may be used in any combination, except that the features are mutually exclusive. Processes or units are combined. Unless expressly stated otherwise, the features disclosed in this specification (including accompanying claims, abstract and drawings) may be replaced by alternative features serving the same, equivalent or similar purpose.

Furthermore, those skilled in the art will appreciate that although some of the embodiments described herein include certain features, but not others, included in other embodiments, that combinations of features of different embodiments are intended to be within the scope of the present application within and form different embodiments. For example, in the claims, any of the claimed embodiments may be used in any combination.

The various component embodiments of the present application may be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. Those skilled in the art should understand that a microprocessor or a digital signal processor (DSP) may be used in practice to implement some or all functions of some modules according to the embodiments of the present application. The present application can also be implemented as a program of apparatus (eg, computer programs and computer program products) for performing part or all of the methods described herein. Such a program implementing the present application may be stored on a computer-readable storage medium, or may be in the form of one or more signals. Such signals may be downloaded from Internet sites, or provided on carrier signals, or in any other form.

It should be noted that the above-described embodiments illustrate rather than limit the application, and that alternative embodiments may be devised by those skilled in the art without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses shall not be construed as limiting the claim. The application can be implemented by means of hardware comprising several different elements and by means of a suitably programmed computer. In a unit claim enumerating several means, several of these means may be embodied by one and the same item of hardware. The use of the words first, second, and third, etc. do not denote any order. These words can be interpreted as names.

The above are only specific embodiments of the present application or descriptions of the specific embodiments, and the protection scope of the present application is not limited thereto. Any changes or substitutions should be included within the protection scope of the present application. The protection scope of the present application shall be subject to the protection scope of the claims.

Claims (42)

1. A data processing method, characterized in that the method comprises:

   at least two image data are obtained by the image sensor for the same pixel position, the at least two image data including the first image data and the second image data;

   determining a residual corresponding to the second image data based on a preset mapping relationship between the second image data and the first image data;

   performing compression coding on the residual to obtain compressed data;

   Both the first image data and the compressed data are communicated to an image processor as pixel data at the pixel locations.
2. The method according to claim 1, wherein the obtaining by the image sensor at least two image data for the same pixel position comprises:

   At least two exposures by the image sensor for the same pixel location to obtain at least two image data, each exposure for a different time; or

   Exposure data is obtained by exposing the image sensor once for the same pixel position, and at least two analog-to-digital conversions are performed on the exposure data to obtain at least two image data, and the gain of each analog-to-digital conversion is different.
3. The method according to claim 2, wherein the determining the residual corresponding to the second image data based on the preset mapping relationship between the second image data and the first image data comprises:

   Representing the second image data as a function of the first image data based on a preset mapping relationship to obtain derivative image data;

   calculating the difference between the second image data and the derived image data to obtain a residual corresponding to the second image data;

   Wherein, the preset mapping relationship is determined based on the ratio between exposure times used in different exposure times and the linearity of the image sensor, or the preset mapping relationship is based on the respective analog-to-digital conversion times of different times. The ratio between the gain and the linearity of the image sensor is determined.
4. The method according to claim 1 or 2, characterized in that, when both the first image data and the second image data are not zero or do not reach a limit maximum value, the residual error close to zero.
5. The method according to claim 1 or 2, wherein after the first image data is obtained, the first image data is first transmitted to the image processor, and then obtained based on the preset mapping relationship the derived image data.
6. The method of claim 2, wherein when the image sensor is exposed twice for the same pixel position:

   If the time of the first exposure is longer than the time of the second exposure, the first image data is data obtained by long exposure, and the second image data is data obtained by short exposure;

   If the time of the first exposure is shorter than the time of the second exposure, the first image data is data obtained by short exposure, and the second image data is data obtained by long exposure.
7. The method according to claim 2, wherein when the image sensor is exposed more than twice for the same pixel position:

   The first image data is the output data obtained after the first exposure, and the second image data is the output data obtained after the remaining exposures; or

   The first image data is part of all the output data obtained after each exposure, and the second image data is other part of the data.
8. The method according to any one of claims 1-7, wherein the compression-encoding the residual comprises: difference-encoding the residual.
9. The method according to claim 8, wherein the difference encoding is performed on the residual to obtain compressed data, comprising:

   Taking the true value of the previous residual corresponding to one or more pixel positions before the pixel position corresponding to the current residual as the predicted value of the current residual;

   Calculate the difference between the real value of the current residual and the predicted value as error data;

   performing positive mapping on the error data to obtain positive data;

   Quantifying the positive value data to obtain quantized data;

   Entropy encoding is performed on the quantized data to obtain the compressed data.
10. The method according to claim 9, wherein entropy coding the quantized data comprises:

    Huffman coding or Grumbu coding is performed on the quantized data.
11. A data processing method, characterized in that the method comprises:

    receiving at least two image data output by the image sensor for the same pixel position, the at least two image data including compressed data corresponding to the first image data and the second image data;

    Decoding the compressed data to obtain decoded data;

    determining the second image data based on the first image data, a preset mapping relationship between the second image data and the first image data, and the decoded data;

    The first image data and the second image data are fused to obtain the pixel value of the pixel position.
12. The method according to claim 11, wherein the determining the first image data based on the first image data, the preset mapping relationship between the second image data and the first image data, and the decoded data Two image data, including:

    The first image data is mapped to derived data based on the preset mapping relationship, and the sum of the derived data and the decoded data is calculated to obtain the second image data.
13. The method according to claim 11 or 12, wherein the preset mapping relationship is received from the image sensor.
14. The method according to claim 11, wherein, among the at least two received image data, the first image data is received first, and then the compressed data is received.
15. An image sensor, characterized in that the image sensor includes a photoelectric device, a storage device, an encoding device and an interface device, wherein:

    The optoelectronic device is used to obtain at least two image data for the same pixel position, the at least two image data includes first image data and second image data;

    The storage device is used for storing a computer program executed by the encoding device, and the computer program executes the following steps when executed by the encoding device: a preset based on the second image data and the first image data The mapping relationship determines the residual corresponding to the second image data, and compresses and encodes the residual to obtain compressed data;

    The interface device is configured to transmit both the first image data and the compressed data to an image processor as pixel data at the pixel locations.
16. The image sensor according to claim 15, wherein the photoelectric device is exposed at least twice for the same pixel position to obtain at least two image data, and the time of each exposure is different; or

    The photoelectric device is exposed once for the same pixel position to obtain exposure data, and the exposure data is subjected to at least two analog-to-digital conversions to obtain at least two image data, and the gain of each analog-to-digital conversion is different.
17. 17. The image sensor according to claim 16, wherein the determination based on the preset mapping relationship between the second image data and the first image data is executed when the computer program is executed by the encoding device. The residual corresponding to the second image data includes:

    Representing the second image data as a function of the first image data based on a preset mapping relationship to obtain derivative image data;

    calculating the difference between the second image data and the derived image data to obtain a residual corresponding to the second image data;

    Wherein, the preset mapping relationship is determined based on the ratio between exposure times used in different exposure times and the linearity of the image sensor, or the preset mapping relationship is based on the respective analog-to-digital conversion times of different times. The ratio between the gain and the linearity of the image sensor is determined.
18. 16. The image sensor according to claim 15 or 16, wherein when both the first image data and the second image data are both non-zero or not reaching a limit maximum value, the residual The difference is close to zero.
19. The image sensor according to claim 15 or 16, wherein after the optoelectronic device outputs the first image data, the interface device first transmits the first image data to the image processor, The compressed data is then obtained from the encoding device and transmitted to the image processor.
20. 17. The image sensor of claim 16, wherein the optoelectronic device is exposed twice for the same pixel location, and

    If the time of the first exposure is longer than the time of the second exposure, the first image data is data obtained by long exposure, and the second image data is data obtained by short exposure;

    If the time of the first exposure is shorter than the time of the second exposure, the first image data is data obtained by short exposure, and the second image data is data obtained by long exposure.
21. 17. The image sensor of claim 16, wherein the optoelectronic device is exposed more than twice for the same pixel location, and:

    The first image data is the output data obtained after the first exposure, and the second image data is the output data obtained after the remaining exposures; or

    The first image data is part of all the output data obtained after each exposure, and the second image data is other part of the data.
22. The image sensor according to any one of claims 15-21, wherein the encoding device is used for difference encoding the residual.
23. The image sensor according to claim 22, wherein the encoding device performs differential encoding on the residual to obtain compressed data, comprising:

    Taking the true value of the previous residual corresponding to one or more pixel positions before the pixel position corresponding to the current residual as the predicted value of the current residual;

    Calculate the difference between the real value of the current residual and the predicted value as error data;

    performing positive mapping on the error data to obtain positive data;

    Quantifying the positive value data to obtain quantized data;

    Entropy encoding is performed on the quantized data to obtain the compressed data.
24. The image sensor according to claim 23, wherein the entropy encoding performed by the encoding device on the quantized data includes Huffman encoding or Glumbus encoding.
25. An image processor, characterized in that the image processor includes an interface device, a storage device and a decoding device, wherein:

    The interface device is configured to receive at least two image data output by the image sensor for the same pixel position, the at least two image data including compressed data corresponding to the first image data and the second image data;

    The storage device is used for storing a computer program executed by the decoding device, and the computer program executes the following steps when executed by the decoding device: decoding the compressed data to obtain decoded data; based on the first The image data, the preset mapping relationship between the second image data and the first image data, and the decoded data determine the second image data; fuse the first image data and the second image data , to obtain the pixel value of the pixel position.
26. 26. The image processor according to claim 25, wherein the computer program is executed based on the first image data, the second image data and the first image data when the computer program is executed by the decoding device. The preset mapping relationship of the image data and the decoded data determine the second image data, including:

    The first image data is mapped to derived data based on the preset mapping relationship, and the sum of the derived data and the decoded data is calculated to obtain the second image data.
27. The image processor according to claim 25 or 26, wherein the preset mapping relationship is received by the interface device from the image sensor and transmitted to the decoding device.
28. The image processor according to claim 25, wherein the interface device receives the first image data first, and then receives the compressed data.
29. An electronic device, characterized in that the electronic device includes an image sensor and an image processor, wherein:

    The image sensor comprises the image sensor of any one of claims 15 to 24;

    The image processor includes the image processor of any one of claims 25 to 28.
30. A storage medium, characterized in that, a computer program is stored on the storage medium, and when the computer program is run by a processor, the computer program causes the processor to execute the data according to any one of claims 1-14 Approach.
31. A data processing method, characterized in that the method comprises:

    The image sensor generates at least two pixel data for the same pixel position; wherein, the at least two pixel data includes the first image data and the compressed data obtained by compressing the residual corresponding to the second image data; the residual is determined based on the preset mapping relationship between the second image data and the first image data;

    The image sensor sends the at least two pixel data to an image processor.
32. The method according to claim 31, wherein the first image data and the second image data are obtained by exposing the image sensor to the same pixel position at least twice, and each exposure time is different ;or

    The first image data and the second image data are obtained by exposing the image sensor once for the same pixel position to obtain exposure data, and performing at least two analog-to-digital conversions on the exposure data. The gain of the conversion is different.
33. The method of claim 32, wherein the residual is determined by:

    Representing the second image data as a function of the first image data based on a preset mapping relationship to obtain derivative image data;

    calculating the difference between the second image data and the derived image data to obtain a residual corresponding to the second image data;

    Wherein, the preset mapping relationship is determined based on the ratio between exposure times used in different exposure times and the linearity of the image sensor, or the preset mapping relationship is based on the respective analog-to-digital conversion times of different times. The ratio between the gain and the linearity of the image sensor is determined.
34. A data processing method, characterized in that the method comprises:

    The image processor receives at least two pixel data output from the image sensor for the same pixel position; wherein, the at least two pixel data includes the first image data and the compression obtained by compressing the residual corresponding to the second image data data; the residual is determined based on a preset mapping relationship between the second image data and the first image data;

    The image processor obtains pixel values of the pixel positions based on the first image data and the compressed data.
35. The method according to claim 34, wherein the image processor obtains the pixel value of the pixel position based on the first image data and the compressed data, comprising:

    The image processor maps the first image data to derived data based on the preset mapping relationship, and calculates the sum of the derived data and the decoded data of the compressed data to obtain the second image data, The first image data and the second image data are fused to obtain the pixel value of the pixel position.
36. An image sensor, characterized in that the image sensor includes a storage device, a processing device and an interface device, wherein:

    The storage device is configured to store a computer program executed by the processing device, the computer program executing the following steps when executed by the processing device: generating at least two pixel data for the same pixel position; wherein the at least two The pieces of pixel data include first image data and compressed data obtained by compressing residuals corresponding to the second image data; the residuals are based on a preset mapping relationship between the second image data and the first image data definite;

    The interface device is used for sending the at least two pixel data to the image processor.
37. The image sensor of claim 36, wherein the image sensor further comprises an optoelectronic device for obtaining the first image data and the second image data, wherein:

    The optoelectronic device is exposed at least twice for the same pixel position to obtain the first image data and the second image data, and the time of each exposure is different; or

    The photoelectric device is exposed once for the same pixel position to obtain exposure data, and the exposure data is subjected to at least two analog-to-digital conversions to obtain the first image data and the second image data, and the gain of each analog-to-digital conversion is different.
38. 38. The image sensor of claim 37, wherein the residual is determined by:

    Representing the second image data as a function of the first image data based on a preset mapping relationship to obtain derivative image data;

    calculating the difference between the second image data and the derived image data to obtain a residual corresponding to the second image data;

    Wherein, the preset mapping relationship is determined based on the ratio between exposure times used in different exposure times and the linearity of the image sensor, or the preset mapping relationship is based on the respective analog-to-digital conversion times of different times. The ratio between the gain and the linearity of the image sensor is determined.
39. An image processor, characterized in that the image processor includes an interface device, a storage device and a processing device, wherein:

    The interface device is configured to receive at least two pixel data output from the image sensor for the same pixel position; wherein the at least two pixel data include first image data, and compress residuals corresponding to the second image data The obtained compressed data; the residual is determined based on the preset mapping relationship between the second image data and the first image data;

    The storage device is configured to store a computer program executed by the processing device, the computer program executing the following steps when executed by the processing device: obtaining the pixel positions based on the first image data and the compressed data pixel value.
40. The image processor according to claim 39, wherein the computer program is executed by the processing device to obtain the pixel value of the pixel position based on the first image data and the compressed data, include:

    The first image data is mapped to derived data based on the preset mapping relationship, the sum of the derived data and the decoded data of the compressed data is calculated to obtain the second image data, and the first image data is One image data and the second image data are fused to obtain the pixel value of the pixel position.
41. An electronic device, characterized in that the electronic device comprises an image sensor and an image processor, wherein:

    the image sensor comprising the image sensor of any of claims 36-38;

    The image processor includes the image processor of claim 39 or 40.
42. A storage medium, characterized in that a computer program is stored on the storage medium, and when the computer program is executed by a processor, the computer program causes the processor to execute the data according to any one of claims 31-35 Approach.

Images