EP2871634A1

EP2871634A1 - Color improvement of selected part of an image

Info

Publication number: EP2871634A1
Application number: EP20130306523
Authority: EP
Inventors: Jurgen Stauder; Patrick Morvan; Emmanuel Jolly
Original assignee: Thomson Licensing SAS
Current assignee: Thomson Licensing SAS
Priority date: 2013-11-06
Filing date: 2013-11-06
Publication date: 2015-05-13

Abstract

This method comprises notably the steps of:
- color transforming the selected part of this image,
- color calibrating these selected part such that a first display device reproduces approximately the same colors as a second display device,
- replacing the selected part of this image by this transformed and calibrated selected part.

Description

Technical Field

The invention is directed to methods and systems for color transformation of image adapted to provide predictable results on display devices having different color rendering characteristics, notably different color gamuts.

Background Art

Display device technologies and with them the available color gamuts evolve regularly. For example, CRT displays were replaced by LCD and PLASMA displays around the year 2000. Later, classical LCD displays were replaced by LED backlit LCD displays. In near future, LCD displays will likely be replaced by OLED displays.
Each introduction of a new display device technology brings a change of the color gamut that the display device can generate. For example, PLASMA displays exhibit a very strong white level. RGB LED backlit LCD display devices show a color gamut with much more saturated colors compared to white LED LCD display devices. OLED display devices will bring large luminance contrast values along with saturated colors.
As detailed below, three problems are introduced each time when a display technology changes.
The first problem is that display devices with new technology are generally available only in small sizes, i.e. for small TV sets, for tablets, and for mobile phones, and are only later available in larger sizes, notably for most common TV sets and color monitors. The late availability of display devices with larger size does not allow exploiting on these devices the specific features of this new technology immediately when this technology is available. Then, a user has only two choices: either use the same old display technology with lower color characteristics on any size of display devices or use the new display technology with higher color characteristics but with a limitation to small size.
The second problem is that a new display technology requires a calibration of color reproduction. Colors should be reproduced on the display of new technology in a way to fully exploit the color gamut of this technology while keeping a reasonably close look to the image shown on a post-production reference display of an older technology for which the image has been created by the content creator. Such an exploitation of larger color gamut generally implies color mapping of colors from the gamut of the post-production reference display into the gamut of the display of the new technology. Therefore, color calibration of the display of new technology is needed. The color calibration of the display of new technology means that a color on the display of new technology should have the same XYZ coordinates as the corresponding color on the reference display of the older technology. Each time, a new display technology is available, is brings one or both of the following challenges for color calibration. The first challenge is to use the new color options (for example lower black level, better saturation) in order to reproduce colors with same colorimetry. The second challenge is to use these new options (for example higher contrast values) in order to generate appealing images.
The third problem is the consistence of colors shown on the display devices of two different technologies, for example a new technology and a former technology. When changing a rendering intent from a first rendering intent to a second one, the colors on both displays will change. It might be, notably, that a first rendering intent gives acceptable results on both displays while a second rendering intent gives better results on the first display and worse results on the second display. A colorist who is looking for the "best" rendering intents for the two displays might have checked a large range of colors and a series of rendering intents in order to find a result that is satisfying for him. He also has so compare each time the two displays in order to verify the colors. If the two displays are of different technologies, for example a new technology and a former technology, and if the displays are of different sizes, the work may become quite complex for the colorist.

Summary of invention

The invention addresses the three problems above, notably in the following way: using a first display device of an old technology and with a large size to display large images, using a second display device of a new technology and with a smaller size to display selected parts of the large images displayed on the first display device, displaying these selected parts on the second display device and taking advantage of the larger possibilities of color reproduction of this second display device to transform colors of these selected parts, and transferring back the color transformation of these selected parts to these images to display improved versions of the large images on the first display device.
For this purpose, the subject of the invention is a method of generating a second version of a first image from a first version of said image, comprising the steps of:

selecting a portion of the first version of said first image, resulting in a first version of a second image,
color transforming the device dependent color coordinates representing the first version of said second image into transformed device dependent colors coordinates representing a second version of said second image,
color calibrating these transformed device dependent color coordinates into calibrated device dependent color coordinates such that a first display device controlled by said calibrated device dependent color coordinates reproduces approximately the same colors as a second display device controlled by said transformed device dependent color coordinates,
in said first image according to its first version, replacing the device dependent color coordinates representing said selected portion by the corresponding calibrated device dependent color coordinates representing the second version of this second image, resulting in said second version of said first image.

The selection of a portion of the first version of the first image corresponds generally to a cropping action.
Preferably, the device dependent color coordinates representing the first version of the second image are obtained through a color calibration of the original device dependent color coordinates corresponding to the selected portion of the first version of the first image, said color calibration being performed such that the second display device controlled by these device dependent color coordinates representing the first version of the second image reproduces approximately the same colors as the first display device controlled by these original device dependent color coordinates. Such a calibration step before the color transformation step is optional. Alternatively, there is no calibration step, and the device dependent color coordinates representing the first version of the second image are directly the original device dependent color coordinates.
Preferably, the step of color transforming is performed according to a parametric model with predetermined color correction parameters. Generally, a user chooses the value of these color transformation parameters that defines how colors are reproduced on the second display device. These values may be notably chosen according to a specific rendering intent, for example for preserving the colorimetry or color appearance, and/or for extending the color gamut, and/or for optimizing color saturation.
Preferably, the step of color transforming is performed by retouching a reproduction of the first version of the second image on the second display device. Usual color retouching tools can be used for this purpose. In such an embodiment, the second display device is used as a proof-viewing display device.
Preferably, the step of selecting a portion of the first version of the first image comprises itself a step of reproducing this first version on said first display device and a step of pointing within the reproduction of said first version.
In a first variant, for such a pointing step, a camera is used. Preferably, this camera is part of the second display device.
In a second variant, for such a pointing step, a gyroscopic remote control set adapted to control the first display device is used, or a mouse connected to the first display device is used.
Preferably, the method according to the invention comprises also the steps of:

reproducing said first and second versions of the first image using said first display device,
reproducing said first and second versions of the second image using said second display device.

In such a situation, the first display device would be a main display as a TV set and the second display device would be a secondary display as a tablet.
An object of the invention is also a system for generating a second version of a first image from a first version of said image, comprising:

means for selecting a portion of the first version of said first image, resulting in a first version of a second image,
means for color transforming the device dependent color coordinates representing the first version of said second image into transformed device dependent colors coordinates representing a second version of said second image,
means for color calibrating these transformed device dependent color coordinates into calibrated device dependent color coordinates such that a first display device controlled by said calibrated device dependent color coordinates reproduces approximately the same colors as a second display device controlled by said transformed device dependent color coordinates,
means for replacing, in said first image according to its first version, the device dependent color coordinates representing said selected portion by the corresponding calibrated device dependent color coordinates representing the second version of this second image, resulting in said second version of said first image.

Brief description of drawings

The invention will be more clearly understood on reading the description which follows, given by way of non-limiting example and with reference to the appended figures in which:

Figure 1 is a diagram of a main embodiment of the method according to the invention;
Figure 2 is a schematic of a system adapted to implement the main embodiment of figure 1.

Description of embodiments

It will be appreciated by those skilled in the art that the diagram presented on figure 1 represents conceptual views of illustrative circuitry embodying the invention. It may be substantially represented in computer readable media and so executed by a computer or processor, whether or not such computer or processor is explicitly shown. The functions of the various elements shown in the figures may be provided through the use of dedicated hardware as well as hardware capable of executing software in association with appropriate software.
A first display device and a second display device have different color reproduction characteristics, notably different color gamuts. The invention is typically implemented by using an LCD TV set as a first display device and an OLED tablet as a second display device. The LCD TV set displays images and portions of these images can be reproduced on the OLED tablet. Due to its larger color characteristics, the OLED tablet allows to discover brilliant colors on some regions of these images, that are not shown as brilliant on the LCD TEV set. When changing the rendering intent using a color correction software run on the OLED tablet, the transformed colors can be pasted on the LCD TV set such as if the tablet was a brush.
In a main non-limiting embodiment, out of the above display devices, the system used to implement the invention comprises:

an image server configured to distribute images to the first display device though a video network; it could be for instance a hard disk drive connected to the first display device through a WiFi network.
a pointing device configured to point a zone of an image displayed on the first display device, here a camera belonging to the second display device. A gyroscopic remote control device or a mouse connected to the first display device can be used instead of this camera.
a position measuring device configured to measure the relative position, orientation and size of the camera belonging to the second display device with respect to the first display device.
image cropping and pasting means configured to crop and paste an image reproduced on the first display device according to the positions measured by the position measuring device; such means are generally implemented by a software run on a CPU.
a first and second color calibrators, also implemented by a software run on a CPU. These color calibrators are notably configured to implement color management using the methods of the International Color Consortium (ICC) as defined in the specification ICC.1:2004-10 (Profile version 4.2.0.0). Such a method is detailed below.
a color transformer configured to transform colors of an image according to color grading instructions sent by a user. Usual color correction softwares implemented on a CPU can be used, including those using a "color brush" or "paint brush".

The system used to implement the invention generally comprises at least a CPU to implement all the above softwares, with its usual peripheral devices such as memories and interfaces, connected through a bus. Such a CPU with peripheral devices is preferably integrated in the second display device and its camera on a specific device like a tablet or a smartphone.
A method implementing the invention on the above system will now be described.
The image server distributes a first version of a first image to the first display device. This first version is then reproduced on this first display device by controlling this first display device with original device dependent color coordinates as distributed by the server.
Using the pointing device, a user then selects a portion of this image, as reproduced on this first display device. When a camera is used for pointing, the position measuring device measures the relative position, orientation and viewing angle (size of portion) of this camera, i.e. of the second display device, with respect to the first display device. Such pointing methods with measurement of relative position are described for instance in US2006-038833 , US2009-174653 or US2012-214591 . The selection performed by the user corresponds notably to a zone of color interest, as appreciated by this user. Then, using the image cropping means, the first image is cropped according to the selected portion, resulting in a first version of a second image.
Then, using the first color calibrator, device dependent color coordinates representing the first version of this second image are obtained through a color calibration of the original device dependent color coordinates that are used to control the first display device in order to reproduce, as explained above, the selected portion of the first version of the first image. The device dependent color coordinates which are obtained through this calibration are then such that the second display device controlled by these device dependent color coordinates reproduces approximately the same colors as the colors shown on the first display device controlled by the original device dependent color coordinates before their calibration. In brief, it means that the second display device is color calibrated by the first color calibrator such that the second image can be reproduced on this second display device as shown within the first image reproduced on the first display device. This calibration is generally limited by the capacities of the second display device. First, colors outside the color gamut of this second display device cannot be reproduced such as on the first display device. Second, precision limits of the second display device such as quantization of encoded color coordinates RGB for red, green and blue, can lead to slight color derivations when compared to the first image reproduced on the first display device.
Preferably, a color calibration using the methods of the International Color Consortium (ICC) such as defined in the specification ICC.1:2004-10 (Profile version 4.2.0.0) entitled "Image technology colour management - Architecture, profile format, and data structure" is used.
The ICC profile according to this specification allows referring device-dependent display input colors to a Profile Connection Space (PCS). The PCS is intended to enable the user of ICC profiles to determine or compensate colors differences. Among the types of profiles, we choose the display profile in this implementation. The profile can include:

1D curves or LUTs;
Linear matrices:
Regular N-to-M LUTs, up to 2563 entries, 8 or 16 bit precision;
Black and white point of the display;
For emitting displays: Luminance level [Cd/m2];
Viewing condition;
Illuminant and surround XYZ [Cd/m2];
Device technology.

However, ICC profile PCS is based on CIE 1931 XYZ color space (or CIELAB derived from it). The following known limitations of XYZ are inherent in ICC profiles:

Viewing field of 2 degrees;
Photopic human vision;
Limited to the metamerism of a small group of human observers.

In this implementation, the first and second display devices are each described by an ICC profile. Each profile has a forward and an inverse color transform. For the ICC profile, the following parameters are preferably chosen:

The rendering intend is chosen to be the media-relative colorimetric rendering intend. The media white point (ICC mediaWhitePointTag) is set to D60.
The ICC Profile Connection Space (PCS) defines the color space used for the device independent colors. ICC profiles allow for CIEXYZ and CIELAB space. In this implementation, if not otherwise mentioned, CIEXYZ color space is used.
The profile type is a display profile containing a forward transforms for perceptual and colorimetric rendering intent (AToB0Tag and AtoB1Tag tags, respectively) as well as inverse transforms for perceptual and colorimetric rendering intent (BToA0Tag and BtoA1Tag tags, respectively).
Each color transform is realized either by a LUT with 3x8 bit LUT entries (ICC lut8Type structure), 3x16 bit LUT entries (ICC lut16Type structure) or a combination of 3D LUT, 1D LUTs, linear matrix or/and parametric transfer functions (ICC lutAtoBType structure). However, in this implementation, 3x16 bit LUT entries (ICC lut16Type structure) are used.

For this color calibration, the first color calibrator applies the forward color transform of the ICC profile of the first display device and the inverse transform of the ICC profile of the second display device.
Such a calibration step before the color transformation step is optional, as shown on the figures by using dotted lines instead of continuous lines. Alternatively, the device dependent color coordinates that are used to reproduce the selected portion of the first image on the first display device are used, without color calibration, as the device dependent color coordinates representing the first version of the second image. In this case, the second display devices controlled by the device dependent color coordinates representing the first version of the second image would not reproduce the same colors compared to the first image reproduced on the first display device.
This first version of the second image is then reproduced on the second display device by controlling it using the device dependent color coordinates representing the first version of the second image, as above color calibrated when they are. Displaying this first version is optional but advantageous for the next step.
Using the color transformer, these device dependent color coordinates representing the first version of the second image are then transformed into transformed device dependent colors coordinates that represents a second version of the second image. Preferably, a color correction software allowing control of hue, saturation and brightness of the second image, globally or locally in the second image, globally for all colors or specifically for a certain range of colors, is used. Then, looking at the second image as reproduced on the second display device, the user uses the control of hue, saturation and brightness to find the color transformation of the colors according to his intent. In such a situation, any time the user modifies hue, saturation and brightness of the second image, a second version of this second image is displayed on the second display device, up to a last modification of hue, saturation and brightness, resulting in a final version of the second image.
Using the second color calibrator, the final transformed device dependent color coordinates that are obtained and used to reproduce as above the final second version of the second image are color calibrated into calibrated device dependent color coordinates resulting into a calibrated final second version of the second image such that the first display device controlled by these calibrated device dependent color coordinates reproduces approximately the same colors as those shown on this second display device. The same method using the specification ICC.1:2004-10 (Profile version 4.2.0.0) is preferably used. For this color calibration, the second calibrator applies the color forward transform of the second display device and the inverse transform of the first display device. In brief, it means that the first display device is color calibrated by the second color calibrator such that the second version of the second image can be reproduced on this first display device as it is reproduced on the second display device after the last modification of hue, saturation and brightness. This calibration is again limited by the capacities of the first display device. First, colors outside the color gamut of this first display device cannot be reproduced such as on the second display device. Second, precision limits of the first display device such as quantization of encoded color coordinates RGB for red, green and blue, can lead to slight color derivations when compared to the transformed device dependent colors coordinates of the final second version of the second image reproduced on the second display device.
Then, using the image pasting means, this calibrated final version of the second image is pasted in the first image at the same position at which the second image has been cropped, such a position being given by the position measuring device, as described above. During such a pasting operation, the original device dependent color coordinates representing the selected portion are replaced by their corresponding calibrated device dependent color coordinates. Such a pasting results in a second version of the first image.
The last step is then to reproduce this second version of the first image on the first display device, using, to control it the device dependent color coordinates that are color calibrated after transformation, as described above. Due to this color calibration, the brilliant colors that have been discovered by the user on the first image and transformed on the second display device that has higher color capabilities compared to the first display device, are corrected and reproduced on the first display device.
Therefore, using the method as implemented above, the display devices of old technology and large size can take benefit of the higher color characteristics of display devices of new technology even of smaller size. Therefore, it is possible to take advantage of a new display technology of display device as soon as it is available. This invention can be advantageously valorized through software application downloaded from specific internet sites dedicated to new technology display devices. This invention can also advantageously valorized in postproduction of content for new technology display devices however, using the large size of display devices of old technology.
Although the illustrative main embodiment of the invention have been described herein with reference to the accompanying drawings, it is to be understood that the present invention is not limited to this precise embodiment, and that various changes and modifications may be effected therein by one of ordinary skill in the pertinent art without departing from the invention. All such changes and modifications are intended to be included within the scope of the present invention as set forth in the appended claims.
It is to be understood that the invention may be implemented in various forms of hardware, software, firmware, special purpose processors, or combinations thereof. The invention may be notably implemented as a combination of hardware and software. Moreover, the software may be implemented as an application program tangibly embodied on a program storage unit. The application program may be uploaded to, and executed by, a machine comprising any suitable architecture.

Claims

Method of generating a second version of a first image from a first version of said image, comprising the steps of:
- selecting a portion of the first version of said first image, resulting in a first version of a second image,

- color transforming the device dependent color coordinates representing the first version of said second image into transformed device dependent colors coordinates representing a second version of said second image,

- color calibrating these transformed device dependent color coordinates into calibrated device dependent color coordinates such that a first display device controlled by said calibrated device dependent color coordinates reproduces approximately the same colors as a second display device controlled by said transformed device dependent color coordinates,

- in said first image according to its first version, replacing the device dependent color coordinates representing said selected portion by the corresponding calibrated device dependent color coordinates representing the second version of this second image, resulting in said second version of said first image.
Method according to claim 1 wherein the device dependent color coordinates representing the first version of the second image are obtained through a color calibration of the original device dependent color coordinates corresponding to the selected portion of the first version of the first image, said color calibration being performed such that the second display device controlled by these device dependent color coordinates representing the first version of the second image reproduces approximately the same colors as the first display device controlled by these original device dependent color coordinates.
Method according to claim 1 or 2 wherein the step of color transforming is performed according to a parametric model with predetermined color correction parameters.
Method according to any one of claims 1 to 3 wherein the step of color transforming is performed by retouching a reproduction of the first version of the second image on the second display device.
Method according to any one of claims 1 to 4 wherein the step of selecting a portion of the first version of the first image comprises itself a step of reproducing this first version on said first display device and a step of pointing within the reproduction of said first version.
Method according to any one of claims 1 to 5 comprising the steps of:
- reproducing said first and second versions of the first image using said first display device,

- reproducing said first and second versions of the second image using said second display device.
System for generating a second version of a first image from a first version of said image, comprising:
- means for selecting a portion of the first version of said first image, resulting in a first version of a second image,

- means for color transforming the device dependent color coordinates representing the first version of said second image into transformed device dependent colors coordinates representing a second version of said second image,

- means for color calibrating these transformed device dependent color coordinates into calibrated device dependent color coordinates such that a first display device controlled by said calibrated device dependent color coordinates reproduces approximately the same colors as a second display device controlled by said transformed device dependent color coordinates,

- means for replacing, in said first image according to its first version, the device dependent color coordinates representing said selected portion by the corresponding calibrated device dependent color coordinates representing the second version of this second image, resulting in said second version of said first image.