After an evaluation, GNOME has moved from Bugzilla to GitLab. Learn more about GitLab.
No new issues can be reported in GNOME Bugzilla anymore.
To report an issue in a GNOME project, go to GNOME GitLab.
Do not go to GNOME Gitlab for: Bluefish, Doxygen, GnuCash, GStreamer, java-gnome, LDTP, NetworkManager, Tomboy.
Bug 492048 - Detect color space in Exif 2.21/DCF 2.0 option files
Detect color space in Exif 2.21/DCF 2.0 option files
Status: RESOLVED FIXED
Product: GIMP
Classification: Other
Component: libgimp
2.4.x
Other All
: Normal enhancement
: 2.10
Assigned To: GIMP Bugs
GIMP Bugs
: 580842 (view as bug list)
Depends on:
Blocks: 118202
 
 
Reported: 2007-10-31 11:38 UTC by Christoph Moench-Tegeder
Modified: 2015-10-08 09:57 UTC
See Also:
GNOME target: ---
GNOME version: ---


Attachments
Helper script for installing the Adobe ICC profiles (1.11 KB, text/plain)
2007-10-31 22:09 UTC, Christoph Moench-Tegeder
  Details
Proposed Colour Management preferences (112.01 KB, image/png)
2014-01-31 00:45 UTC, Martin Ramshaw
  Details
Recent patch against GIT code, enabling fallback ICC profiles to be attached (12.53 KB, patch)
2014-01-31 00:48 UTC, Martin Ramshaw
none Details | Review
ClayRGB1998, from the argyllcms 1.6.2 source distribution. (584 bytes, application/vnd.iccprofile)
2014-02-01 22:30 UTC, Mike Henning (drawoc)
  Details
A program that dumps the lcms sRGB profile. (282 bytes, text/x-csrc)
2014-02-02 18:41 UTC, Mike Henning (drawoc)
  Details
makes two profiles for DCF (4.47 KB, patch)
2014-05-20 20:58 UTC, Elle Stone
none Details | Review
Cleaned up patch (5.96 KB, patch)
2014-06-06 23:33 UTC, Michael Natterer
none Details | Review
Adobe DCF tif (315.02 KB, image/tiff)
2015-09-28 19:51 UTC, Elle Stone
  Details
spreadsheet showing before/after metadata (19.39 KB, application/vnd.oasis.opendocument.spreadsheet)
2015-10-01 08:53 UTC, Elle Stone
  Details

Description Christoph Moench-Tegeder 2007-10-31 11:38:19 UTC
Exif 2.21 is an update to the widely known Exif 2.2 standard, released (and hardly promoted) in 2003. The most important part of this update affects Exif tag 0x0001 "InteroperabilityIndex". With this tag set to "R03" the enclosed JPEG data is supposed to be in AdobeRGB color space, even if no ICC profile is embedded in the file. Exif 2.21/DCF 2.0 calls this a "DCF option file".
GIMP should warn the user when opening such files, so the correct profile can be applied. Optionally, GIMP could offer to use the AdobeRGB profile if it's already known in the "Apply Profile" dialog.
References:
- the exiftool documentation http://www.sno.phy.queensu.ca/~phil/exiftool/
- information about Exif 2.21/DCF 2.0
  http://www.cipa.jp/exifprint/contents_e/01exif2_2_e.html
  Sadly, the exif 2.21 standard is not freely available online.
Comment 1 Raphaël Quinet 2007-10-31 14:22:54 UTC
I would like to fix this bug, but this is not possible without a freely available version of the standard and without any test files.

The DCF documents that I saw on the CIPA web site describe changes to the file names (irrelevant when the file is transferred out of the camera) and changes to he EXIF interoperability tag, but without describing precisely what is the "correct" behavior and what to do in some corner cases.

If you have any additional information that could help us and ensure that we conform to the new EXIF standard, please add it to this bug report.
Comment 2 Christoph Moench-Tegeder 2007-10-31 18:29:16 UTC
The best documentation I could find was the exiftool documentation. Of course, JEITA has the details, but the want you to buy them here: http://www.jeita.or.jp/english/standard/list/list.asp?cateid=1&subcateid=4
Nevertheless I'm attaching two images of a bunch of pens as a testcase: first is shot in AdobeRGB, second in sRGB. I did not modify them so Exif data should be untouched.
Comment 3 Sven Neumann 2007-10-31 18:35:09 UTC
I am not really willing to support this. We can't include a copy of the AdobeRGB color profile with GIMP and without this profile we can not do anything useful. This extension sounds like a very bad idea. It doesn't add anything of value as people could just include the color profile if they want to work in a color space different than sRGB.
Comment 4 Christoph Moench-Tegeder 2007-10-31 18:40:19 UTC
OK, files are up at
http://www.burggraben.net/hacks/_mg_0989.jpg (AdobeRGB)
http://www.burggraben.net/hacks/img_0990.jpg (sRGB)
Both should confirm to Exif 2.21, first is a "DCF optional file", second the same subject, same lighting, but as a "DCF original file". Sorry, but that's all I have for now.
Comment 5 Christoph Moench-Tegeder 2007-10-31 18:45:23 UTC
(In reply to comment #3)
> I am not really willing to support this. We can't include a copy of the
> AdobeRGB color profile with GIMP and without this profile we can not do
> anything useful.

Even displaying a warnig to the user would be helpful, just to inform him that he is about to wreck his image.

> This extension sounds like a very bad idea. It doesn't add
> anything of value as people could just include the color profile if they want
> to work in a color space different than sRGB.

That's difficult, as it's the camera that writes such files. And a modern (post-2003) camera just goes for Exif 2.21 instead of including the profile with every picture taken.

Comment 6 Sven Neumann 2007-10-31 20:34:15 UTC
Ok, so then the best way to solve this is probably the following:

The JPEG load plug-in should detect the tag and open a dialog that informs the user. This dialog should have a GimpColorProfileComboBox that allows the user to select the AdobeRGB profile. Perhaps even give information how it can be obtained.

When the profile has been specified, the JPEG plug-in can use the plug-in-icc-profile-set procedure to set the profile.  The plug-in should also store this information in a persistent parasite. The next time the tag is encountered, the association of the profile can then happen without user interaction.

An alternative would be to add the adobe-rgb profile to GimpColorConfig but I am reluctant to do that.
Comment 7 Christoph Moench-Tegeder 2007-10-31 22:09:56 UTC
Created attachment 98284 [details]
Helper script for installing the Adobe ICC profiles

After the user has downloaded the profiles distribution from http://www.adobe.com/support/downloads/iccprofiles/icc_eula_unix_end.html this script can be used for installing them in either ~/.color/icc or /usr/local/share/color/icc
Comment 8 Christoph Moench-Tegeder 2007-10-31 22:16:30 UTC
(In reply to comment #6)
> Ok, so then the best way to solve this is probably the following:
> 
> The JPEG load plug-in should detect the tag and open a dialog that informs the
> user. This dialog should have a GimpColorProfileComboBox that allows the user
> to select the AdobeRGB profile. Perhaps even give information how it can be
> obtained.

Right, that's what I had in my mind.
The profiles can be obtained from Adobe at http://www.adobe.com/digitalimag/adobergb.html (Linux users chose http://www.adobe.com/support/downloads/iccprofiles/icc_eula_unix_dist.html).
Attachment 98284 [details] has a script for extracting the profiles from the rpm.tar.gz
distribution (needs rpm2cpio).

> When the profile has been specified, the JPEG plug-in can use the
> plug-in-icc-profile-set procedure to set the profile.  The plug-in should also
> store this information in a persistent parasite. The next time the tag is
> encountered, the association of the profile can then happen without user
> interaction.

That would be just great.

> An alternative would be to add the adobe-rgb profile to GimpColorConfig but I
> am reluctant to do that.

I guess Adobe's terms are too tight for the profiles to be distributed with Free Software.
> 

Comment 9 Michael Schumacher 2008-02-28 17:27:33 UTC
Is Exif 2.21 even maintained anymore? 

The pages they link to in order to obtain the docs (even if one has to pay for it) seem to be 404.
Comment 10 Michael Schumacher 2009-04-30 09:18:41 UTC
*** Bug 580842 has been marked as a duplicate of this bug. ***
Comment 11 Frank Gore 2010-01-18 00:57:20 UTC
Here's an example of how this affects us negatively:

Right now, if I open a JPG file from my Pentax K-7 that was set to sRGB, and my working space in Gimp is set to Adobe RGB, no color profile is detected and the image is assumed to be in Adobe RGB, thereby over-saturating all the colors.

The opposite is also true. If my camera was set to Adobe RGB and my working space is set to sRGB, Gimp detects no color profile in the file from my camera just assigns it an sRGB profile (which makes the colors bland and lifeless).

Sure, I can assign a different color profile manually if I know what it should be, but that's an inconvenient step to have to remember every single time. If I receive a picture file from someone else and want to edit it, I have to check what the color profile should be by wading through the EXIF metadata because Gimp doesn't detect it properly.

All of this is defeating the whole point of seamless color management.

If this was a fringe case, I could understand ignoring it. But this affects all modern digital cameras, which is a lot of potential Gimp users. All modern cameras use DCF 2.0, which means they specify the intended colorspace in the EXIF 2.21 metadata rather than embedding an ICC profile. The very first chapters of Akkana Peck's book cover image enhancements for pictures right out of the camera.

The new link to the JEITA DCF 2.0 specification is here:
http://www.jeita.or.jp/english/standard/html/1_4.html
It sucks that a specification like this needs to be purchased...
Comment 12 Michael Schumacher 2010-01-18 11:30:26 UTC
The new link is a 404.

But anyway, if the spec is not freely distributable (or is it?), then Free Software may not be able to support this at all. Has anyone approached JEITA about this problem yet?
Comment 13 Frank Gore 2010-01-18 13:22:07 UTC
JEITA Web Site:

"Our website is temporarily closed due to further investigation after we learnt, on January 15th, 2010, that we had been possibly listing affected webpages on our website."
Comment 14 Frank Gore 2010-02-03 20:25:55 UTC
The DCF 2.0 specification may be for sale from Jeita, however it's also available for download from all kinds of places all over the web. I don't know what restrictions it has, I'm no lawyer. But the folks at the LittleCMS project were well-versed in how this works. Here's an email from one of the devs there:

On Sun, Jan 17, 2010 at 7:32 AM, Larry Reeve <xxx@xxx.xxx> wrote:
> The first image is in "DCF Colorspace". This can be seen from the following
> entries in the EXIF:
>
> Color Space:    Uncalibrated
> WhitePoint X:   0.31
> WhitePoint Y:   0.33
> Chromaticity Red(X):    0.64
> Chromaticity Red(Y):    0.33
> Chromaticity Green(X):  0.21
> Chromaticity Green(Y):  0.71
> Chromaticity Blue(X):   0.15
> Chromaticity Blue(Y):   0.06
> YCbCrCoefficient 1:     0.30
> YCbCrCoefficient 2:     0.59
> YCbCrCoefficient 3:     0.11
>
> A DCF aware application sees this information and processes the image
> accordingly (which means use an Adobe RGB profile).

And a further reply when I asked which application is supposed to handle this:

On Sun, Jan 17, 2010 at 5:03 PM, Larry Reeve <xxx@xxx.xxx> wrote:
> LCMS does not have anything to do with the color space defined in the EXIF
> block of a JPEG file. There is no embedded profile. It is up to the
> application using LCMS to evaluate the color space definitions in the EXIF
> and take the action to load and use an Adobe RGB profile if it is needed.
> The DCF Optional Color Space (see paragraph 6.2 of of JEITA CP-3461 "Design
> rule for Camera File system DCF Version 2.0") defines the values that should
> alert an application to take this action:

A summary of the changes to DCF 2.0 can be found here:
http://www.cipa.jp/exifprint/contents_e/01exif_e/ExifDCFsummary_E.pdf

And as I mentioned above, the full specification can easily be found by looking up "DCF 2.0" in Google and checking the first 3 results.
Comment 15 Martin Ramshaw 2014-01-31 00:43:28 UTC
As noted, since Exif 2.2.1 embedded ICC profiles are now no longer required.

The problem can be seen in detail (along with sample files, too large to upload here) at:

    http://blog.meetthegimp.org/episode-195-whose-rgb/

The attached patch allows the user to configure fallback RGB profiles for the case where no embedded ICC profiles are found.

Which profiles these ARE is left entirely to the user; GIMP has excellent profile management tools and remembers ICC profiles.

My suggestion would be the ArgyllCMS profiles, but the problematic AdobeRGB1998 profile may also be downloaded and installed, again at user discretion.

A screen capture of the fallback RGB configuration is also attached.
Comment 16 Martin Ramshaw 2014-01-31 00:45:39 UTC
Created attachment 267694 [details]
Proposed Colour Management preferences

Fallback CM options are highlighted; also keyboard shortcuts are corrected
Comment 17 Martin Ramshaw 2014-01-31 00:48:48 UTC
Created attachment 267695 [details] [review]
Recent patch against GIT code, enabling fallback ICC profiles to be attached

Patch also adds fallback configuration options to CM preferences

Patch verified by Elle Stone
Comment 18 Martin Ramshaw 2014-02-01 03:26:59 UTC
A better solution seems to be to use lcms.c to handle the ICC profiles.

Also, probably a good idea to keep ICC profile parasite processing in lcms.c

I will proceed along these lines and post an updated patch if I make progress.
Comment 19 Mike Henning (drawoc) 2014-02-01 04:12:35 UTC
I don't think it's really necessary to have which profiles are chosen to be user configurable. ArgyllCMS ships with a profile named ClayRGB, which I believe is both AdobeRGB compatible and freely redistributable. So, we could simply ship the GIMP with that profile, and use it for these purposes.

I'd agree that this code probably doesn't belong in the jpeg plugin. lcms.c is probably a good place to put it. You might also want to look at putting this code in along with the metadata handling code. (I haven't really looked too closely at this to see where the best place for it is, so put it wherever you think is best.)

Anyway, thanks for working on this!
Comment 20 Martin Ramshaw 2014-02-01 20:42:42 UTC
Thanks for the comments Mike.

This problem is most common with JPEGs, but may occur with TIFFs and PNGs.

[Yes, I am also poking into metadata, some work has been done but far from complete]

So best to locate the processing somewhere common rather than in JPEG code.

Cameras generally offer sRGB and AdobeRGB1998 as their only options (and may or - increasingly - may not embed this colour profile in their output).

A completely sRGB work flow is fine for the web, but prints are different.

The main problem of profile proliferation is actually with sRGB (which it turns out, is very far from being standard). Choosing an AdobeRGB default is quite easy - there are only two real options (being Clay, as noted, or the encumbered Adobe). But for sRGB (which is thought to be a standard) there are at least a dozen or more valid choices. And with the advent of LCMS2, some are definitely a lot better than others.

I started off with some quick hacking to fix the simple problem, but I have been piping-up on the more long-term idea of distributing colour profiles with GIMP. Of course, this will require some general consensus, as it's not really a technical or coding issue.
Comment 21 Mike Henning (drawoc) 2014-02-01 22:28:16 UTC
LCMS has the function cmsCreate_sRGBProfile, which creates an srgb profile in memory.  Gimp already uses this function for its sRGB support, and it's probably fine to just reuse the function for this too.

I don't really think redistributing ClayRGB is much of an issue at all if we just stick it in one of GIMP's data directories. At 584 bytes, we might even want to convert it to a static array in a c file and simply compile it into the gimp. (lcms can load profiles from memory.)

Anyway, I'll attach ClayRGB.icm here for convenience.
Comment 22 Mike Henning (drawoc) 2014-02-01 22:30:09 UTC
Created attachment 267803 [details]
ClayRGB1998, from the argyllcms 1.6.2 source distribution.
Comment 23 Martin Ramshaw 2014-02-02 04:44:10 UTC
As the following article shows, perhaps not the best choice:

    http://ninedegreesbelow.com/photography/linux-icc-profiles.html#adobe-compatible

Yes to ClayRGB1998, but the argyllcms 1.6.0 version is a better choice.

The article illustrates quite nicely how profiles can 'drift' and differ.

Likewise I personally would prefer to choose my sRGB profile rather than leave it to the mercy of a function call buried deep in the code.

Colour profiles are far trickier than might be thought at first glance:

    http://www.23hq.com/mramshaw/album/6935940

There are plenty of tools for examining colour profiles, but this is not really an option unless the profile is easily available for inspection.
Comment 24 Mike Henning (drawoc) 2014-02-02 18:09:24 UTC
Ah, I didn't know about that regression in argyllcms.

Anyway, you can pretty easily save the lcms srgb profile in three lines of code. (See attached program). If there's something wrong with that profile (which Elle's article suggests), I'd generally prefer that you file a bug against lcms and get it fixed there instead of shipping a separate profile with the gimp.
Comment 25 Mike Henning (drawoc) 2014-02-02 18:41:32 UTC
Created attachment 267858 [details]
A program that dumps the lcms sRGB profile.
Comment 26 Martin Ramshaw 2014-02-02 21:50:04 UTC
Thanks for the comments, but this bug is NOT about bundling colour profiles.

For comments about bundling colour profiles, either Bug 344525 or 608961.

For a number of users, their camera does not embed colour profiles which makes interpreting the images difficult. Even if they select sRGB in-camera, the choice of picture style may well mean that the actual profile *used* in the picture is a very strong departure from so-called 'standard' sRGB. In this case using any 'standard' sRGB will give incorrect results. For this reason, the user needs to be able to choose *which* sRGB to use when interpreting the image.

[And I believe all of the above also applies to AdobeRGB.]

The code I submitted allows (but does not require) default ICC profiles to be used if an embedded profile is not found. It allows the user to choose the sRGB profile (and if one is not specified I believe Gimp already uses an internal sRGB as Mike describes); it also allows the user to specify which Adobe-compatible profile to use - which can be either Clay or Adobe (if I was sending my pictures to a high-end commercial print shop, I would not use Clay - even though it is functionally equivalent to Adobe).
Comment 27 Elle Stone 2014-02-04 22:49:49 UTC
The problem with the Argyllcms ClayRGB1998.icm and sRGB.icm profiles doesn't affect profiles from Argyllcms 1.6.0 and before, and is fixed in version 1.6.3 and following. There was a tiny, tiny variation in some of the values in the 1.6.1 and 1.6.2 colorant tags. The 1.6.3 profile is exactly right, as were the profiles from 1.6.0 and before.

As far sRGB goes, there are two problems. The first problem is that profiles actually use hexadecimal values, so there is quantization during the profile making process, which can be compensated for. The Argyllcms sRGB and ClayRGB1998 profiles (except 1.6.1 and 1.6.2) do take hexadecimal rounding into account.

The other problem is different profile makers use different values for the D65 source white point. The Argyllcms sRGB and ClayRGB1998 profiles both use x=0.3127, y=0.3290. The other set of D65 values is x=0.3127, y=0.3291. Just the y value is different.

These two references quote both sets of xy values right on the same page, in two different tables, each time saying the values are the CIE values for D65:
http://www.w3.org/Graphics/Color/sRGB
http://www.color.org/sRGB.xalter

From Table  0.1 and notes:
Illuminant White x = 0.3127, y = 0.3291 (D65) The chromaticities of the illuminant white are those of CIE D65.

Farther down the page, Table 0.2:
The CIE chromaticities for the red, green, and blueITU-R BT.709 reference primaries, and for CIE Standard Illuminant D65, are given in Table 0.2 [which says]
D65 x 0.3127 y 0.3290 

sRGB is supposed to be compatible with Rec709. 
Poynton gives x=0.3127 y=0.3290 as the right values for D65.
(http://www.poynton.com/PDFs/ColorFAQ.pdf, Q17).

This page: 
http://www2.units.it/ipl/students_area/imm2/files/Colore1/sRGB.pdf
gives x = 0.3127, y = 0.3290. 

This page from the wayback machine's archive of the original sRGB color space website:
http://web.archive.org/web/20030212204955/http://www.srgb.com/basicsofsrgb.htm
gives x = 0.3127, y = 0.3290.

The AdobeRGB specifications give x = 0.3127, y = 0.3290. As AdobeRGB reputedly was derived by accident from sRGB by modifying the green primary, it seems like they ought to use the same source white point values.

The ICC spec version 4 gives X=0.9505 Y=1 Z=1.0890, which when converted to xyY is x=0.31271591, y=0.32900148, which rounded to 4 decimal places is x=0.3127, y=0.3290

The LCMS2 API gives x=0.3127, y=0.3291. I will ask on the lcms mailing list about this.
Comment 28 Elle Stone 2014-02-06 18:28:48 UTC
I did ask on the lcms mailing list about the right source white point for making an sRGB profile:
http://sourceforge.net/mailarchive/forum.php?thread_name=52F26705.3000701%40ninedegreesbelow.com&forum_name=lcms-user
Comment 29 Michael Natterer 2014-05-19 01:39:17 UTC
Can I please get some *short* summary about what the outcome of above
discussion is, if any?
Comment 30 Elle Stone 2014-05-20 17:27:51 UTC
*Something* needs to be done as DCF tags are commonly used with camera generated jpegs/tiffs, which might in turn have been exported as some other file format by some other image editor. So it's not just jpegs.

Martin Ramshaw's patch worked. It nicely allowed the user to choose an appropriate default profile from disk, if they had one.

Some users really will want to assign a profile from disk rather than have GIMP make/supply a profile. Others will want GIMP to take care of the matter.

I submitted elsewhere the right values to use with LCMS to make an AdobeRGB1998-compatible profile. Also I submitted the line of code that tells LCMS to make a V2 profile, which is better for embedding in images even at this point in time, for the sake of backwards compatibility with older editing software.

Also the ArgyllCMS profiles are freely distributable and perfect. Just use profiles that are from ArgyllCMS 1.6.2 or later. So distribute ArgyllCMS or make on the fly with LCMS, whichever is easier, the end result is the same.

Once assigned, the image should be saved with the profile embedded, whether sRGB or AdobeRGB. Otherwise the next editing program will have the same problem of dealing with the DCF information.

It can't be a seamless, painless operation because GIMP has no way of knowing what other editing program might have handled the file without also embedding an appropriate ICC profile. So there needs to be appropriate user options and popups to let the user know what's going on. Blame the camera industry for complicating color management by not embedding actual ICC profiles.

The DCF information is part of the image's "from the camera metadata" and so shouldn't be modified/updated/removed/tampered with in any way, even if the image is converted to some entirely other ICC profile. But the new ICC profile must be embedded.

If there is DCF information and also an embedded ICC profile, completely ignore the embedded DCF information and use the ICC profile. If the DCF happens to be right and the ICC profile wrong, that's a user problem to discover and solve. Always asking about DCF when there is also an embedded profile would be annoying.

That's my best (not short enough) summary!
Comment 31 Elle Stone 2014-05-20 17:30:26 UTC
(In reply to comment #30)

> Also the ArgyllCMS profiles are freely distributable and perfect. Just use
> profiles that are from ArgyllCMS 1.6.2 or later. 

Sorry, that should be 1.6.3 or later.
Comment 32 Michael Natterer 2014-05-20 18:42:45 UTC
Ok I see, I think :)

Could you as a first step perhaps create functions in libgimpcolor/gimplcms
in the spirit of gimp_lcms_create_srgb_profile() which create these
clearly defined profiles? I strongly tend to generate them on the fly
rather than depending on constant files that have to be shipped and can
be messed up by the user.

I think you are the person best qualified for that task around here :)
Comment 33 Elle Stone 2014-05-20 20:58:23 UTC
Created attachment 276894 [details] [review]
makes two profiles for DCF

This patch isn't functional. It just adds two functions that make V2 Adobe and sRGB profiles for embedding in DCF files. I tested that they were created correctly by putting in write statements and calling the functions from gimp_lcms_create_srgb_profile ().
Comment 34 Michael Natterer 2014-06-06 23:33:14 UTC
Created attachment 278063 [details] [review]
Cleaned up patch

What now?
Comment 35 Elle Stone 2014-06-07 12:07:38 UTC
Here's a starting point for thinking about what next. Hopefully other people will check to see if this covers all the bases:

1. Upon opening the image, if there's an ICC profile embedded, use that profile. Even if there is contradictory DCF information, use the embedded ICC profile.

2. If there is DCF information and no embedded ICC profile, tell the user what the DCF information says and tell them a GIMP-generated profile has been assigned:

"This image doesn't have an embedded ICC profile. DCF information says it's an Adobe RGB (1998) (or sRGB as the case may be) image. An Adobe RGB (1998)-compatible (or sRGB) profile has been assigned to the image." The word "compatible" is necessary to avoid infringing Adobe's trademark.

3. Ask the user if they would like to assign a different ICC profile from disk instead? If yes, open the appropriate dialog.

4. Ask the user if they would like to export/overwrite the image and embed the assigned profile.

If yes, export the image with the assigned profile embedded. If no, it might be nice to tell the user that this dialog will pop up every time they open that particular image, until such time as an ICC profile is embedded.

Martin's patch allows the user's answers to questions 2/3 to be given in advance. For people who routinely open camera-generated jpegs/tiffs/etc, Martin's patch would be a great convenience.

It might be the case that exporting/overwriting jpegs with the newly assigned ICC profile embedded could have adverse affects on jpegs. Exiftool can embed an ICC profile in a jpeg without causing any additional artifacts from compression. I don't know whether GIMP/exiv2 can do this or not.

Martin's patch of course can't address the possibility that in some cases the user's pre-chosen ICC profile might be wrong for the particular image. In this latter case it would be up to the user to notice that something was wrong and assign the right ICC profile.

Again, blame the camera manufacturers for deciding that DCF was an adequate substitute for a real ICC profile.
Comment 36 Michael Natterer 2015-08-19 16:40:49 UTC
Before we proceed here I have a question:

The patch adds two new functions named something_dcf_something(). The first
one seems to create a profile identical to the GIMP built-in sRGB profile,
which we nowadays get from gimp_color_profile_new_srgb(). The second function
creates an AdobeRGB profile but has also "dcf" in the name. Is that profile
specific to DCF files or just the standard AdobeRGB? I assume it is the
standard one and not DCF-specific.

I propose to simply add *one* function called

gimp_color_profile_new_adobe_rgb()

and be done, then re-visit the other patch which assigns profiles
based on the image tags.
Comment 37 Elle Stone 2015-08-19 19:12:57 UTC
A function to make an AdobeRGB1998-compatible profile is all that's needed. 

I put "dcf" in the function names to indicate that the profiles were being used with camera-generated jpegs with DCF information, a not very successful attempt at code clarification. These camera jpegs with DCF information need the standard sRGB or AdobeRGB1998 profile assigned.

Originally I suggested creating and embedding V2 profiles to accomodate software that can't read V2 profiles. But that's not a good idea, at least for sRGB when dealing with higher-than-8-bit image depths, because of how LCMS treats point-curve TRCs. Arguments could be made either way, but at this point I'd recommend making and embedding V4 profiles and letting the user sort out any issues that might ensue when dealing with V2-only software.
Comment 38 Michael Natterer 2015-08-19 21:12:37 UTC
Ok, done. Now for the actual bugfix...

commit 9fc64e3664ab84b75bd21f53d405aa21941a0d04
Author: Michael Natterer <mitch@gimp.org>
Date:   Wed Aug 19 23:10:56 2015 +0200

    Bug 492048 - Detect color space in Exif 2.21/DCF 2.0 option files
    
    Add gimp_color_profile_new_adobe_rgb(), will need this later to fix
    this bug.

 libgimpcolor/gimpcolor.def      |  1 +
 libgimpcolor/gimpcolorprofile.c | 78 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 libgimpcolor/gimpcolorprofile.h |  2 ++
 3 files changed, 81 insertions(+)
Comment 39 Michael Natterer 2015-08-19 21:29:22 UTC
Of the huge patch, we should IMO ignore the whole preferences and config
part and simply do the easiest thing:

1. if there is an ICC profile, use it, the apply profile dialog will
   handle the rest (and is to be re-done anyway)

2. if DCF info says sRGB, attach no profile, this is our default anyway

3. if DCF info says AdobeRGB, attach the profile created with
   gimp_color_profile_new_adobe_rgb(), do nothing else and continue with 1.

What profile to export is an entirely different issue, first we must get
the pixels correctly imported.
Comment 40 Michael Natterer 2015-08-20 09:49:15 UTC
I was just about to add the few needed lines to jpeg-load.c when I realized
that this should probably go into the generic metadata handling code
in gimp_image_metadata_load_finish() so it is executed for all file formats
that have this Exif tag set.

Or is this a JPEG-specific Exif tag?
Comment 41 Elle Stone 2015-08-20 11:42:54 UTC
Unfortunately it's not a JPEG-specific EXIF tag. 

*  A few cameras do allow to save in-camera-interpolated files as tiffs as well as jpegs. Possibly some cameras allow saving in pngs, etc. So generically one could speak of "in-camera-interpolated files", which are usually jpegs, but might be some other file format.

*  Also the same DCF information that is embedded in in-camera-interpolated files, is also embedded in raw files, should the user choose to save images as raw files instead of (or in addition to) saving in-camera-interpolated files. For example, for a sample raw file (only saved as a raw file - no accompanying jpeg/etc), here's what exiftool reveals:

exiftool -a -S -G0 -ColorSpace -InteropIndex -WhitePoint -PrimaryChromaticities -Gamma -ICC_Profile:all *.cr2
[MakerNotes] ColorSpace: Adobe RGB
[EXIF] ColorSpace: Uncalibrated
[EXIF] InteropIndex: R98 - DCF basic file (sRGB)

Notice the MakerNotes contradicts the EXIF DCF information. I'm not sure how exiv2 deals with makernotes, but obviously contradictions between maker notes and standard DCF adds more room for error when dealing with DCF information.

*  Most editing software, including raw processing software, rightly leaves the original EXIF data alone except for user-initiated changes. So for example the user might open an sRGB camera jpeg, maybe convert it to some other color space, and save it in a new file format that might or might not be able to embed ICC profiles, and/or the newly embedded ICC profile might have been stripped out somewhere else along the line, but the EXIF data wasn't stripped out. 

Some aspects of the problem to consider:

(1) Having GIMP handle DCF information by automatically assigning either a built-in sRGB or a built-in AdobeRGB1998, for images that have DCF information but no embedded ICC profile, will be very helpful for users who routinely produce in-camera-interpolated jpegs/etc. 

(2) But even for these users, for files that do have DCF information and don't have embedded ICC profiles, the embedded DCF information might not be correct for all images the user might open.  

(3) You might assume the user will instantly know if the wrong ICC profile has been assigned because the colors will look wrong. But depending on the image colors, the "wrongness" can be subtle, but still lead to editing difficulties farther down the line. 

(4) There are good reasons why a user might want to assign their own profiles from disk, even for AdobeRGB1998 and sRGB images, which I can enumerate if you like. Personally I would always choose to assign a profile from disk because my profileless images (usually interpolated raw files with still-embedded DCF information) are almost never actually in the DCF-indicated color space, and even when they are, I still prefer to assign a profile from disk.

A popup dialog could addess these problems by notifying the user that "This image doesn't have an embedded ICC profile. Based on DCF information, an internal sRGB/AdobeRGB1998 profile has been generated and assigned." User choices in the popup would be to either keep the assigned profile or choose a profile from disk. I think such a popup is pretty important to avert cases where the user doesn't realize the wrong profile has been assigned.

Someone who opens a lot of "in-camera-interpolated" files is going to get tired of dealing with the constant popups. It's been a long time since I looked at the user options added by Martin's patch, but as I recall they did allow preselected choices that nicely covered cases where the user didn't want to use a built-in sRGB or AdobeRGB1998 profile, but rather preferred to assign their own profiles from disk. 

We could deal with DCF-related Preferences/Color Management UI changes, along with changes such as dealing with copy/paste/etc, in an "uber-color-management-preferences" bug report that links back to this bug report. Should I file such a bug report?
Comment 42 Michael Natterer 2015-08-20 12:45:26 UTC
Thanks Elle, that sheds some light around here. I think we're dealing
with two problems here:

1. interpret whatever colorspace the file and its metadata thinks
   its pixels are, and assign a profile, not bothering the user with
   questions and stuff, because it's unclear if she's able to resolve
   inconsistencies in the metadata at all. This is what *this* bug is
   about.

2. once the image is loaded, present the user with the to-be-improved
   "Apply color profile" dialog, this is what bug 555562 is about, or
   what I made it to be about :)

I'm going to push a patch that deals with Exif.Iop generically in
gimp_image_metadata_load_finish(), but in a pretty simplistic manner,
similar to the original patch attached here.

When searching the web for Exif.Iop, i came across this bug:

https://bugs.kde.org/show_bug.cgi?id=191113

They deal with the many different colorspace informations that can be
present this way:

https://websvn.kde.org/trunk/KDE/kdegraphics/libs/libkexiv2/libkexiv2/kexiv2image.cpp?view=markup&pathrev=1019840

Look at the function starting at line 377. Could I have your opinion
on that logic? It looks pretty sane and would be easy to implement in GIMP.
Comment 43 Michael Natterer 2015-08-20 13:03:03 UTC
This git repo seems to have more recent and improved code in that function:

https://github.com/veaceslav/Exiv2-hacking/blob/master/libkexiv2/kexiv2image.cpp
Comment 44 Michael Natterer 2015-08-20 13:09:58 UTC
Step one, assign the built-in adobe profile if InteroperabilityIndex is R03.
This would be enough to close the bug as FIXED, given the bug's initial
scope, but I think we should keep it open until we have sorted:

Exif.Photo.ColorSpace
Xmp.exif.ColorSpace
Exif.Nikon3.ColorSpace
Exif.Canon.ColorSpace

I conclude: they are completely on crack...

commit 6bb117286be111a8fe6292b0933d1cb52178ed07
Author: Michael Natterer <mitch@gimp.org>
Date:   Thu Aug 20 15:05:04 2015 +0200

    Bug 492048 - Detect color space in Exif 2.21/DCF 2.0 option files
    
    Add flag GIMP_METADATA_LOAD_COLORSPACE which defaults to TRUE, and in
    gimp_image_metadata_load_finish(), assign AdobeRGB to the image if
    Exif.Iop.InteroperabilityIndex says "R03". This is most likely very
    incomplete because there are quite some other colorspace tags in
    various parts of the image metadata.

 libgimp/gimpmetadata.c     | 26 ++++++++++++++++++++++++++
 libgimpbase/gimpmetadata.h |  1 +
 2 files changed, 27 insertions(+)
Comment 45 Elle Stone 2015-08-20 13:17:02 UTC
(In reply to Michael Natterer from comment #43)
> This git repo seems to have more recent and improved code in that function:
> 
> https://github.com/veaceslav/Exiv2-hacking/blob/master/libkexiv2/kexiv2image.
> cpp

FWIW, I think this is the official "browse kde git" file:
https://quickgit.kde.org/?p=libkexiv2.git&a=blob&h=6006c33e9a15f8c36d114e561362c477ac79df47&hb=937af95156e09e0c6080a52c8c51b8a56b8f4284&f=libkexiv2%2Fkexiv2image.cpp
Comment 46 Elle Stone 2015-08-20 15:55:58 UTC
_mg_0989.jpg from comment 4 above has DCF information for the Adobe RGB 1998 color space profile: exiftool -a -S -G0 -ColorSpace -InteropIndex -WhitePoint -PrimaryChromaticities -Gamma -ICC_Profile:all _mg_0989.jpg

[MakerNotes] ColorSpace: Adobe RGB
[EXIF] ColorSpace: Uncalibrated
[EXIF] InteropIndex: R03 - DCF option file (Adobe RGB)
[EXIF] WhitePoint: 0.313 0.329
[EXIF] PrimaryChromaticities: 0.64 0.33 0.21 0.71 0.15 0.06
[EXIF] Gamma: 2.2

And img_0990.jpg has DCF information for the sRGB color space profile:

[MakerNotes] ColorSpace: sRGB
[EXIF] ColorSpace: sRGB
[EXIF] InteropIndex: R98 - DCF basic file (sRGB)

In both cases the color information was retrieved from DCF. But the two cases are not treated the same:

Upon opening the sRGB jpg, the sRGB built-in profile is assigned, and the user doesn't see any popup.

Upon opening the Adobe jpg, here is the message printed in the pop-up:

"The image '_mg_0989.jpg has an embedded color profile:
Compatible with Adobe RGB (1998)

Convert the image to the RGB working space (GIMP built-in sRGB)?". The popup provides options to Keep or Convert, plus the very dangerous option "Don't ask me again".

The two cases are exactly analogous and should be treated the same so the user knows what was done to the image.

In the Adobe case, I'm told the image had an embedded ICC profile, but it really didn't have an embedded profile. Rather it had DCF information. 

As a user, I'm inclined to trust that an embedded ICC profile *probably* is the right ICC profile for the image. But I'm less inclined to trust DCF information.

For many images, for example flower images, there aren't any reliable visual clues that the wrong profile has been assigned. So the degree of trust that the user can place in information presented in the GIMP popup really does matter. The exact source of and reason why the color profile assigned to the image does matter.

Also, having been told that there is an embedded ICC profile in the jpeg, as a user I'm going to be upset when the next image editor opens the same jpeg and tells me that there really isn't any embedded ICC profile in the image.

The "Don't ask me again" part of the Adobe popup is dangerous because it sets the user up for future automatic profile conversions to a very small color space, with potentially irreversible loss of color data, depending on the bit-depth of the image. I don't think this option should be presented in any popup - it's already dangerous enough having automatic profile conversions be options in the Color Management preferences. For reasons why:
   * See the discussion of options E3 and E4 under http://ninedegreesbelow.com/photography/digikam-settings-behavior-tab.html#untagged.
   *  For a graphical demonstration of just how much color data can be lost, see Figure 3 in http://ninedegreesbelow.com/photography/icc-profile-conversion-clipped-colors-examples.html#flower-color-gamut
Comment 47 Michael Natterer 2015-08-20 16:58:41 UTC
I agree that dialog is an inadequate mess, and that the source of the assumed
profile (embedded, dcf, or whatever) should be stated in the dialog. And yes,
the "Don't ask me again" toggle for a potentially destructive operation
is completely evil.

But about also showing the dialog when the image was sRGB (from whatever
source), I disagree, at least for GIMP 2.10. Images are either the default
sRGB, in which case the user is never asked any questions, or are tagged
with a profile on import, in which case we show the dialog.

All the digital camera images I've looked at (except the ones linked in
bug reports which use AdobeRGB), specify sRGB via one of the various
confusing ways of doing that, see comment #44. We don't really want to
show a dialog for *every* imported image, most of the time saying:

"This image seems to be sRGB, do you want to convert it to sRGB".

I would suggest that only when an image suggests that it is something
*else* than sRGB we go through that dialog mess on import, otherwise just
silently accept the sRGB image.

This way we also make sure that the majority of these images are imported
as-is, without any conversion, so without any potential data loss.
Comment 48 Elle Stone 2015-08-20 18:44:59 UTC
(In reply to Michael Natterer from comment #47)
> I agree that dialog is an inadequate mess, and that the source of the assumed
> profile (embedded, dcf, or whatever) should be stated in the dialog. And yes,
> the "Don't ask me again" toggle for a potentially destructive operation
> is completely evil.

Maybe the dialog could say something like:

"This image doesn't have an embedded ICC profile. DCF tags indicate Adobe RGB 1998 and GIMP's built-in Adobe RGB 1998-compatible profile has been assigned", with options to 
(1)Accept the assignment and stay in the Adobe color space.
(2)Accept the assignment and convert the image to the GIMP built-in sRGB color space.
(3)Assign another profile from disk and stay in that color space.
(4)Assign another profile from disk and convert to the GIMP built-in sRGB color space.

> But about also showing the dialog when the image was sRGB (from whatever
> source), I disagree, at least for GIMP 2.10. Images are either the default
> sRGB, in which case the user is never asked any questions, or are tagged
> with a profile on import, in which case we show the dialog.

For images that don't have an embedded profile and also don't have any DCF information, silently assigning sRGB is fine: being pestered with requests for what to do is annoying; and the user is responsible for knowing that sRGB is assigned  to "completely untagged" images, and also responsible for assigning a different profile if that's what they want to do. 

It might help users get a better handle on what's happening with their images if the default information on the title or status bar were changed to show what profile has been assigned to the image.

Many image editors do provide color management choices that allow the user to specify what to do with completely untagged images. Personally, I always choose "assign sRGB" and "don't do any automatic conversions", and then I assign the correct profile upon import. So I'm happy with what GIMP already provides. But some workflows might benefit from having explicit choices about what to assign. Such color management options maybe could wait until 3.0.

> 
> All the digital camera images I've looked at (except the ones linked in
> bug reports which use AdobeRGB), specify sRGB via one of the various
> confusing ways of doing that, see comment #44. We don't really want to
> show a dialog for *every* imported image, most of the time saying:
> 
> "This image seems to be sRGB, do you want to convert it to sRGB".

Well, really the dialog would say something like "This image doesn't have an embedded ICC profile. DCF tags indicate sRGB and GIMP's built-in sRGB profile has been assigned", followed by options to assign a different profile from disk, with or without a subsequent conversion to the GIMP built-in sRGB profile. 

> 
> I would suggest that only when an image suggests that it is something
> *else* than sRGB we go through that dialog mess on import, otherwise just
> silently accept the sRGB image.
> 
> This way we also make sure that the majority of these images are imported
> as-is, without any conversion, so without any potential data loss.

You make a valid point (though if the "lower precision options for memory usage" were eliminated, data loss would be minimal). For 2.10, I wouldn't argue against lumping handling of DCF sRGB together with other images that don't have embedded profiles:
     * Most of the sRGB variants are so close to one another and to GIMP's sRGB that doing an "assign and then convert" is not warrented for most circumstances.
     * Users already have the option to assign an sRGB profile from disk, just as for images that don't have any DCF information.

Adding DCF-related color management preferences would allow users who save in-camera-interpolated files - whether as Adobe RGB or sRGB or both - to indicate their preferences for DCF-tagged images, so they wouldn't have to be pestered all the time. But maybe the massive changes being made to GIMP color management make a good reason to wait until 3.0?
Comment 49 Michael Natterer 2015-08-20 20:35:58 UTC
Didn't we want to do this ages ago?

commit b8efc79d036e81d54eb050a42c39fc4d20559b7d
Author: Michael Natterer <mitch@gimp.org>
Date:   Thu Aug 20 22:34:43 2015 +0200

    app: add the image's color profile's label to th default window title

 app/config/gimpdisplayconfig.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
Comment 50 Michael Natterer 2015-08-20 20:40:33 UTC
I think I'll hack up the improved (and easily improvable) new dialog
that pops up on import, then we can see what we do with that much more
flexible code (bye bye lcms plug-in...).

Adding all sorts of preferences is made easily possible by the new color
management code, but I think we all prefer to keep the number of prefs
options as small as possible.

But first the new dialog, then it's config options :)
Comment 51 Michael Natterer 2015-09-28 17:57:07 UTC
More stuff parsed, I think we can close this bug as FIXED now, any
comments?

commit 7e9957851e6197a77bc8149488135897ce8530c4
Author: Michael Natterer <mitch@gimp.org>
Date:   Mon Sep 28 19:54:18 2015 +0200

    Bug 492048 - Detect color space in Exif 2.21/DCF 2.0 option files
    
    Copy a ton of logic from darktable and libkexiv2 and parse more
    metadata tags which contain colorspace information, namely:
    
    Exif.Photo.ColorSpace
    Xmp.exif.ColorSpace
    Exif.Nikon3.ColorSpace
    Exif.Canon.ColorSpace

 libgimp/gimpimagemetadata.c | 91 +++++++++++++++++++++++++++++++++++++++++++++++++---------
 1 file changed, 78 insertions(+), 13 deletions(-)
Comment 52 Elle Stone 2015-09-28 19:51:54 UTC
Created attachment 312325 [details]
Adobe DCF tif

GIMP recognized and assigned the Adobe-compatible profile to an R03 jpeg file "DCF-Adobe.jpg"

I used ImageMagick to convert "DCF-Adobe.jpg" to a png:
$ convert DCF-Adobe.jpg DCF-Adobe.png

GIMP recognized the R03 tag in the png file, and assigned the Adobe-compatible profile to the png.

Then I used ImageMagick to convert "DCF-Adobe.jpg" to the attached tif:
$ convert DCF-Adobe.jpg DCF-Adobe.tif

GIMP opens this tif as an sRGB image, but the embedded interoperability index is R03:
$ exiftool -a -S -InteroperabilityIndex DCF-Adobe.tif
InteroperabilityIndex: R03

I only know of one camera that could save in-camera-generated tiffs, and that was the original Dimage 7, but there are probably a few others. 

As a probably irrelevant aside (unless someone tries to replicate using a file from a different camera) ImageMagick doesn't always succeed in transferring the interperability index information from jpeg to tiff. It seems to depend on the source file (maybe different maker notes or something). It does seem to write chromaticity information to the tiff, but I didn't do a lot of testing.
Comment 53 Elle Stone 2015-09-28 19:57:11 UTC
(In reply to Michael Natterer from comment #42)

> 2. once the image is loaded, present the user with the to-be-improved
>    "Apply color profile" dialog, this is what bug 555562 is about, or
>    what I made it to be about :)

Do you want a summary of comments about changing the UI to allow more complicated user choices to be put in bug 555562? Or maybe a link to bug 492048 should be put in bug 555562?
Comment 54 Michael Natterer 2015-09-29 11:56:42 UTC
Thanks for taking care of keeping track of these. I think they should go
into bug 555562, this one is about to be closed.

Another thing: I realized that metadata can also contain an RGB
matrix, we should parse that too and create a profile. There is code
in Darktable I can use. Let's keep this bug open until this is done
too (assuming they are part of the DCF spec).
Comment 55 Elle Stone 2015-09-29 13:31:16 UTC
(In reply to Michael Natterer from comment #54)
> Thanks for taking care of keeping track of these. I think they should go
> into bug 555562, this one is about to be closed.

OK, I'll do that today, and link back to this bug report.

> 
> Another thing: I realized that metadata can also contain an RGB
> matrix, we should parse that too and create a profile. There is code
> in Darktable I can use. Let's keep this bug open until this is done
> too (assuming they are part of the DCF spec).

The matrix, white point, and gamma are part of the DCF specs, but I don't think they are required (the verbiage is imprecise):

http://www.cipa.jp/std/documents/e/DC-009-2010_E.pdf
"4.5.4.4 Color space
  The DCF optional color space is used in a DCF optional file, and its characteristics are defined using the WhitePoint tag, PrimaryChromaticities tag, YCbCrCoefficients tag, and Gamma tag."

Reading further down the page, I think it means that *if* you want to use the color space part of DCF, then the listed tags are mandatory.

It might be a good idea to not support these tags as part of dealing with camera DCF information, even if the tags are available:

1. If my interpretation is correct, the tags might not be there anyway.
2. If the tags are there, then you are in the position of creating whatever color space might result, which has the following issues:
   * LCMS doesn't compensate for hexadecimal rounding, so the resulting ICC profile might not be exactly well-behaved.
   * Not every camera vendor will put the same information in these tags, even when the color space is nominally the same.
   * If the information comes from something like "convert file.jpg file.tif", the conversion process can slightly change the transferred values.
   * You run the risk of setting up the user to do an entirely unmerited color space conversion from "almost the GIMP sRGB profile" to the GIMP sRGB profile.

So it might be better to ignore the chromaticities/whitepoint/etc information in the exif data as "not relevant" to dealing with the usual camera-saved jpeg. And maybe eventually a user will open a bug report because they want to use DCF chromaticities information, at which point the issue can be revisited. 

I'm not pretending to have any kind of definitive answer here. This is just thinking through the "what ifs".
Comment 56 Elle Stone 2015-09-29 16:04:57 UTC
(In reply to Michael Natterer from comment #54)
> Thanks for taking care of keeping track of these. I think they should go
> into bug 555562, this one is about to be closed.

I posted a comment summarizing UI considerations for dealing with DCF images:
https://bugzilla.gnome.org/show_bug.cgi?id=555562#c17

As the interoperability index tag is definitely required and almost certainly deals with all normal use cases, and the chromaticities information is probably not a required tag and presents a range of issues to be dealt with, maybe this bug can be closed?
Comment 57 Michael Natterer 2015-09-30 11:01:06 UTC
I'm not so sure if we should ignore chromaticities/whitepoint/etc info
because darktable does process them, and I think they have a lot of experience
with users of digital camera pictures. Also, we are going to closely
cooperate with darktable for raw development in 2.10, and we should be on
the same page, metadata parsing wise.

I'll look into it, we should definitely parse them, if we end up using them
let's see when we have the parsing code.
Comment 58 Elle Stone 2015-09-30 14:59:40 UTC
Hmm, my apologies if I'm about to discuss stuff you already know. DCF chromaticities information is one thing. OpenEXR chromaticities information is quite another thing.

All cameras that save DCF information are REQUIRED to save the Interoperability Index information. This is, imho, the only information that GIMP should look at to deal with camera DCF information.

Here is an exiftool command that shows DCF and OpenEXR color-space-related metadata, plus ICC profile information, that's embedded in an image file:

exiftool -a -S -G0 -ColorSpace -InteropIndex -WhitePoint -PrimaryChromaticities -Gamma -ICC_Profile:all -Chromaticities -WhiteLuminance filename.extension


ColorSpace, InteropIndex, WhitePoint, PrimaryChromaticities, and Gamma are tags related to DCF information. These are the tags you find in camera-saved jpegs (and presumably camera-saved tiffs).

ICC_Profile:all shows the ICC profile tags, *if* an ICC profile is embedded in an image, which is not the case for most or all camera-produced jpegs/tiffs.

Chromaticities and WhiteLuminance are OpenEXR tags that don't have anything to do with DCF information.

My apologies, I don't know the equivalent exiv2 tag names.

I took a couple of photographs with a Sony A7 camera, shooting raw+jpeg, first with the camera set to save sRGB jpegs, then with the camera set to save AdobeRGB1998 jpegs. Here's the relevant metadata (from the above exiftool command) in each file:

The sRGB jpeg+raw:

======== DSC00185.JPG
[EXIF] ColorSpace: sRGB
[EXIF] InteropIndex: R98 - DCF basic file (sRGB)
======== DSC00185.ARW
[EXIF] ColorSpace: sRGB
[EXIF] InteropIndex: R98 - DCF basic file (sRGB)

The AdobeRGB1998 jpeg+raw:

======== _DSC0186.JPG
[EXIF] ColorSpace: Uncalibrated
[EXIF] InteropIndex: R03 - DCF option file (Adobe RGB)
[EXIF] WhitePoint: 0.313 0.329
[EXIF] PrimaryChromaticities: 0.64 0.33 0.21 0.71 0.15 0.06
[EXIF] Gamma: 2.2
======== _DSC0186.ARW
[EXIF] ColorSpace: Uncalibrated
[EXIF] InteropIndex: R03 - DCF option file (Adobe RGB)
[EXIF] WhitePoint: 0.313 0.329
[EXIF] PrimaryChromaticities: 0.64 0.33 0.21 0.71 0.15 0.06
[EXIF] Gamma: 2.2

I opened one of the raw files with darktable and saved it to disk as an OpenEXR file (the DCF info is completely irrelevant when interpolating a raw file; what matters is the user-selected camera input profile and the user-selected output profile; the user might indeed choose sRGB or AdobeRGB1998 as the output profile for the interpolated raw file, but that choice has nothing to do with the DCF information embedded in the raw file). 

Here's what exiftool shows for the OpenEXR file saved by darktable:

[OpenEXR] Chromaticities: 0.71867972612381 0.285560429096222 0.169804409146309 1.06514310836792 0.014133732765913 -0.317087233066559 0.343314111232758 0.361693859100342
[OpenEXR] WhiteLuminance: 1

The OpenEXR Chromaticities and WhiteLuminance tags don't have anything to do with DCF PrimaryChromaticities/WhitePoint/Gamma tags.

All the camera files that I've ever looked at have the InteropIndex tag and the ColorSpace tag, whether the jpeg color space is sRGB or Adobe RGB. But the WhitePoint, PrimaryChromaticities, and Gamma tags are only saved for Adobe RGB images. 

So either the camera makers aren't following the EXIF spec, or I misinterpreted the verbiage for the ColorSpace tag, or there is some other DCF tag called "color space" tag. Either way, for camera-saved jpegs, as far as I've seen, only the Adobe RGB images have the DCF white point, chromaticities and gamma tag. 

If GIMP uses LCMS to make an AdobeRGB1998-compatible profile based on the chromaticities, white point, and gamma in the EXIF DCF data, that profile will not match the specs for the Adobe RGB 1998 color space, because of hexadecimal rounding considerations. Even the gamma won't be right.

So for DCF images, it seems to me that the best thing to do is *ignore* any embedded DCF tags for PrimaryChromaticities, WhitePoint, and Gamma, and *only* rely on the InteropIndex to distinguish between sRGB and Adobe RGB camera-saved jpegs (and tiffs). 

An image with DCF PrimaryChromaticities, WhitePoint, and Gamma tags, but without the Interoperability Index tag, has almost certainly been mishandled by some unknown image editing software before GIMP opened the image. So if the Interoperability Index information is missing from a camera-saved jpeg/tiff, and there still happens to be DCF chromaticities/white point/gamma information, I would recommend treating the image as untagged image (that is, no relevant color space information is available), and just assign the sRGB color space, just like any other image with no associated color space information.

As an aside, the ICC profile that GIMP generated based on the Chromaticities embedded in the OpenEXR file by darktable is "close but not close enough" to the actual color space that I used as the output color space for the interpolated raw file. I think maybe the chromaticities tag embedded by darktable isn't quite right.
Comment 59 Michael Natterer 2015-09-30 17:02:56 UTC
...or maybe I misunderstood what darktable is doing there....

Thanks for the elaborate explanation, and no need to apologize for
maybe-redundant information. This is a good summary. Will ask Tobias
what the code I saw in darktable is actually used for.
Comment 60 Michael Natterer 2015-09-30 18:52:28 UTC
In the meantime, some cleanup:

commit 15b7b17b12ab73b0d90a26c822da3cb20bdeb917
Author: Michael Natterer <mitch@gimp.org>
Date:   Wed Sep 30 20:47:52 2015 +0200

    Bug 492048 - Detect color space in Exif 2.21/DCF 2.0 option files
    
    Some refactoring: add gimp_metadata_get,set_colorspace() and a new
    enum GimpMetadataColorspace which so far can be one of { UNSPECIFIED,
    UNCALIBRATED, SRGB, ADOBERGB }. The setter is untested and I don't
    know if it's doing the right thing, please review. Use the getter in
    gimp_image_metadata_load_finish(), so complex metadata logic and
    profile creation/setting are separated.

 libgimp/gimpimagemetadata.c |  79 +++-----------------
 libgimpbase/gimpbase.def    |   2 +
 libgimpbase/gimpmetadata.c  | 172 +++++++++++++++++++++++++++++++++++++++++++-
 libgimpbase/gimpmetadata.h  |  87 ++++++++++++----------
 4 files changed, 233 insertions(+), 107 deletions(-)
Comment 61 Elle Stone 2015-10-01 08:53:40 UTC
Created attachment 312469 [details]
spreadsheet showing before/after metadata
Comment 62 Elle Stone 2015-10-01 09:27:28 UTC
I couldn't find sample Nikon files. But I had sample files from several other cameras. 

I updated babl/GEGL/GIMP, opened all the files, and exported them under the same file name except with "-save" added to the file name, to not overwrite the original camera-saved image

GIMP assigned the correct ICC profile to each file.

I checked the "before and after" color space information using this exiftool command:

exiftool -a -S -G1 -Model -ColorSpace -InteropIndex -WhitePoint -PrimaryChromaticities -Gamma -ICC_Profile filename.extension

For all of the sRGB jpegs, GIMP removed part of the original color space information. For example:

img_0990.jpg
[IFD0] Model: Canon EOS 30D
[Canon] ColorSpace: sRGB
[ExifIFD] ColorSpace: sRGB
[InteropIFD] InteropIndex: R98 - DCF basic file (sRGB)

img_0990-save.jpg
[IFD0] Model: Canon EOS 30D
[Canon] ColorSpace: sRGB
[ExifIFD] ColorSpace: sRGB

The InteropIndex was removed. Do other image editors remove the InteropIndex? In some ways removing the InteropIndex doesn't seem like a very good idea as the InteropIndex is part of the original metadata saved by the camera, which should be preserved to the extent possible, assuming the person exporting the image actually wants GIMP to save such metadata to exported images (personally I don't actually use GIMP to write EXIF/XMP/etc metadata).

As GIMP doesn't embed the GIMP built-in sRGB profile, the saved file has one less clue as to what the color space is.

The same thing happens with AdobeRGB DCF files, that is, the InteropIndex was removed. But at least an ICC profile was embedded:

_mg_0989.jpg
[IFD0] Model: Canon EOS 30D
[Canon] ColorSpace: Adobe RGB
[ExifIFD] ColorSpace: Uncalibrated
[InteropIFD] InteropIndex: R03 - DCF option file (Adobe RGB)
[IFD0] WhitePoint: 0.313 0.329
[IFD0] PrimaryChromaticities: 0.64 0.33 0.21 0.71 0.15 0.06
[ExifIFD] Gamma: 2.2

_mg_0989-save.jpg
[IFD0] Model: Canon EOS 30D
[Canon] ColorSpace: Adobe RGB
[ExifIFD] ColorSpace: Uncalibrated
[IFD0] WhitePoint: 0.313 0.329
[IFD0] PrimaryChromaticities: 0.64 0.33 0.21 0.71 0.15 0.06
[ExifIFD] Gamma: 22/10
[ICC_Profile] ICC_Profile: (Binary data 752 bytes, use -b option to extract)

Next I converted one of the Adobe RGB jpegs to the GIMP built-in sRGB color space and checked the metadata:

exiftool -a -S -G1 -Model -ColorSpace -InteropIndex -WhitePoint -PrimaryChromaticities -Gamma -ICC_Profile _mg_0989-convert-sRGB-save.jpg
[IFD0] Model: Canon EOS 30D
[Canon] ColorSpace: Adobe RGB
[ExifIFD] ColorSpace: Uncalibrated
[IFD0] WhitePoint: 0.313 0.329
[IFD0] PrimaryChromaticities: 0.64 0.33 0.21 0.71 0.15 0.06
[ExifIFD] Gamma: 22/10

So now the jpeg carries all the original DCF color space information, except the InteropIndex, and that information says it's an Adobe RGB image, but really it's an sRGB image, and there's no embedded ICC profile to override the DCF information, should an overeager image editing program try to make use of the remaining DCF information.

However, upon opening the converted/exported jpeg, Geeqie, showFoto, and GIMP all ignored the still-remaining DCF information and assigned an sRGB profile. So maybe they only actually rely on the InteropIndex? Is this a standard across all image editors? To ignore all DCF color space information except the InteropIndex? In which case I'm not sure what the point is of GIMP reading all the other DCF color space and associated maker-note tags really is.

Anyway, the attached spreadsheet shows all the DCF color space information for all the test files, before and after opening/exporting from GIMP. The second page shows all the metadata for one image, showing what was and wasn't written to the image, when all the "Save" metadata boxes are checked.
Comment 63 Michael Natterer 2015-10-01 10:41:43 UTC
Is that before/after saving with GIMP? Sorry that was a bit premature
because we don't update th tags on export yet, bug 575744 is still
unresolved. I just added the metadata infrastructure function, but
we don't use it yet.
Comment 64 Elle Stone 2015-10-01 11:16:41 UTC
(In reply to Michael Natterer from comment #63)
> Is that before/after saving with GIMP? Sorry that was a bit premature
> because we don't update th tags on export yet, bug 575744 is still
> unresolved. I just added the metadata infrastructure function, but
> we don't use it yet.

I don't know what you are asking. I opened each image with GIMP, and then saved it under a new name.

Bug 575744 is asking for something that really ought not to be done, it seems to me. I think GIMP should leave the original DCF information, including the InteropIndex, completely untouched. 

If the user wants GIMP to embed metadata in the image, the original DCF information should be embedded. GIMP shouldn't change the original DCF information. Instead GIMP should embed an ICC profile, always, even when the image is using the GIMP built-in sRGB profile.

There should be a completely separate "export for the web" option that strips out (or rather doesn't embed) the built-in sRGB image, and also doesn't save any of the original metadata with the image.

Users should be strongly encouraged to always embed the ICC profile for photographic images, even ones uploaded to the web. Otherwise the default Firefox color management settings produce garbage colors for sRGB images. 

Only graphic design images/elements should have the profile not embedded, and those elements also shouldn't be carrying a heavy load of other metadata. 

Anyway, checking with darktable, RawTherapee, showFoto, and Krita, all of these programs leave the original DCF information, including the InteropIndex, just as it was. They don't change these tags and they don't remove the InteropIndex tag:

exiftool -a -S -G1 -ColorSpace -InteropIndex -WhitePoint -PrimaryChromaticities -Gamma -ICC_Profile *.*

======== IMG_3861.JPG (sRGB jpeg from camera)
[Canon] ColorSpace: sRGB
[ExifIFD] ColorSpace: sRGB
[InteropIFD] InteropIndex: R98 - DCF basic file (sRGB)

======== _MG_3860.JPG (AdobeRGB jpeg from camera)
[Canon] ColorSpace: Adobe RGB
[ExifIFD] ColorSpace: Uncalibrated
[InteropIFD] InteropIndex: R03 - DCF option file (Adobe RGB)
[IFD0] WhitePoint: 0.313 0.329
[IFD0] PrimaryChromaticities: 0.64 0.33 0.21 0.71 0.15 0.06
[ExifIFD] Gamma: 2.2

I used the indicated image editor to open the above camera-saved jpeg(s), convert to a Rec.2020 ICC profile, and export to disk. None of the image editors modified the existing DCF information, and they all left the unchanged InteropIndex tag in place:

======== IMG_3861-darktable.jpg
[Canon] ColorSpace: sRGB
[ExifIFD] ColorSpace: sRGB
[InteropIFD] InteropIndex: R98 - DCF basic file (sRGB)
[ICC_Profile] ICC_Profile: (Binary data 1096 bytes, use -b option to extract)

======== IMG_3861-showfoto.jpg
[Canon] ColorSpace: sRGB
[InteropIFD] InteropIndex: R98 - DCF basic file (sRGB)
[ICC_Profile] ICC_Profile: (Binary data 1096 bytes, use -b option to extract)

======== _MG_3860-darktable.jpg
[Canon] ColorSpace: Adobe RGB
[ExifIFD] ColorSpace: sRGB
[InteropIFD] InteropIndex: R03 - DCF option file (Adobe RGB)
[IFD0] WhitePoint: 0.313 0.329
[IFD0] PrimaryChromaticities: 0.64 0.33 0.21 0.71 0.15 0.06
[ExifIFD] Gamma: 2.2
[ICC_Profile] ICC_Profile: (Binary data 1096 bytes, use -b option to extract)

======== _MG_3860-krita.jpg
[Canon] ColorSpace: Adobe RGB
[ExifIFD] ColorSpace: Uncalibrated
[IFD0] WhitePoint: 0.313
[IFD0] PrimaryChromaticities: 0.64
[ICC_Profile] ICC_Profile: (Binary data 1096 bytes, use -b option to extract)

======== _MG_3860-rawtherapee.jpg
[Canon] ColorSpace: Adobe RGB
[ExifIFD] Gamma: 2.2
[ICC_Profile] ICC_Profile: (Binary data 1096 bytes, use -b option to extract)

======== _MG_3860-showfoto.jpg
[Canon] ColorSpace: Adobe RGB
[InteropIFD] InteropIndex: R03 - DCF option file (Adobe RGB)
[IFD0] WhitePoint: 0.313 0.329
[IFD0] PrimaryChromaticities: 0.64 0.33 0.21 0.71 0.15 0.06
[ExifIFD] Gamma: 2.2
[ICC_Profile] ICC_Profile: (Binary data 1096 bytes, use -b option to extract)
Comment 65 Michael Natterer 2015-10-01 12:27:54 UTC
That seems pretty weird to me, we also update the metadata's size, resolution
etc to what the image actually is. Why would we save an image that says
e.g. AdobeRGB when the user has converted it to sRGB?
Comment 66 Elle Stone 2015-10-01 13:05:44 UTC
(In reply to Michael Natterer from comment #65)
> That seems pretty weird to me, we also update the metadata's size, resolution
> etc to what the image actually is. Why would we save an image that says
> e.g. AdobeRGB when the user has converted it to sRGB?

Well, I agree it seems a bit weird. And personally I don't care, because I don't use GIMP to embed metadata, other than the ICC profile and what's automatically embedded to take care of exporting the image in one or another file format.

However, DCF (and all Exif) information is special because it's part of the information the camera writes to the original camera-saved image. Only the user should be allowed to overwrite this information. Opening and exporting an image using GIMP isn't the same as the user consciously choosing to overwrite the original camera-written DCF/Exif information.

Even after an image is converted to some other color space (maybe sRGB, maybe a larger color space), the original camera-saved DCF color space tags still carries some pretty important information, which is the color space the "saved by the camera" original image was in. The AdobeRGB color gamut contains a significant swath of blue-greens, greens, and yellow-greens that sRGB doesn't contain, and so knowing what the original color space was tells the user something about the potential limits of the original image's color gamut (eg did that nice photograph of a green forest capture greens that exceed the sRGB color gamut? or were the greens clipped to fit the sRGB color gamut?).

One could argue that most users won't ever check the DCF color space information, so why should GIMP preserve the information for the "uncommon" user. 

One could also argue that users shouldn't allow GIMP (or any other image editor) to mess with the original "saved from camera" file, and so the original color space information should still be available in the untouched "saved from camera" file.

So if GIMP wants to tamper with the image's original DCF Exif color space information instead of automatically embedding a proper ICC profile, presumably to avoid embedding the built-in sRGB profile, that's not the end of the world. 

Nonetheless, it looks like darktable, Krita, showFoto, and RawTherapee keep the original DCF Exif color space information untouched, and instead embed a proper ICC profile in the image, which profile automatically takes precedence over the DCF information, at least in any properly written image editor. 

Embedding a proper ICC profile in all files that support ICC profiles really should be the "don't have to think about it" option for GIMP users. Exporting a file *without* an embedded ICC profile should require a conscious decision on the part of users. The common use case is graphic design elements, and for this use case the user almost certainly doesn't want to embed any metadata at all.
Comment 67 houz 2015-10-01 17:21:58 UTC
There are several points I'd like to comment on:

1) I agree that even sRGB images should have an ICC profile embedded. Not only to trick Firefox into displaying correct colors but also because there are so many different versions of sRGB profiles floating around that you can never be sure what will be used by another application.

2) It is true that darktable keeps the InteroperabilityIndex. It does however set the Exif.Photo.ColorSpace tag. Which brings me to

3) I would expect that all color space related metadata in an image reflects the current and correct state of the image. So when exporting with another profile than the input had should also result in updated Exif tags. Everything else will at some point in time result in images wrongly interpreted. Just because GIMP does value an embedded profile higher than some Exif tags doesn't mean that other applications will do the same.

4) Therefore I consider darktable not updating the InteroperabilityIndex a bug.

5) There is also a more fundamental/philosophical problem: The GIMP team always stresses that you don't open, edit, save images in GIMP but import into a new XCF, work on that and in the end export to a new image, derived from your XCF. Therefore there is no "copying over" of metadata. In the simple case of importing one JPEG and exporting back that's not really relevant, but what should happen when you paste a sRGB and an AdobeRGB image next to each other? You can only keep one color space info.
Comment 68 Elle Stone 2015-10-01 18:35:32 UTC
(In reply to houz from comment #67)
> There are several points I'd like to comment on:
> 
> 1) I agree that even sRGB images should have an ICC profile embedded. Not
> only to trick Firefox into displaying correct colors but also because there
> are so many different versions of sRGB profiles floating around that you can
> never be sure what will be used by another application.

Yes, except for exporting design elements for a website. Web designers can expect that hardly anyone uses a properly color-managed browser, and the default Firefox settings are still "half-color-managed", despite their promise that "half-color-managed" was a temporary default setting.

> 2) It is true that darktable keeps the InteroperabilityIndex. It does
> however set the Exif.Photo.ColorSpace tag.

Which can only be set to sRGB and Adobe RGB, yes? What do you do for other color spaces the user might choose when exporting an image?

> Which brings me to
> 3) I would expect that all color space related metadata in an image reflects
> the current and correct state of the image. 

> So when exporting with another
> profile than the input had should also result in updated Exif tags.
> Everything else will at some point in time result in images wrongly
> interpreted. Just because GIMP does value an embedded profile higher than
> some Exif tags doesn't mean that other applications will do the same.

One solution, that will work for all RGB working spaces and embedded ICC profiles, rather than just for sRGB and Adobe RGB, is to simply not export the relevant DCF color space tags when writing metadata to the exported image. If the image doesn't have any of the misleading tags embedded, the errant software can't value the no longer relevant camera-written DCF exif information over an embedded ICC profile.

> 
> 4) Therefore I consider darktable not updating the InteroperabilityIndex a
> bug.

For people who export their darktable-interpolated raw files in the ProPhotoRGB color space or some custom color space, what InteroperabilityIndex value would you use? Maybe it would be better to simply not export any of the DCF color space information. Though that leaves the problem of camera maker note equivalent tags. Do you plan on modifying/removing those also?

> 5) There is also a more fundamental/philosophical problem: The GIMP team
> always stresses that you don't open, edit, save images in GIMP but import
> into a new XCF, work on that and in the end export to a new image, derived
> from your XCF. Therefore there is no "copying over" of metadata. In the
> simple case of importing one JPEG and exporting back that's not really
> relevant, but what should happen when you paste a sRGB and an AdobeRGB image
> next to each other? You can only keep one color space info.

You make a good point with respect to files exported from GIMP. But I don't think the solution is to "update" the DCF color space information as the update options are rather limited. Rather the solution might be to simply not export any of that information. And of course do embed an ICC profile.
Comment 69 Michael Natterer 2015-10-01 18:42:06 UTC
I guess you guys should both review the new function

gimp_metadata_set_colorspace()

The cases GIMP_METADATA_COLORSPACE_UNSPECIFIED or
GIMP_METADATA_COLORSPACE_UNCALIBRATED will be used when an image
has a profile that is not either sRGB or AdobeRGB.

Also, please check the general logic of that function (writing tags
only if they were previously set on the imported metadata).

We also still lack support for importing the primaries matrix, Houz,
do values other than the AdobeRGB primaries exist at all in reality?
Comment 70 Elle Stone 2015-10-01 20:35:00 UTC
(In reply to Michael Natterer from comment #69)
> I guess you guys should both review the new function
> 
> gimp_metadata_set_colorspace()
> 
> The cases GIMP_METADATA_COLORSPACE_UNSPECIFIED or
> GIMP_METADATA_COLORSPACE_UNCALIBRATED will be used when an image
> has a profile that is not either sRGB or AdobeRGB.

Do these two things correspond to similar tags found in camera-saved files? Because in the collection of camera-saved files I'm working with, those tags are associated with usually AdobeRGB. For example:

[IFD0] Model: Canon EOS DIGITAL REBEL XTi
[Canon] ColorSpace: Adobe RGB
[ExifIFD] ColorSpace: Uncalibrated
[XMP-exif] ColorSpace: Unknown (Uncalibrated)
[InteropIFD] InteropIndex: R03 - DCF option file (Adobe RGB)

[IFD1] Model: ILCE-7
[ExifIFD] ColorSpace: Uncalibrated
[InteropIFD] InteropIndex: R03 - DCF option file (Adobe RGB)


[XMP-exif] Model: Canon EOS DIGITAL REBEL XTi
[Canon] ColorSpace: sRGB
[ExifIFD] ColorSpace: sRGB
[XMP-exif] ColorSpace: Unknown (sRGB)
[InteropIFD] InteropIndex: R98 - DCF basic file (sRGB)

> 
> Also, please check the general logic of that function (writing tags
> only if they were previously set on the imported metadata).
> 

I don't plan on continuing to argue over this. But for the record:

1. DCF and camera maker note equivalents are part of the information saved by the camera. You shouldn't ever be modifying that information, except by explicit user request. 

2. DCF color space information is a stop-gap measure invented to allow cameras to not save actual ICC profiles to disk. Once the image is opened by an image editor, the proper thing to do is embed an actual ICC profile. Please don't try to make DCF color space information into more than it is. 

3. I've never encountered software that used DCF information when there was an embedded ICC profile. But if some poorly written software wants to read DCF information instead of an embedded ICC profile, that's a bug, and the people who use that software should report the bug. It's not up to GIMP to try to anticipate what other poorly written software is doing. 

4. If you don't want to "confuse other software" by exporting DCF color space information that doesn't match the embedded ICC profile, instead of writing modified DCF color space information, just don't write it at all.

5. Reconciling metadata from an exported layer stack composed of several images isn't just a problem for DCF. It's a problem, period. I will guess GIMP uses the metadata from the image that was used to created the initial XCF file. I can't think of any other sensible solution.

> We also still lack support for importing the primaries matrix, Houz,
> do values other than the AdobeRGB primaries exist at all in reality?

Reading DCF chromaticities information and then making profiles from such information is just asking for problems. If there isn't an InteroperabilityIndex, it would be better to just ignore any DCF chromaticties information, for reasons given in Comment 58 above.
Comment 71 Elle Stone 2015-10-02 11:20:37 UTC
Comparing gimpmetadata.c function  
gimp_metadata_get_colorspace (GimpMetadata *metadata)

with 
http://api.kde.org/4.x-api/kdegraphics-apidocs/libs/libkexiv2/libkexiv2/html/kexiv2image_8cpp_source.html
lines 399 to 474
KExiv2::ImageColorWorkSpace KExiv2::getImageColorWorkSpace()

The flow logic looks very similar. But the place in the flow where gimp_metadata_get_colorspace reaches WORKSPACE_UNCALIBRATED seems to be much earlier than in getImageColorWorkSpace, though very possibly I'm misunderstanding the way the respective nested loops work.

gimp_metadata_get_colorspace seems to reach WORKSPACE_UNCALIBRATED before the function checks for the Nikon and Canon maker notes.

KExiv2::ImageColorWorkSpace KExiv2::getImageColorWorkSpace checks for the Nikon and Canon maker notes after checking for the InteropIndex, with an "if" rather than an "else if", and only reaches WORKSPACE_UNCALIBRATED at the bottom of the loop, just before it returns WORKSPACE_UNSPECIFIED.

I though all cameras saved the interoperability index, but if I understand the flow of the kexiv2 function, it does look like maybe some cameras don't.
Comment 72 Elle Stone 2015-10-02 12:52:34 UTC
Unless I find out from some other developers that my take on DCF color space information is incorrect, I promise this is my last effort to persuade you to not modify the DCF color space information.

DCF color space information not a statement about what color space the image is in after being exported by GIMP or darktable. 

DCF color space information is a statement about the user-chosen camera color space setting that was in effect when the camera saved the original image to disk.

If the camera-saved image is a raw file, the DCF color space information *only* tells you what color space the camera was set to when the raw file was saved to disk by the camera.

If the camera-saved image is a jpeg/tiff, the DCF color space information *also* tells you want color space the camera-saved image was in when it was saved to disk by the camera.

What color space setting was in effect when the user took the photograph is a choice made by the user. Changing the DCF color space information *falsifies* the original information written by the camera, and tells the user that really they made some choice other than the choice they really made. 

If the user opens the image with GIMP and chooses to convert the image to sRGB, *after* the image was saved to disk by the camera as an AdobeRGB image, the current plan seems to be *change* the DCF color space information to indicate the new image color space. And vice versa if the image was saved as sRGB by the camera, and the user chooses to convert to AdobeRGB. 

And if the user chooses to convert to some other color space like ProPhotoRGB or Rec.2020, the current plan is to *change*  the camera-written DCF color space information to something like "unknown" or "unspecified". 

Consider the consequences:

If the user-chosen camera settings were set to write AdobeRGB images, then changing the DCF color space information in an exported image from AdobeRGB to sRGB "because the image color space has changed" tells the user that the camera itself was set to write an sRGB image, when really it was set to write an AdobeRGB image.

If the user-chosen camera settings were set to write sRGB images, then changing the DCF color space information in an exported image from sRGB to AdobeRGB "because the image color space has changed" tells the user that the camera itself was set to write an sRGB image, when really it was set to write an AdobeRGB image.

If the user chooses to convert the camera-saved image to some other non-sRGB/non-AdobeRGB color space, changing the camera-saved DCF color space information tells the user, what? that the camera failed to save the relevant DCF information about the camera-saved image?

DCF color space information is ONLY about the color space the user chose "in-camera". It's not intended to be used to convey information about what the user chooses to do with the image after they opened the image with GIMP or darktable.

After the image is opened with GIMP or darktable, the right thing to do is leave the DCF color space information alone (don't falsify it by changing it), or else don't write it all to an exported image. And embed an ICC profile, even for sRGB images, to indicate the current image color space.
Comment 73 houz 2015-10-02 16:11:29 UTC
(In reply to Elle Stone from comment #72)

> DCF color space information not a statement about what color space the image
> is in after being exported by GIMP or darktable. 
> 
> DCF color space information is a statement about the user-chosen camera
> color space setting that was in effect when the camera saved the original
> image to disk.

Where do you get that from? I can't see anything like that in the Exif specs. And at least in version 2.2 there isn't even a note of 0x02 being a valid value for AdobeRGB in the ColorSpace tag. The InteroperabilityIndex is also not defined to carry color space information in that document. Same with DCF version 2.0. It however says that ColorSpace is mandatory and that

"This tag records Uncalibrated, indicating non-use of the sRGB color space."

So again no mentioning of AdobeRGB. And nothing about color space related metadata is meant to specify what the camera was set to initially instead of what the file currently contains.

By the way, InteroperabilityIndex is mandatory, too.
Comment 74 Michael Natterer 2015-10-03 11:26:29 UTC
I thought about all this again and still come to the same conclusion:

Why would we keep around metadata that does *not* match the file content
but instead what the file originally contained.

We also update the metadata dimensions when we crop and export, and do
not leave them at the dimensions of the original camera image.

About the logic in gimp_metadata_get_colorspace() you are right, I messed
up there, will fix (tho I don't know if kexiv2 is the "correct" reference,
but they seem to have put quite some thought into that logic).

But... I actually asked to have a look at gimp_metadata_SET_colorspace(),
because that one I hacked up in an ad-hoc way, and it's not used yet
because we didn't finish discussing here :)
Comment 75 Elle Stone 2015-10-03 13:40:31 UTC
(In reply to Michael Natterer from comment #74)
> About the logic in gimp_metadata_get_colorspace() you are right, I messed
> up there, will fix (tho I don't know if kexiv2 is the "correct" reference,
> but they seem to have put quite some thought into that logic).

When you first posted commit 15b7b17b12ab73b0d90a26c822da3cb20bdeb917, which says "please review", I looked at the code logic for "get" and for "set", and the loops in "get" seemed not quite right, but I wasn't sure why. 

A couple days ago I downloaded and examined the kexiv2 code. I posted the link to the kexiv2 code not because kexiv2 is an always correct reference, but because the logic flow for that particular set of loops seemed right in light of several more days of pondering the mishmash of dcf/exif/maker-notes tags and DCF sanctioned and unsanctioned tag values that camera makers use in place of embedding a proper ICC profile in their camera-saved images.

In response to some points Houz made about what the standards really say, yes, you are right, "ColorSpace" is not optional. I misread the reference. It said "color space" is optional, and then listed five tags, including "ColorSpace", but apparently "ColorSpace" isn't one of the tags associated with "color space". 

And yes, there are a lot of tags and tag contents written by cameras, that aren't officially sanctioned by the available DCF specs.

Camera manufacturers aren't legally bound to follow the DCF/Exif standards, and they have always played fast and loose with DCF and Exif information as it applies to the color space the image is in. That's because the camera manufacturers are choosing to not embed actual ICC profiles in camera-saved jpegs, but they don't want to be limited to just the sRGB color space.

Speaking of the mish-mash of color-space-related tags, if the goal is to cover all cameras, or at least all reasonably popular cameras, the "gimp_metadata_set/get_colorspace" code is incomplete/out of date (as is the kexiv code). 

Sony has added an additional color space, Wide Gamut RGB, following the popular camera-maker tradition of "expanding" the official specs to suit their needs; also notice the "0xfffe = ICC Profile" tag value:

http://www.sno.phy.queensu.ca/~phil/exiftool/TagNames/EXIF.html

///////Begin quote:
(the value of 0x2 is not standard EXIF. Instead, an Adobe RGB image is indicated by "Uncalibrated" with an InteropIndex of "R03". The values 0xfffd and 0xfffe are also non-standard, and are used by some Sony cameras)
0x1 = sRGB
0x2 = Adobe RGB
0xfffd = Wide Gamut RGB
0xfffe = ICC Profile
0xffff = Uncalibrated
//////End quote

Why should GIMP try to second-guess and *rewrite* all these official and unofficial exif, maker notes, and DCF tags and tag contents?

If there is an embedded ICC profile, ignore the tags, and embed an ICC profile upon export. 

If there isn't an embedded ICC profile, examine the tags and figure out what color space the image was in when it was saved to disk by the camera, assign it, and embed an ICC profile upon export. Then update the "get_metadata" code when users complain that their new camera is saving images in color spaces that GIMP isn't properly detecting.

In either case, copy the color space tags unchanged to the exported image. Or don't copy them. But don't modify them. 

Once an ICC profile has been embedded, the "color space tag soup" has *zero* value for making sure the next image editor properly interprets the image color space, unless the real goal of all this "set_metadata" code is avoiding embedding an sRGB profile in sRGB images exported by GIMP.

> But... I actually asked to have a look at gimp_metadata_SET_colorspace(),
> because that one I hacked up in an ad-hoc way, and it's not used yet
> because we didn't finish discussing here :)

I'm guessing you are asking specifically about whether the gimp_metadata_set_colorspace code covers all the relevant tags written by all possible/supported cameras?

The current code (whether "get" or "set") doesn't cover all the color space tags written by some Sony cameras. I'm wondering if it covers all the color space tags written by some Pentax cameras. What about Fuji, etc? And what about when Canon and Nikon get around to expanding the range of color spaces they offer "in-camera" to the user? It's probably a simple thing for them to add, and "larger color spaces" is a selling point.
Comment 76 Elle Stone 2015-10-03 14:58:14 UTC
(In reply to Michael Natterer from comment #74)
> I thought about all this again and still come to the same conclusion:
> 
> Why would we keep around metadata that does *not* match the file content
> but instead what the file originally contained.

"DCF" stands for "Digital Camera File". 

GIMP isn't a digital camera and doesn't save DCF camera files. Rather GIMP opens camera files and exports bog-standard image files.

The specifically "DCF" part of exif data seems to have been originally designed around the idea that digital cameras would only save sRGB images, and would send these images to various DCF-compliant software and printers. 

GIMP is not DCF software. If GIMP were following DCF guidelines to be a DCF reader, or to modify DCF files to send to a DCF printer, then Annex E "Guidelines for Implementing Tag Information", beginning page 161 of http://www.cipa.jp/std/documents/e/DC-008-2012_E.pdf (reference taken from Exiftool documentation), specifically prohibits modifying the following tags:

p. 169 E.3.4 Exif IFD Primary Image
     The following tags should be copied unchanged.
ColorSpace; ID= 40961

p. 172 E.3.5 InterOprerability IFD
     Tags in the InterOperability IFD should be copied unchanged.

p. 172 E.3.7 1st IFD
     The following tags should be copied unchanged.
WhitePoint; ID= 318
PrimaryChromaticities; ID= 319

Well, GIMP isn't and hopefully never will be DCF-compliant software. But the point remains that if GIMP exports an image with *modified* values for the camera-written DCF color space information, here is the result:

GIMP has REMOVED information that is of potential value to the user, which is to say the color space in which the image was originally saved to disk. GIMP really ought to at least give the user a choice in this matter.

GIMP has REPLACED the original camera-saved color space information with information about the color space the image was in when GIMP exported the image to disk. This is REDUNDANT if GIMP embeds a proper ICC profile, which should be the default action. And it MISLEADS the user who might think the original camera-saved color space information is still contained in the appropriate DCF/exif/maker notes color space tags.

Why stop with the color space tags? Maybe if the user changes the brightness, you should modify the corresponding brightness tag written by the camera. 

And if the user sets a white balance that isn't the white balance in the exif data, maybe darktable should rewrite the camera-saved white balance tag. 

Once you start down this road, where do you draw the line?

Especially any effort to rewrite the camera maker notes seems very, very, very wrong.
Comment 77 Elle Stone 2015-10-03 21:51:21 UTC
Assuming for the moment that rewriting the dcf/exif/maker-note color-space-related tags is a rational thing to do, and looking at the code currently in gimp_metadata_set_colorspace:

Comment 69 says "The cases GIMP_METADATA_COLORSPACE_UNSPECIFIED or
GIMP_METADATA_COLORSPACE_UNCALIBRATED will be used when an image
has a profile that is not either sRGB or AdobeRGB." I don't understand what is meant. What use-case is UNSPECIFIED supposed to cover? How is it different from UNCALIBRATED?

In the specs (http://www.cipa.jp/std/documents/e/DC-008-2012_E.pdf, 4.6.5 Exif IFD Attribute Information, B. Tags Relating to ColorSpace, page 44), UNCALIBRATED means "anything other than sRGB". But I think you are using it to mean "anything other than sRGB or AdobeRGB", which coheres with what some, maybe most, camera makers currently do. Though presumably Sony sometimes, depending on the camera, means "anything other than sRGB or AdobeRGB or WideGamutRGB or an ICC profile" (Comment 75).

case GIMP_METADATA_COLORSPACE_SRGB seems to cohere with what at least some and maybe most camera manufacturers write.

case GIMP_METADATA_COLORSPACE_ADOBERGB also seems to cohere with what at least some and maybe most camera manufacturers write. What are you planning to do with PrimaryChromaticities, WhitePoint, and Gamma tags (and maybe YcbCrCoefficients), if these tags are present and no longer correct for the user-chosen color space upon export? 

What about support for older Minolta cameras? According to Wikipedia (https://en.wikipedia.org/wiki/Design_rule_for_Camera_File_system), "Prior to the introduction of DCF 2.0, some camera vendors (e.g. Minolta and Konica Minolta) chose to use the file extension to indicate the recorded color space, that is, "JPG" was used for sRGB, whereas "JPE" was used for Adobe RGB."  I remember seeing this convention in use.

What about support for the new Sony tags (Comment 75)?

I can't find any image editing software besides GIMP and darktable that propose to modify the mish-mash of camera-saved color-space-related tags after converting the camera file to another color space and exporting the converted file. We already have ICC profiles for indicating color space information.
Comment 78 Michael Natterer 2015-10-06 19:36:15 UTC
This fixes gimp_metadata_get_colorspace(). I'm closing this bug as fixed
now because the loading part feels done. About saving the tags we have
bug 575744, I'll add a comment about the discussion here.

commit 6c20609f965e960877e5793a593b7aca544820ae
Author: Michael Natterer <mitch@gimp.org>
Date:   Tue Oct 6 21:32:12 2015 +0200

    Bug 492048 - Detect color space in Exif 2.21/DCF 2.0 option files
    
    Change the logic in gimp_metadata_get_colorspace() to be like in the
    respective KExiv2 function, which looks pretty well done. No guarantee
    of correctness, this just looks more logical than before :)

 libgimpbase/gimpmetadata.c | 100 +++++++++++++++++++++++++--------------------
 1 file changed, 55 insertions(+), 45 deletions(-)
Comment 79 BugsBunny 2015-10-08 09:57:36 UTC
Hello, I wanted to weigh in:

1- Stick to standards where possible. This is not always possible, random example: According to the CIPA and JEITA Exif specifications, the ImageDescription tag should actually contain the title of the image, not the description. According to the Metadata Working Group, the description tag stores "user comment", "caption", "abstract" or "description", and it doesn't explain where the title of the image should be stored. Confusion! As a result, Adobe consistently uses XMP and IPTC tags instead. See http://dev.exiv2.org/issues/985
I suppose the situation with the metadata which describes the color properties of the image could be in a similar state of disorder. The important thing is that there is a color profile embedded in the image, always when dealing with photos, even sRGB! When dealing with web graphics (buttons, etc) then one is not needed.

2- I agree with Houz comment 67 point 1. There is more than one sRGB. Always embed a color profile in photos.

3- It has become expected that the Exif information reflects the parameters of the camera at the time of capture. We don't want software to change the Exposure Compensation field when increasing exposure in a raw development program, neither do we want it to change the white balance info, etc.

4- It is expected that some tags should get updated, e.g. image resolution. What about the color space tags? What does the standard say and who adheres to it? What does Adobe do? It would be foolish to ignore that. I apologize if this has been covered in the comments further on, I didn't get to reading to the end, not enough time.
It would seem Adobe prefers using IPTC/XMP instead of touching the messy Exif.

5- This whole wild west situation is rightfully worked around by embedding a color profile.