Gray balance approach reasonable? (snibgo)

Discuss digital image processing techniques and algorithms. We encourage its application to ImageMagick but you can discuss any software solutions here.
Post Reply
User avatar
Skygaze
Posts: 5
Joined: 2018-03-19T15:42:47-07:00
Authentication code: 1152
Contact:

Gray balance approach reasonable? (snibgo)

Post by Skygaze »

Hi there.
I would appreciate some advice/direction:

I would like to transform the color balance from a sensor with "no IR-filter" (Canon 60Da) to a "normal" one, say Canon 5DMark I. I have written up a web page describing the problem and a potential solution, including a couple of images: https://sites.google.com/site/alistarga ... -to-normal

Now of course I can adjust one image in GIMP, but I have several thousands of images, so I would prefer an ImageMagick solution I can script into a loop instead of going via Photoshop/Lightroom.

In brief, I think my approach is: under a range of bright to dark twilight conditions, use TWO cameras to take bracketed exposures of a gray card. Then using snibgo's Gray balance approach (http://im.snibgo.com/colchkcha.htm#graybal) create a library of transforms indexed to the solar depression and exposure value. With that in hand, for any picture past or present, I can push it through the appropriate transform. I know it won't be exact, but it will be closer to what my eye sees than the current excess of reds.

Am I on the right path? Is there something obvious I've missed?

Alister.
P.S. This forum boosts my confidence that humanity will make it! Thanks to all who contribute to the community.
snibgo
Posts: 12159
Joined: 2010-01-23T23:01:33-07:00
Authentication code: 1151
Location: England, UK

Re: Gray balance approach reasonable? (snibgo)

Post by snibgo »

Interesting problem. A general assumption in image processing is that we are dealing with real colours, those inside the CIE horseshoe. In principle, we know the transformation each camera makes from "scene referred" to "camera referred", so we know the inverse of each transformation, so we can transform the output from either camera to look as if it was made by the other. Or we can take short cuts and consider just gray balance (my page you linked) and contrast and saturation and so on.

But the general assumption isn't true for you. One of your cameras captures non-real colours, colours you can't see, beyond the red end of the spectrum. It transforms those colours into visible ones, colours you can see on the computer screen or whatever. I guess that it also "shuffles up" ordinary visible red colours, to make room for the colours that were infrared, but I could be wrong.

I have no experience of IR, so I can only speculate.

Two important factors are: the amount of IR in the illumination, and the IR reflectivity of the object. About half the radiation from the sun is IR. Does this proportion vary by time of day, or moisture in the air? I don't know. What proportion of light from blue sky or cloud is IR? I don't know, but I suspect not much. How much from a sunset? Different artificial lights will have different proportions of IR (eg inefficient incandescent versus modern fluorescent or LED). Green vegetation is a good IR reflector. In addition, all objects above absolute zero emit IR radiation, so cameras sensitive to IR can see people even when there is no light source.

You might get hold of a photographic IR filter, the type that screws to the front of a lens. Or even just a piece of IR gel. (Or improvise with a rectangular container of water!) Then you can take photos of the same scene with the same camera and lens, with and without the IR filter. If you also have an ordinary camera with a sensor IR filter, then take the same photo, and you have three images. You can create a series of tests with different illuminations (midday, before sunset, after sunset, artificial lights), and of different objects (gray card, people, foliage, landscape, skyscape). Test with auto-exposure, but also the same manual exposure for all three photos.

(We have a naming problem: the IR filter at the sensor blocks IR light ("cut-IR"), but the IR filter at the front of the lens blocks everything except IR ("pass-IR").)

These tests will show you what the Canon 60Da does to images in the presence or absence of IR. The photos with the "pass-IR" filter measure how much IR there is. I would expect an obvious tonal shift, and probably a red-shift in the areas where there is most IR present. For example, as shown on your page, the 60Da shows the red sky as lighter and redder, but doesn't change the blue sky.

For a constant subject (eg sunsets with black silhouette foreground) I expect a transformation from one camera to the other might be quite simple. Two stages:

1. Create a mask that is white where you have most IR, black where there is none, and gray in between. This is made from the image, according to redness and brightness. For example:

Code: Select all

magick input.png ( +clone -fill Red -colorize 100 ) -compose Difference -composite -grayscale RMS -auto-level mask.png
2. Create a transformation from "ordinary camera" to "IR camera". This might be a red shift and lighter tones, eg "-modulate 120,100,80".

3. Composite with the mask, eg:

Code: Select all

magick toes.png ( +clone -modulate 120,100,80 ) mask.png -compose Over -composite out.png
Sorry, I've rambled a bit.
snibgo's IM pages: im.snibgo.com
User avatar
Skygaze
Posts: 5
Joined: 2018-03-19T15:42:47-07:00
Authentication code: 1152
Contact:

Re: Gray balance approach reasonable? (snibgo)

Post by Skygaze »

Thanks for the reply! I concur with your speculations. Fortunately for my current need, I don't need to worry about the complete range of subjects. I too thought about those times when something is visible in IR (Sun setting through forest fire smoke) but not in the normal camera, the transform falls apart in the strict sense - will just have to live with that.

Hopefully I will have time this evening to give your code snippets a try. As for gray card imaging under twilight, looks like my weather won't permit that for a number of days.

Much appreciated,
Alister.
snibgo
Posts: 12159
Joined: 2010-01-23T23:01:33-07:00
Authentication code: 1151
Location: England, UK

Re: Gray balance approach reasonable? (snibgo)

Post by snibgo »

Good stuff.

To expand slightly on this point:
snibgo wrote:2. Create a transformation from "ordinary camera" to "IR camera". This might be a red shift and lighter tones, eg "-modulate 120,100,80".
The transformation can go in the opposite direction, of course, and is likely to be more complex. If you have two photos of the same subject under different conditions, eg with and without a filter, there are many ways of finding a transformation that makes one look like the other. These methods include, roughly increasing in complexity and accuracy:
Colour checker charts: Gray balance
Setting the mean
Gain and bias
Set mean and stddev
Colours to matrix and polynomials
Matching histograms
snibgo's IM pages: im.snibgo.com
User avatar
Skygaze
Posts: 5
Joined: 2018-03-19T15:42:47-07:00
Authentication code: 1152
Contact:

Re: Gray balance approach reasonable? (snibgo)

Post by Skygaze »

Making progress! While I wait to borrow a 6x4 color chart from one of my friends, I played with the "forward" transform, that is from my IR-sensitive 60Da to the non-IR "normal" shot. At first I got a little confused with the modulate command until I realized the values were not R,G,B. Then I saw the matrix method on the IM webpage and realized it was much closer to the direction of the colchkcha method.
mask:

Code: Select all

magick 60Da.jpg ( +clone -fill Red -colorize 100 ) -compose Difference -composite -grayscale RMS -auto-level mask.png
negate it:

Code: Select all

magick mask.png -negate mask_neg.png
transform with matrix:

Code: Select all

magick 60Da.jpg ( +clone -color-matrix " 0.9 0.15 0.1 0.10 0.9 0.2 0 0 0.8 " ) mask_neg.png -compose Over -composite 60Da2MP_90_15_10_10_90_20_b80.png
In essence, I am reducing red by giving it 90% R then feeding in it 15% G and 10% B, dropping the green a bit by mixing in some red and blue, and finally reducing the blue a bit. I iterated those matrix numbers around until I got something I liked, posted on the updated page:
https://sites.google.com/site/alistarga ... -to-normal

Next step is to implement the methods and do it properly. I've downloaded the zipped bats and had a look. At first I was hoping that I would not have to go via the CygWin implementation http://im.snibgo.com/customim.htm (I am currently running IM with the Windows env.), but it appears that to run the methods which are in c, then CygWin is required. Given my current schedule/calendar, it will likely be a few days before the configs/makes are working properly. Thanks so much for detailing how to get there!
Last edited by Skygaze on 2018-03-23T13:43:27-07:00, edited 1 time in total.
snibgo
Posts: 12159
Joined: 2010-01-23T23:01:33-07:00
Authentication code: 1151
Location: England, UK

Re: Gray balance approach reasonable? (snibgo)

Post by snibgo »

My process module cols2mat (see Colours to matrix and polynomials) calculates the 6x6 colour matrix that best transforms the first image to the second. For example, taking the first two images on your page:

Code: Select all

set SRC1=20180308_AL_60Da_nofilter.jpg
set SRC2=20180308_MP_IRfilter.jpg

%IMDEV%convert ^
  %SRC1% ^
  %SRC2% ^
  -process 'cols2mat method Cross' ^
  noIRtoIR.jpg
The result noIRtoIR.jpg is:
Image
The text output is:

Code: Select all

c2matrix=-0.735168,3.20423,-1.54449,0,0,0.115537,-0.609434,1.35115,0.00547097,0,
0,0.0658214,-0.42206,-0.156742,1.19194,0,0,0.0513584,0,0,0,1,0,0,0,0,0,0,1,0,0,0
,0,0,0,1
Warning: may clip Red highlights
Warning: may clip Green highlights
Adding line-breaks to the matrix for legibility:

Code: Select all

-0.735168,3.20423,-1.54449,0,0,0.115537,
-0.609434,1.35115,0.00547097,0,0,0.0658214,
-0.42206,-0.156742,1.19194,0,0,0.0513584,
0,0,0,1,0,0,
0,0,0,0,1,0,
0,0,0,0,0,1
The last three lines are constant. The final column is an addition to each channel, and we see they are calculated to be quite small. The top-left 3x3 sub-matrix can be directly compared to the numbers you found manually.

By default, cols2mat find the best matrix to transform the entire image. If you have a mask of which parts of the image are affected by IR, you can use that to make the rest of the image transparent, and use the "weightAlpha" option to get a matrix optimized for just the IR parts.


My process modules are written in C. To use them, you must be able to compile, link and install IM from source. I do that with Cygwin tools on Windows, documented on my pages.
snibgo's IM pages: im.snibgo.com
User avatar
Skygaze
Posts: 5
Joined: 2018-03-19T15:42:47-07:00
Authentication code: 1152
Contact:

Re: Gray balance approach reasonable? (snibgo)

Post by Skygaze »

Looks even better, as I would expect! And gives me the confidence that the effort I put into copying your CygWin implementation will really pay off.

I'm really looking forward to getting it going!

Thanks so much.
User avatar
Skygaze
Posts: 5
Joined: 2018-03-19T15:42:47-07:00
Authentication code: 1152
Contact:

YES !!!! Re: Gray balance approach reasonable? (snibgo)

Post by Skygaze »

WOOHOO! Holy smokes, thank you so much for all your help snibgo. It was a bit of a steeple chase fixing the compile errors and the changes in MagickCore function calls from V6.9x to V7.0x, but I am so grateful for your guidance, and happy with myself at not having given up in the face of "yet another compiler error".

Here is a result fresh oout of the computational oven, using images from the other night, where I ran 2 cameras, side by side with the identical exposure settings, 60Da, 5DMark II, transformed 60Da image:
https://sites.google.com/site/alistarga ... l/Trio.jpg

Hot damn!

The images are out of focus because I was switching the pair of cameras from shooting the twilight sky to shooting a nearby Xrite ColorChecker passport. I intend to use those to calibrate different exposures and solar depression angles, so that I can take any 60Da image past or future and transform it to something pretty close to a normal camera's colour balance. In the meantime, a single pair produces a transform matrix (using cols2mat method Cross above) is a fantastic start.

I intend to post updates to my web page https://sites.google.com/site/alistarga ... -to-normal when I finish the calibration process. The colour matrix here is: c2matrix=0.53761,0.237494,0.0655367,0,0,-0.0618967,0.0230337,0.522579,0.442921,0,0,-0.0514341,-0.197834,0.169792,1.01265,0,0,-0.0241132,0,0,0,1,0,0,0,0,0,0,1,0,0,0,0,0,0,1

I'm really pumped that this all worked, with much appreciation to snibgo!
Alister.
snibgo
Posts: 12159
Joined: 2010-01-23T23:01:33-07:00
Authentication code: 1151
Location: England, UK

Re: Gray balance approach reasonable? (snibgo)

Post by snibgo »

That's a good result, Alister. Thanks for sticking with it. With your encouragement, I've updated my modules to build under v7.0.7-28. They all build and run, and most of them run correctly. When I've fixed those that don't, I'll update my web pages.
snibgo's IM pages: im.snibgo.com
Post Reply