How to preserve IPTC/XMP metadata when compositing?

[ohai]

Hi,

I'm trying to come up with the right command line to layer an alpha transparent TIFF image onto a solid checkerboard image, and output the final result as a 24-bit image which essentially shows the checkerboard through the alpha transparent areas. The top image in the stack contains metadata. So:

top image (32-bit TIFF, contains EXIF/IPTC/XMP metadata and transparency)
bottom image (24-bit TGA, gray and white checkerboard, designed to highlight the transparent areas of the top image)
=
result image (24-bit TIFF, containing the 32-bit image with the gray and white checkerboard showing through the transparent areas)

I've tried using:
convert bottom_image.tif top_image.tga -composite output_image.tif
but it strips the metadata from top_image.tif... Is there any way to preserve the metadata of the overlay / top_image instead of the background?

EDIT: Just as I posted this I thought perhaps I could stack 3 images instead:

top_image (contains metadata but gets wiped out)
bottom_image (contains checkerboard to show the transparency of the image above it)
top_image (contains metadata which is hopefully preserved since this image becomes "background")

That way the bottom-most image would be the one containing the metadata, even if that image is visually wiped out by the images stacked above it, but I'm not sure what the command line would look like for this, and hopefully there's a more elegant way to do it...

[snibgo]

I don't know if ImageMagick can copy all metadata. I would use exiftool:

Code: Select all

exiftool -tagsFromFile top.tiff result.tiff

[ohai]

Yeah that's already going to be my fallback approach..

I was just hoping there would be some way for IM to preserve existing metadata in the top file, rather than copy it... If there isn't a way in IM i'll use the EXIFTool approach...

[snibgo]

You might find that swapping the image files, then using "-compose DstOver -composite", does what you want.

[fmw42]

You might find that swapping the image files, then using "-compose DstOver -composite", does what you want.

As I understand it, the meta data will be copied from the first image in a convert command. So this suggestion to rearrange the images may just work.

[ohai]

Hey that seems to work.. I've only run a couple tests but later tonight will test some fully tagged images and verify that all the tags in the source image survive through to the output. Thanks for the suggestion, looks good so far and saves the extra step of copying the tags over in EXIF Tool.

[ohai]

Was too good to be true.. ImageMagick seems to mangle something in the metadata. After following this method, and then using EXIF Tool to read the files into a csv, the files that were edited with ImageMagick all have "Bad Tag" errors in the metadata, in the EXIF:EXIFToolWarning column

EDIT:
I realized I was running IM 6.5.6-10 and that seems like it was the cause..
I just grabbed the portable 6.8.0-8 version and ran a test folder on it and so far it isn't generating the same errors when I use EXIF Tool to read the resulting files to csv.

EDIT2:
Still problematic.. IM is definitely corrupting IPTC tags when I run:
imconvert "topfile.tif" "bottomfile.tga" -compose DstOver -composite "outputfile.tif"
I was able to correct almost all of the errors by running them one folder at a time. If I batch 20+ folders at once, approx half of them end up with errors. Doing them one folder at a time works, except for one folder which is yielding corrupted metadata every time I run that IM command.
(I have verified that the original files do not have corrupted metadata, so it isn't something that's simply carrying over from already damaged files)

[snibgo]

How are you using "convert" to process all the files in one directory, let alone multiple directories, in one command? Perhaps something similar to:

Code: Select all

convert *.tiff subdir\x_%d.tiff

This will read all the input files into memory before doing any processing and writing to the outputs. This can result in out-of-memory problems that IM doesn't always catch. Doing the processing in a shell loop so "convert" is executed separately for each file is safer.

[ohai]

The way I'm running it currently is compiling all the directories and files into a batch file. The resulting batch file ends up being a long line-by-line list of all the 'convert' commands.

I'm not sure how I would run your sample command on my directory structure.. Here's a sample of what it looks like:

\main\
\main\subdir1\image_sources\ (containing the pre-IM 32-bit top_images)
\main\subdir1\image_outputs\ (containing the post-IM 24-bit flattened output w/ checkerboard visible where there was transparency)
\main\subdir2\image_sources\
\main\subdir2\image_outputs\
\main\subdir3\image_sources\
\main\subdir3\image_outputs\
\central_dir_containing_bottom_image\

So each line in that batch file looks like so:

convert "\main\subdir1\image_sources\top_image1.tif" "\central_dir_containing_bottom_image\checkerboard.tga" -compose DstOver -composite "\main\subdir1\image_outputs\top_image1.tif"
convert "\main\subdir1\image_sources\top_image2.tif" "\central_dir_containing_bottom_image\checkerboard.tga" -compose DstOver -composite "\main\subdir1\image_outputs\top_image2.tif"
convert "\main\subdir1\image_sources\top_image3.tif" "\central_dir_containing_bottom_image\checkerboard.tga" -compose DstOver -composite "\main\subdir1\image_outputs\top_image3.tif"
convert "\main\subdir2\image_sources\top_image1.tif" "\central_dir_containing_bottom_image\checkerboard.tga" -compose DstOver -composite "\main\subdir2\image_outputs\top_image1.tif"
convert "\main\subdir2\image_sources\top_image2.tif" "\central_dir_containing_bottom_image\checkerboard.tga" -compose DstOver -composite "\main\subdir2\image_outputs\top_image2.tif"
convert "\main\subdir2\image_sources\top_image3.tif" "\central_dir_containing_bottom_image\checkerboard.tga" -compose DstOver -composite "\main\subdir2\image_outputs\top_image3.tif"

and so on, grabbing all the top_image(s) from all the directories, and outputting them to a parallel subdirectory in each directory as it runs through the list. In my actual scenario, what I've numbered here as subdir1 - subdir3 is actually about 50 directories containing about 6-12 images each.

[snibgo]

Okay, you are doing one convert command per file. That's good, because it means a problem with one file has no chance of "infecting" other files.

If you'd like to post the URL of a file that is giving problems, someone can take a look. But I fear the solution will be to use exiftool. ImageMagick is really about the image itself, rather than the metadata.

[ohai]

EXIF Tool actually doesn't work in this situation.. Once IM outputs that corrupted file, EXIF Tool is no longer able to write metadata to it. I've already tried.

The next thing I'll try is to resave the images from the photoshop masters and see if that makes a difference. I did notice that the files I was able to "fix" by going a directory at a time were all originally saved from Photoshop CS2, and this one set that can't be fixed was created in Photoshop CS5. I doubt that has anything to do with it, but it is one difference between the sets.

[snibgo]

Once IM outputs that corrupted file, EXIF Tool is no longer able to write metadata to it.

Eek! I suggest you post up a sample file, before and after conversion, with the exact command that did the conversion.

[ohai]

Hi, I've been away for the last week, just got back yesterday. I've put a sample image up on bitly. It's a TIF so I can't attach it here.

(issue resolved / link removed)

The source image: Test_Image.tif
The resulting image: Test_Image_postIM.tif

The exact command being used to do the conversion:

Code: Select all

convert "Test_Image.tif" "Checkerboard_lrg.tga" -compose DstOver -composite "Test_Image_postIM.tif"

If you run the following command on the source image:

Code: Select all

exiftool -a -u -g1 Test_Image.tif

you'll see it shows up nice and clean.

If you run the same command on the resulting image:

Code: Select all

exiftool -a -u -g1 Test_Image_postIM.tif

you'll see the following at the very top:

---- ExifTool ----
ExifTool Version Number : 8.91
Warning : Bad IPTC data tag (marker 0x0)
---- System ----
File Name : Test_Image_postIM.tif

This is happening a lot, with various different files, and I can't seem to find the root cause of it.

[snibgo]

My copy of exiftool (v9.02) can read your result with no problem. However, it can't update it. Windows 7 script:

Code: Select all

D:\web\im\IMconvert_problem>copy /y Test_Image_postIM.tif t2.tif
        1 file(s) copied.

D:\web\im\IMconvert_problem>exiftool -tagsFromFile Test_Image.tif t2.tif
Warning: [minor] Unrecognized data in IPTC trailer - t2.tif
Error: Error writing IFD0:IPTC-NAA - t2.tif
    0 image files updated
    1 files weren't updated due to errors

However, when I reproduce your convert, the result can be read and updated by exiftool.

Code: Select all

D:\web\im\IMconvert_problem>"c:\program files (x86)\imageMagick-6.7.9-Q16\convert" -size 1024x1024 xc:red red.tga

D:\web\im\IMconvert_problem>"c:\program files (x86)\imageMagick-6.7.9-Q16\convert"   "Test_Image.tif" red.tga -compose DstOver -composite out.tif

D:\web\im\IMconvert_problem>exiftool -tagsFromFile Test_Image.tif out.tif
    1 image files updated

I conclude your IM conversion was bad. What version IM do you have? What platform?

[ohai]

I'm running ImageMagick-6.8.0-8 on Windows. My copy of exiftool is 8.91 (going to update it after I submit this post). It can read the results with no problem also. Regardless of whether or not it can read it, there is a bad tag in the metadata.

Take my result file and run the following:
exiftool -a -u -g1 Test_Image_postIM.tif

and at the top of the readout you will see:
ExifTool Version Number : 8.91
Warning : Bad IPTC data tag (marker 0x0)

(yours will say exiftool 9.02 of course). I'm not convinced the problem is the IM conversion. I've tested the convert command on that same set of files 9 separate times now with A/V on and off, malware scanner on and off, etc, and every time I get the same bad metadata result. However, I have also run the same convert command, using the same checkerboard.tga backdrop on several thousand other files with no problems, so it's something intermittent.

I copied your process just now, and created a red.tga using IM, and then used that red.tga in place of my checkerboard.tga:
convert Test_Image.tif red.tga -compose DstOver -composite out1.tif
and the resulting out1.tif file does *not* have the bad tag notice, so now I wonder if something about my checkerboard.tga is causing the problem..?

Is there a way to have IM create and dump a checkerboard pattern into a TGA image file? I'd like to try a checkerboard created someplace other than my copy of Photoshop.

Also, would you be willing to try my checkerboard image on your end and see if it yields the same bad tags? I've placed it here: (issue resolved / link removed)
Try: convert "Test_Image.tif" "1024_Alpha_Checkers_lrg.tga" -compose DstOver -composite output.tif
and see if the output.tif you get has damaged tags.

EDIT: I should also mention, though I doubt it matters, I have my IM folder in my path, and have renamed convert.exe to imconvert.exe (since windows itself has a built-in convert.exe) but it's never caused any problems in the 3 years I've been doing it that way, so I don't think it would be creating an issue here.