Large 1bit Tiffs take forever

Questions and postings pertaining to the usage of ImageMagick regardless of the interface. This includes the command-line utilities, as well as the C and C++ APIs. Usage questions are like "How do I use ImageMagick to create drop shadows?".
DustinYoder
Posts: 14
Joined: 2019-09-24T12:41:39-07:00
Authentication code: 1152

Large 1bit Tiffs take forever

Post by DustinYoder »

I have very high res TIF files that need to be rotated. So I notice that magick is creating a temp file to do this as the file is over 1GB in size on the drive. So I just do a convert {INFILE} -rotate 270 {OUTFILE} and it processes small 1 bit tiffs fairly well, still not super fast, but the larger ones take FOREVER. Like hours. I can add -debug All to the command and see that it is still making progress, but very slowly. It keeps writing to the pixel cache.
So any ideas? I'm increasing my memory limits in the policy.xml file as well as making it a single thread. The main issue with these tiffs that may be unique is that they are 1 bit or 2 color only. Black and White, no gray. Any thing I can do to speed this up? Is there a way to rotate that doesn't take much calculation? Thanks.
snibgo
Posts: 12159
Joined: 2010-01-23T23:01:33-07:00
Authentication code: 1151
Location: England, UK

Re: Large 1bit Tiffs take forever

Post by snibgo »

I suppose you don't have enough memory. I suggest you try a specialist program such as tiffcrop.
snibgo's IM pages: im.snibgo.com
User avatar
fmw42
Posts: 25562
Joined: 2007-07-02T17:14:51-07:00
Authentication code: 1152
Location: Sunnyvale, California, USA

Re: Large 1bit Tiffs take forever

Post by fmw42 »

What version of ImageMagick and what platform? IM 6 will treat it as color (>=3 channels). But IM 7 will treat it as one channel, assuming no transparency. So IM 6 will need more RAM. How big is your image in width and height? How much RAM do you have?
DustinYoder
Posts: 14
Joined: 2019-09-24T12:41:39-07:00
Authentication code: 1152

Re: Large 1bit Tiffs take forever

Post by DustinYoder »

The image is 40" x 30" at 2540 ppi so in the range of 101,600 x 76,200 pixels. I tried the latest version 7 as well as the latest version 6. The computer is pretty new, it's got an i7 6 core 3.2ghz with 32 GB of ram. I tried zip, rle, lzw and no compression in the tiffs. All seem to just never finish processing. I can see the log writing to the image cache temp file every 5 or 10 second or so when I debug.
DustinYoder
Posts: 14
Joined: 2019-09-24T12:41:39-07:00
Authentication code: 1152

Re: Large 1bit Tiffs take forever

Post by DustinYoder »

Where could I download a windows version of tiffcrop? Having trouble finding that.
User avatar
fmw42
Posts: 25562
Joined: 2007-07-02T17:14:51-07:00
Authentication code: 1152
Location: Sunnyvale, California, USA

Re: Large 1bit Tiffs take forever

Post by fmw42 »

It comes with libtiff, the delegate that ImageMagick uses for processing tiff files. Search Google for libtiff
DustinYoder
Posts: 14
Joined: 2019-09-24T12:41:39-07:00
Authentication code: 1152

Re: Large 1bit Tiffs take forever

Post by DustinYoder »

windows binaries for libtiff are at http://gnuwin32.sourceforge.net/packages/tiff.htm, but after installing this, I still don't see a tiffcrop.exe anywhere. Anyone have a compiled version for windows that I can install with all required dll's etc?

I tried using 2TIFF to rotate and that is basically just taking forever to execute too. I'm afraid we're going to end up paying heidelberg $5000 for some custom software they sell for this purpose which can actually handle this type of thing. It does it very quickly too, so I think there is an easy way to do 90 degree increments that these libraries are not taking advantage of.
User avatar
fmw42
Posts: 25562
Joined: 2007-07-02T17:14:51-07:00
Authentication code: 1152
Location: Sunnyvale, California, USA

Re: Large 1bit Tiffs take forever

Post by fmw42 »

100,000x75,000 = 7.5 GB as a single channel 8-bit image. Even though you have binary data, ImageMagick will convert that to 32*bits internally to process, I believe, So 7.5*4=30 GB. So you are likely thrashing to disk a lot. If you use ImageMagick 7, I think ImageMagick will keep it as a single 8-bit image. So you will have 4x less space required. You probably can modify your policy.xml file for processing huge images. See https://imagemagick.org/Usage/files/#massive and policy.xml file at https://imagemagick.org/script/resources.php. Sorry, I am not an expert on this subject. So cannot advise on the resource tuning.
snibgo
Posts: 12159
Joined: 2010-01-23T23:01:33-07:00
Authentication code: 1151
Location: England, UK

Re: Large 1bit Tiffs take forever

Post by snibgo »

Tiffcrop is available as a tool in the Cygwin package for Windows. I expect tiffcrop can be downloaded without all the other Cygwin stuff.

However, sadly, tiffcrop gave me an "integer overflow" when I tried it on a 100000x10000 pixel image.


You have 32GB of memory, but how much free memory? Did you have any other programs running, eg web browsers? Did you try Q8 IM, non-HDRI?

If that needs a temporary file on disk, then try converting the input TIFF to a MPC file, and use that as input to a rotation.

If the rotation still isn't fast, perhaps try: chop the input into four (or 9 or 16) pieces, then rotate each piece, then reassemble.
snibgo's IM pages: im.snibgo.com
Werty
Posts: 66
Joined: 2010-08-06T05:37:36-07:00
Authentication code: 8675308

Re: Large 1bit Tiffs take forever

Post by Werty »

Since on windows you can try opening it with windows photo viewer and rotate it there, i just tried with a 24.000 x 24.000 NASA TIFF image of the moon and it only took a couple minutes, this on i7 4 core on a 32bit win7 with 4gb ram and a mechanical harddrive. Loading took the most of the time, the rotating process was near instant, and saving took 5-10 seconds
Windows 7 user
DustinYoder
Posts: 14
Joined: 2019-09-24T12:41:39-07:00
Authentication code: 1152

Re: Large 1bit Tiffs take forever

Post by DustinYoder »

Thanks for the replies, I'll try a few of these ideas. I think I can use powershell to using photoviewer to rotate via the command line. So I could try that as well. I need to use command line because I'm running this off of a watched hotfolder, so it can be automatic. I'm going to check the policy.xml and there is a -depth 1 setting to set output to 1 bit depth, but I don't know if that will help during rotation or not. Thanks for the ideas, I'll try to let you know what I find out.
User avatar
fmw42
Posts: 25562
Joined: 2007-07-02T17:14:51-07:00
Authentication code: 1152
Location: Sunnyvale, California, USA

Re: Large 1bit Tiffs take forever

Post by fmw42 »

-depth 1 will only help when writing the output, not while doing the rotation. If you cannot tune the policy.xml file to get adequate speed, you might consider other tools such as VIPS, but I have not experience with it.
DustinYoder
Posts: 14
Joined: 2019-09-24T12:41:39-07:00
Authentication code: 1152

Re: Large 1bit Tiffs take forever

Post by DustinYoder »

Well I was able get my limits set to the point where it didn't need to write to disk. Now I rotate the image in about 2.5 minutes. So I think that will work. I had to put the -limit parameters in the command because my policy.xml file was not being used for some reason. I tried uncommenting the xml comments in that file, but identify limits kept showing lower numbers
So now I'm good except the output tiff is TIFF64 format and so is considered invalid by my destination software. The input tiff is simply TIFF format. How can I force TIFF output and not TIFF64. I think that is the only reason the output file is not opening, it's not opening in photoshop either though so I'm worried the TIFF64 thing is not the only problem. The identify [file] command shows everything same except for the format. Thanks.
User avatar
magick
Site Admin
Posts: 11064
Joined: 2003-05-31T11:32:55-07:00

Re: Large 1bit Tiffs take forever

Post by magick »

Thanks for the problem report. We can reproduce it and will have a patch to fix it in GIT master branch @ https://github.com/ImageMagick/ImageMagick later today. The patch will be available in the beta releases of ImageMagick @ https://www.imagemagick.org/download/beta/ by sometime tomorrow.
DustinYoder
Posts: 14
Joined: 2019-09-24T12:41:39-07:00
Authentication code: 1152

Re: Large 1bit Tiffs take forever

Post by DustinYoder »

Thanks! I appreciate that. Would anyone be able to compile that beta version for 8 bit imagemagick, 64bit Windows? I could use this today and I probably would spend much of the day getting it to compile properly. Or instructions on compiling this for windows in the easiest way. Thanks again.
Post Reply