Setting map limit

Questions and postings pertaining to the usage of ImageMagick regardless of the interface. This includes the command-line utilities, as well as the C and C++ APIs. Usage questions are like "How do I use ImageMagick to create drop shadows?".
Post Reply
craig.francis
Posts: 6
Joined: 2020-01-31T14:22:28-07:00
Authentication code: 1152

Setting map limit

Post by craig.francis »

I'm currently trying to get the sizes out of a large TIFF file.

But if I set the "map" resource in "policy.xml" to something high (1GiB), I get a "Cannot allocate memory" error.

Code: Select all

grep 'resource' /etc/ImageMagick-6/policy.xml
  <policy domain="resource" name="memory" value="3GiB"/>
  <policy domain="resource" name="map" value="1GiB"/>
  <policy domain="resource" name="width" value="32KP"/>
  <policy domain="resource" name="height" value="32KP"/>
  <policy domain="resource" name="area" value="-1"/>
  <policy domain="resource" name="disk" value="-1"/>

Code: Select all

time /usr/bin/identify -debug cache,resource -format "%p / %w %x %U / %h %y \n" ./test.tif 
2020-01-31T21:28:37+00:00 0:00.010 0.000u 6.9.7 Resource identify[21406]: resource.c/AcquireMagickResource/318/Resource
  Map: 863.5MB/823.5MiB/1GiB
2020-01-31T21:28:37+00:00 0:00.010 0.000u 6.9.7 Resource identify[21406]: resource.c/RelinquishMagickResource/1013/Resource
  Map: 863.5MB/0B/1GiB
2020-01-31T21:28:37+00:00 0:00.010 0.000u 6.9.7 Resource identify[21406]: resource.c/AcquireMagickResource/318/Resource
  Map: 863.5MB/823.5MiB/1GiB
identify: Memory allocation failed `Cannot allocate memory' @ fatal/string.c/AcquireStringInfo/182.

real	0m4.182s
user	0m0.980s
sys	0m3.204s
But if I set it a bit lower (768MiB)

Code: Select all

grep 'resource' /etc/ImageMagick-6/policy.xml
  <policy domain="resource" name="memory" value="3GiB"/>
  <policy domain="resource" name="map" value="768MiB"/>
  <policy domain="resource" name="width" value="32KP"/>
  <policy domain="resource" name="height" value="32KP"/>
  <policy domain="resource" name="area" value="-1"/>
  <policy domain="resource" name="disk" value="-1"/>
Then it's fine:

Code: Select all

time /usr/bin/identify -debug cache,resource -format "%p / %w %x %U / %h %y \n" ./test.tif 
2020-01-31T21:30:08+00:00 0:00.000 0.000u 6.9.7 Resource identify[21522]: resource.c/AcquireMagickResource/318/Resource
  Map: 863.5MB/823.5MiB/768MiB
2020-01-31T21:30:08+00:00 0:00.000 0.000u 6.9.7 Resource identify[21522]: resource.c/AcquireMagickResource/318/Resource
  Map: 863.5MB/1.6083GiB/768MiB
2020-01-31T21:30:17+00:00 0:08.780 8.790u 6.9.7 Resource identify[21522]: resource.c/AcquireMagickResource/318/Resource
  Width: 16KB/16KB/32KB
2020-01-31T21:30:17+00:00 0:08.790 8.790u 6.9.7 Resource identify[21522]: resource.c/AcquireMagickResource/318/Resource
  Height: 4.8KB/4.8KB/32KB
2020-01-31T21:30:17+00:00 0:08.860 8.870u 6.9.7 Resource identify[21522]: resource.c/AcquireMagickResource/318/Resource
  Width: 17.2KB/17.2KB/32KB
2020-01-31T21:30:17+00:00 0:08.860 8.870u 6.9.7 Resource identify[21522]: resource.c/AcquireMagickResource/318/Resource
  Height: 6.79KB/6.79KB/32KB
2020-01-31T21:30:17+00:00 0:08.860 8.870u 6.9.7 Resource identify[21522]: resource.c/AcquireMagickResource/318/Resource
  Width: 17.2KB/17.2KB/32KB
2020-01-31T21:30:17+00:00 0:08.860 8.870u 6.9.7 Resource identify[21522]: resource.c/AcquireMagickResource/318/Resource
  Height: 6.79KB/6.79KB/32KB
2020-01-31T21:30:17+00:00 0:08.860 8.870u 6.9.7 Resource identify[21522]: resource.c/AcquireMagickResource/318/Resource
  Width: 16KB/16KB/32KB
2020-01-31T21:30:17+00:00 0:08.860 8.870u 6.9.7 Resource identify[21522]: resource.c/AcquireMagickResource/318/Resource
  Height: 689B/689B/32KB
2020-01-31T21:30:17+00:00 0:08.860 8.870u 6.9.7 Resource identify[21522]: resource.c/AcquireMagickResource/318/Resource
  Width: 16KB/16KB/32KB
2020-01-31T21:30:17+00:00 0:08.860 8.870u 6.9.7 Resource identify[21522]: resource.c/AcquireMagickResource/318/Resource
  Height: 4.8KB/4.8KB/32KB
2020-01-31T21:30:17+00:00 0:08.860 8.870u 6.9.7 Cache identify[21522]: cache.c/DestroyPixelCache/1123/Cache
  destroy ./test.tif[0]
2020-01-31T21:30:17+00:00 0:08.860 8.870u 6.9.7 Cache identify[21522]: cache.c/DestroyPixelCache/1123/Cache
  destroy 
2020-01-31T21:30:17+00:00 0:08.960 8.940u 6.9.7 Cache identify[21522]: cache.c/DestroyPixelCache/1123/Cache
  destroy ./test.tif[0]
2020-01-31T21:30:17+00:00 0:08.960 8.950u 6.9.7 Cache identify[21522]: cache.c/DestroyPixelCache/1123/Cache
  destroy ./test.tif[0]
2020-01-31T21:30:17+00:00 0:08.960 8.950u 6.9.7 Cache identify[21522]: cache.c/DestroyPixelCache/1123/Cache
  destroy ./test.tif[0]
2020-01-31T21:30:17+00:00 0:08.960 8.950u 6.9.7 Cache identify[21522]: cache.c/DestroyPixelCache/1123/Cache
  destroy ./test.tif[0]
identify: Incompatible type for "RichTIFFIPTC"; tag ignored. `TIFFFetchNormalTag' @ warning/tiff.c/TIFFWarnings/912.
0 / 16000 400 PixelsPerInch / 4800 400 
1 / 17211 400 PixelsPerInch / 6793 400 
2 / 17211 400 PixelsPerInch / 6793 400 
3 / 15969 400 PixelsPerInch / 689 400 
4 / 16000 400 PixelsPerInch / 4800 400 

real	0m8.963s
user	0m1.799s
sys	0m7.162s
I am using a slightly over version of ImageMagick (6.9.7-4 Q16 i686 20170114), which comes with the current version of Ubuntu 18.04.3 LTS, so that might be related.

But it's annoying, as I need to use higher limits for when I re-size the image... and just seems weird that it would break on a machine with 4GB RAM (and 4GB SWAP).

And maybe related, but even with a high "memory" limit, ImageMagick doesn't really use more than 1.3GB... but that's probably a different issue.

snibgo
Posts: 12904
Joined: 2010-01-23T23:01:33-07:00
Authentication code: 1151
Location: England, UK

Re: Setting map limit

Post by snibgo »

The problem is that you don't have enough memory for the pixel caches and other stuff. When you limit the memory used for pixel caches, you don't then hit problems when allocating memory for other structures.

You might try "-ping" as the first option. This may prevent IM from reading pixels.
snibgo's IM pages: im.snibgo.com

craig.francis
Posts: 6
Joined: 2020-01-31T14:22:28-07:00
Authentication code: 1152

Re: Setting map limit

Post by craig.francis »

Thanks snibgo,

I've been trying to read the documentation, and am I right in saying that ImageMagick will use MAGICK_MEMORY_LIMIT first, for the pixel cache, and if that's not enough, it moves on to MAGICK_MAP_LIMIT?

https://imagemagick.org/script/resources.php

If so, I should be able to set MAGICK_MEMORY_LIMIT and MAGICK_MAP_LIMIT to "1GiB", to give a total of "2GiB"... and for this to work on a system with 4GB of RAM, and 4GB of SWAP (unfortunately it doesn't work).

As an aside, and while it's not good practice, I can set `MAGICK_TEMPORARY_PATH=/run/shm`, and process the image using the temporary filesystem (tmpfs) that resides in RAM.

snibgo
Posts: 12904
Joined: 2010-01-23T23:01:33-07:00
Authentication code: 1151
Location: England, UK

Re: Setting map limit

Post by snibgo »

craig.francis wrote:... am I right in saying that ImageMagick will use MAGICK_MEMORY_LIMIT first, for the pixel cache, and if that's not enough, it moves on to MAGICK_MAP_LIMIT?
That's not correct. Both limits are always active. If either limit is reached, disk is used. See https://www.imagemagick.org/script/resources.php .

You have 4 GB of RAM. But is that 4 GB the total, or the amount available for IM?
snibgo's IM pages: im.snibgo.com

User avatar
magick
Site Admin
Posts: 11217
Joined: 2003-05-31T11:32:55-07:00

Re: Setting map limit

Post by magick »

ImageMagick attempts to allocate the pixel cache in memory since it has the lowest latency. The allocation might fail if the allocation exceeds any set memory limits or if the system request fails, e.g. the request exceeds the available free memory on the system. If the memory allocation fails, ImageMagick allocates the pixel cache on disk and then attempts to memory-map it which can be 5x faster when reading than direct disk access. The allocation might fail if the allocation exceeds any memory map limits or if the system request fails, e.g. the request exceeds the available memory map pages on the system. If memory-mapping fails, ImageMagick attempts to allocate the pixel cache on disk. This allocation might fail if the allocation exceeds any set disk limits or if the system call fails, e.g. you don't have permission to write on disk. The pixel cache on disk could fail while writing if the available free disk is exceeded. The pixel cache on disk has lazy allocation, meaning you might be well into writing before the system reports there is not enough space. To ensure there is enough disk space before processing begins on an image, set the MAGICK_SYNCHRONIZE environment variable which is ensures the entire file can fit in the disk partition before any processing begins.

craig.francis
Posts: 6
Joined: 2020-01-31T14:22:28-07:00
Authentication code: 1152

Re: Setting map limit

Post by craig.francis »

snibgo wrote:
2020-02-01T09:07:09-07:00
You have 4 GB of RAM. But is that 4 GB the total, or the amount available for IM?
In total.

The server typically uses about 90MB of RAM, and when running with 1GiB for both MAGICK_MEMORY_LIMIT and MAGICK_MAP_LIMIT, I would assume it's hitting those limits, then move processing to disk... but it crashes instead.

If I set MAGICK_MAP_LIMIT to 768MiB, then it does successfully start using the disk.

But I would like to increase those limits, to use as much of the RAM as possible (i.e. slightly less than) 4GB of RAM... that said, even using some of the 4GB of SWAP would probably help.

craig.francis
Posts: 6
Joined: 2020-01-31T14:22:28-07:00
Authentication code: 1152

Re: Setting map limit

Post by craig.francis »

magick wrote:
2020-02-01T15:08:59-07:00
To ensure there is enough disk space before processing begins on an image, set the MAGICK_SYNCHRONIZE environment variable which is ensures the entire file can fit in the disk partition before any processing begins.
Hi magick, thanks for the background.

I've tried `export MAGICK_SYNCHRONIZE="true"`, and I can't see any difference (it still fails):

Code: Select all

free -h -s 1
              total        used        free      shared  buff/cache   available
Mem:           3.8G         94M        1.5G        3.6M        2.2G        3.4G
Swap:          3.9G         35M        3.9G

Code: Select all

grep 'resource' /etc/ImageMagick-6/policy.xml 
  <policy domain="resource" name="memory" value="1GiB"/>
  <policy domain="resource" name="map" value="1GiB"/>
  <policy domain="resource" name="width" value="32KP"/>
  <policy domain="resource" name="height" value="32KP"/>
  <policy domain="resource" name="area" value="-1"/>
  <policy domain="resource" name="disk" value="-1"/>

Code: Select all

export MAGICK_SYNCHRONIZE="true";
time /usr/bin/identify -debug cache,resource -format "%p / %w %x %U / %h %y \n" './test.tif'
2020-02-03T11:16:37+00:00 0:00.000 0.000u 6.9.7 Resource identify[30133]: resource.c/AcquireMagickResource/318/Resource
  Map: 863.5MB/823.5MiB/1GiB
2020-02-03T11:16:37+00:00 0:00.010 0.000u 6.9.7 Resource identify[30133]: resource.c/RelinquishMagickResource/1013/Resource
  Map: 863.5MB/0B/1GiB
2020-02-03T11:16:37+00:00 0:00.010 0.000u 6.9.7 Resource identify[30133]: resource.c/AcquireMagickResource/318/Resource
  Map: 863.5MB/823.5MiB/1GiB
identify: Memory allocation failed `Cannot allocate memory' @ fatal/string.c/AcquireStringInfo/182.

real	0m4.125s
user	0m1.131s
sys	0m2.989s
Where I've run this `identify` command a few times (it's the same with `convert`) - and `free` does not get above 1.3GB of "used" memory.

And I should note that during `convert`, with MAGICK_MAP_LIMIT set to 768MiB (i.e. so it works), I do see the temporary files get created in /tmp/, so file permissions aren't an issue, nor is disk space (61GB free).

And if I set MAGICK_TEMPORARY_PATH="/run/shm/", it works faster (as it's RAM based), and does not have memory limit issues.

User avatar
magick
Site Admin
Posts: 11217
Joined: 2003-05-31T11:32:55-07:00

Re: Setting map limit

Post by magick »

The pixel cache is consuming all the available memory so even some small allocation requests are refused by the system. The problem is likely that your TIFF image drops into the generic TIFF read method that requires width x height memory. Try the latest release of ImageMagick. It has been optimized, in most cases, to only use memory the size of a strip or tile. ImageMagick has control over how the pixel cache is allocated but does not have control over how much memory a delegate library consumes.

If that does not solve the problem, see https://imagemagick.org/script/architec ... tera-pixel. Based on the recommendations, try

Code: Select all

identify -debug cache -limit memory 2GiB -limit map 4GiB -define registry:temporary-path=./ -format "%p / %w %x %U / %h %y \n" './test.tif'
The pixel cache allocates any large requests on disk, which is slower, however this assures plenty of heap memory for ImageMagick to complete.

craig.francis
Posts: 6
Joined: 2020-01-31T14:22:28-07:00
Authentication code: 1152

Re: Setting map limit

Post by craig.francis »

Thanks magick,

Unfortunately I'm not in a position to compile a new version of ImageMagick at the moment.

I did try using your suggested command, and it results in the same error.

The only way I can get `identify` or `convert` to work is by setting `-limit map 768MiB`, anything larger results in the "Cannot allocate memory" error (even when `free` reports that these commands never uses more than 1.3GB).

Don't worry about it, it might be something else going on, or it could have been fixed already; so I'll keep my current work around, and hope for the best in future.

Thanks again.

User avatar
magick
Site Admin
Posts: 11217
Joined: 2003-05-31T11:32:55-07:00

Re: Setting map limit

Post by magick »

There is a portable version of ImageMagick @ https://imagemagick.org/download/binaries/magick. Its always the latest release of ImageMagick. No compiling required.

craig.francis
Posts: 6
Joined: 2020-01-31T14:22:28-07:00
Authentication code: 1152

Re: Setting map limit

Post by craig.francis »

That's impressive, thanks for making a portable version, I had no idea that existed.

However I think it's just pointed me to a fairly big issue:

Code: Select all

magick;
-bash: ./magick: cannot execute binary file: Exec format error

uname -i;
i686
Yep, a 32bit system... great, so I'll have to see what I can do to sort that one out (that server isn't mine, and it's 75 miles away).

Post Reply