Page 1 of 1
needed memory should be computed in advance
Posted: 2011-10-15T13:06:37-07:00
by matteosistisette
Consider this command:
imagemagick -crop 128x128 srcimage.png dstimage.png
I ran this with a big source image (about 14,000x12,000 pixels) and it ran for about 20 minutes before failing. There's no unpredictable factor affecting the memory needed to perform such an operation; the amount of memory needed can be perfectly known in advance, and hence it is fundamentally wrong to start performing the operation just to find out, when probably the last-but-one bit is being processed, that there is not enough memory.
Re: needed memory should be computed in advance
Posted: 2011-10-15T13:58:04-07:00
by magick
ImageMagick asks for memory and your OS claims the memory is available but later on decided its not. Your OS has an option that ensures the memory is pre-allocated.
Re: needed memory should be computed in advance
Posted: 2011-10-15T15:59:26-07:00
by matteosistisette
But then, instead of "asking" for available memory and later take it (when in the meantime it has become unavailable), can't you just "take" it from the beginning?
Re: needed memory should be computed in advance
Posted: 2011-10-15T16:01:48-07:00
by magick
This is controlled by the OS, not the application layer. Its not a problem specific to ImageMagick. The OS can kill any application with large memory requirements. You must configure the OS to fix this problem.
Re: needed memory should be computed in advance
Posted: 2011-10-15T17:46:24-07:00
by matteosistisette
This has nothing to do with killing. I m not talking about imagemagick getting killed. ThT was another post. This is not related. The question here is why allocating memory piece by piece if the total amount needed is known in advance and the task cannot be completed unless all the memory can be allocated.
Re: needed memory should be computed in advance
Posted: 2011-10-15T17:59:53-07:00
by magick
ImageMagick asks and gets the memory from the OS to support an entire image. Only after it starts using the memory does the OS then decide that it will not provide the remaining memory that has already been allocated. Again, this is an memory allocation algorithm controlled by the OS and it can be turned off. You can tell your OS that once memory is allocated it must be available to the application:
By default, Linux follows an optimistic memory allocation strategy.
This means that when malloc() returns non-NULL there is no guarantee
that the memory really is available. In case it turns out that the
system is out of memory, one or more processes will be killed by the
OOM killer. For more information, see the description of
/proc/sys/vm/overcommit_memory and /proc/sys/vm/oom_adj in proc(5), and
the kernel source file Documentation/vm/overcommit-accounting.
Re: needed memory should be computed in advance
Posted: 2011-10-16T05:15:14-07:00
by matteosistisette
Then maybe the issue I'm observing is a different one? Because in this case it doesn't look like imagemagick gets kiled at all. It terminates with a lot of error messages.
In one case I did observed it being killed and it only said "killed" without any error message at all. But here what I'm talking about is that it outputs a lot of error messages about not being able to do dunnow-what with strings and stuff, and then terminates, and the work "killed" never appears.
Re: needed memory should be computed in advance
Posted: 2011-10-16T05:16:55-07:00
by matteosistisette
Oh I see: maybe it's ghostscript (or something) (called by imagemagick) that gets killed, right?