Commons users often upload images larger than 128 Mpx. That's quite large, so they may want to crop the images using CropTool. However, CropTool uses ImageMagick from the heroku-22 stack, which sets a maximum pixel cache size of 128 Mpx.
/etc/ImageMagick-6/policy.xml:
<policymap> <policy domain="resource" name="memory" value="256MiB"/> <policy domain="resource" name="map" value="512MiB"/> <policy domain="resource" name="width" value="16KP"/> <policy domain="resource" name="height" value="16KP"/> <policy domain="resource" name="area" value="128MP"/> <policy domain="resource" name="disk" value="1GiB"/> <policy domain="delegate" rights="none" pattern="URL" /> <policy domain="delegate" rights="none" pattern="HTTPS" /> <policy domain="delegate" rights="none" pattern="HTTP" /> <policy domain="path" rights="none" pattern="@*"/> <policy domain="cache" name="shared-secret" value="passphrase" stealth="true"/> </policymap>
This file comes from the Heroku base image, but the numerical limits are the same as the ones in the Debian package, which in turn are mostly the same as the ones in an example policy file in the upstream documentation.
memory means memory allocated from malloc(), and map means an anonymous mmap(). For security purposes, these are basically the same. They are different in this configuration file because it is an example not meant for production use.
Note the powers of two in the policy: 256MiB memory, 512MiB map, 1GiB disk. It's like that because it's an example. There is no rational reason for these values to be separated by factors of two.
The Debian patch file claims to be imposing these limits for safety, but the values they came up with do not differ from the example policy, and do not have a documented justification.
Given multiple configuration sources, ImageMagick always uses the lowest specified limit. So setting a higher limit by adding a configuration file, setting an environment variable, or calling the relevant API has no effect. Overriding this limit in an application as non-root would require some extreme hackery.
So for reasonable operation of ImageMagick on Debian or Heroku, it is necessary to customise or remove the policy.xml file.
[EDIT: Wrong: see policy precedence below]
For CropTool and other applications that use PHP's imagick extension with FPM, I believe it will prove useful to reduce the "memory" limit to a low value, say 10MB, and to set the "map" limit to almost all of the pod's memory quota. With FPM, memory allocated with malloc() will usually not be returned to the OS after request termination, so the average memory usage of the pod should be reduced by encouraging ImageMagick to use mmap() for large allocations. In Toolforge, the memory quota is controlled by the user -- an administrator building the base image can't know what quota will be active when the image is used.
Commons images frequently have a width and height of much more than 16,000 px.
In CropTool, I don't want the disk to be used at all, since it's just a tmpfs so disk usage will be counted towards the memory limit of the pod, which I want to fully consume with the "map" limit.
My suggestion for a new policy.xml file:
<policymap> <policy domain="resource" name="memory" value="1TiB"/> <policy domain="resource" name="map" value="1TiB"/> <policy domain="resource" name="width" value="1MP"/> <policy domain="resource" name="height" value="1MP"/> <policy domain="resource" name="area" value="1PP"/> <policy domain="resource" name="disk" value="1TiB"/> <policy domain="delegate" rights="none" pattern="URL" /> <policy domain="delegate" rights="none" pattern="HTTPS" /> <policy domain="delegate" rights="none" pattern="HTTP" /> <policy domain="path" rights="none" pattern="@*"/> <policy domain="cache" name="shared-secret" value="passphrase" stealth="true"/> </policymap>
This will allow the limits to be set by the application when building the image, or by using the API.
Setting limits in a policy.xml unnecessarily constrains what applications can do. It is not beneficial for security.
Policy precedence
Contrary to what I said when I opened the ticket, there is an order of precedence applied when loading policy files. ImageMagick builds a list of policies, and traverses the list for each limit type, applying the first match. So the first policy file which has a value for a given limit will be used, from the following list:
*$MAGICK_CONFIGURE_PATH/policy.xml, if the environment variable exists
- Various fixed system locations
- Various user-configurable locations
- Internal defaults
So it is possible for users to override the system policy.xml, by setting MAGICK_CONFIGURE_PATH and placing a policy.xml in that directory. The file needs to actually have values — if they are missing then it will fall back to the system policy.