The memory size change is due to PDF-595 which was added in 5.0.3. The fix makes sure the the base image matches the size of the image mask. The sample file in question does not render correctly in 5.0.2 and earlier, the image rendered appears much larger then it should. The reason for this is the image mask is quite a bit larger then the base image.
I'm going to update the ImageUtility so that there is choice about quality over memory.
org.icepdf.core.imageScale.quality=true
Default behavior, image will be scaled to the largest of the two images which should give higher quality output.
org.icepdf.core.imageScale.quality=false
Configurable, image will be scaled to the smallest of the two images which will save memory but the rendered output will be of lower quality
PDF-712test file.