This turned out to be a bit of a rabit hole. The memory manager system properties work a little differently then one might first thing. The maxMemory property dates back to JDK 1.1 where there was no way of getting the max memory assigned to the JDK, infact maxMemory has no effect on ICEpdf 3.x > as we can get the max memory of the jvm via JDK 1.5 support. The minMemory specifies the amount of memory that should be kept free on the heap which is a bit misleading.
While in the memory code base it became apparent that we weren't freeing up as much memory as we could when a uses the call checkMemory(). The checkMemory() is called before a large resource is to be put into memory. However it may be some time before this call will be made leading to significant memory creep far above that set by minMemory. To help keep the memory spikes to a minimum the method MemoryManger.isLowMemory() is now called before a page is initialized. This helps keep the memory contract specified by minMemory in check.
So if a developer is using ICEpdf in their application that has a -Xmx512m and they want ICEpdf to only ever use 128MB of memory then they would set -Dorg.icepdf.core.minMemory=368m. I did quite a bit of profiling and the core library seem to respect his. Then only exception is during a large resource load which might cause a slight spike above the desired thresh hold but the memory manager will quickly come in and straighten things out.
Should note that these changes should be made to the MemoryManager Class.