Installed and ran Wireshark on the server, capturing network traffic between the IE client machine and the OS X server, and found that the xml updates are being transmitted from the server and acknowledge by the client. That is, the packets for the response are receiving their ACK. So, definitely the response is being sent by the server, it's just a question of what the client application is doing with the xml updates response.
I added a timer that would periodically check the hidden iframe for the response xml, in case the response was being received by the browser but maybe the iframe onload wasn't getting called for whatever reason. With the smaller uploads, that completely work, the polling would find the response in the DOM right after the iframe's onload would fire. So, doubly working right. But in the large file scenario with no iframe onload, the polling would never find it either.
From Wireshark I saw that the server push with the push id, and the client post to get the progress percent file were using different sockets, so that worried me that with those two plus the file upload we were exceeding the max 2 sockets, which might somehow becausing the file upload POST response to get dropped or ignored. So I took out the POSTs for the progress percent file, which should bring the sockets count down by one, and leave the progress in indeterminate mode. But that didn't fix it. So I commented out all icepush code use in the fileEntry javascript and removed the icepush jar, so that it should just be the one upload socket. But that didn't fix it.
I did notice from Wireshark that the server was sending a FIN flag, which means that the data is over and the socket is being closed. Investigating that pointed to the keepalive time being exceeded, so in case the response was being discarded on a closed socket, I reconfigured my tomcat 7.0.23 to have keepAliveTimeout="-1" and so disable the keepalive timing out, and keep the socket open indefinitely. In Wireshark I could see then no more FIN flag. But that didn't fix it.
I've uploaded two files to the iceads1 server (P:\Users\arran\Case 11732) to help in reproducing this issue.
(Restricted to icesoft-internal-developers group)