Quick fix for dialog issue is to set position of tabset to static:
<ace:tabSet id="tabSet" style="position:static;" ...
(Tested with FF and also IE9. Seemed OK.)
See screenshot 1.
Somehow the relative position (set by YUI stylesheet) of the tabset was creating an extra stacking context in IE7 which is below the overlay. Since the dialog belongs to the tabset, it is also pulled under. (No matter how high a z-index you give it.)
Still can't figure out how this happens in IE7 only. The overlay in IE is nowhere to be found in the HTML inspection tab or the source view, so it's impossible to investigate further. (In FF, it clearly shows as a div appended at the end of body.)
The dialog problem could be caused by the tabset. If we don't use tabset then it works fine. See video at http://screencast.com/t/uBYkky6E9m. Here is simplified test code:
<!-
<ace:tabSet>-><!-
<ace:tabPane>-><ui:include src="./tabs/detailPopup.xhtml"/>
<ui:include src="./tabs/screen1.xhtml"/>
<!-
</ace:tabPane>-><!-
</ace:tabSet>->Works if tabset commented out; remove comments and problem appears. So we can be sure that something in tabset is interfering with dialog in IE7?