Attached test case and screen shot that shows the issue.

The issue seems to be caused by IE7's own way of handling the z-index of elements, where the actual order in which elements are displayed not only depends on the numeric values of their z-index properties alone, but their 'stacking' contexts' also determine this order. So, if a parent element has a lower z-index, no matter how high we set the z-index of the dialog, it will always be displayed under the overlay. For example, if the dialog is placed outside of the tab set (i.e. no parent elements), then the dialog will be displayed correctly on top of the overlay.

Committed fix to trunk at revision 32185.
After trying several possible solutions, what solved the problem was to bring the modal dialog overlay element to the same dom tree level as the dialog root node, so that the specified z-index values are handled as they were initially intended. This is done only for IE7 browsers. The fix had to go in the jquiery-ui code because it's not possible to access the reference to the overlay node from outside of that code. This fix was also tested outside of a tabset, and it doesn't affect the component in that case. The same situation was present with ace:confirmationDialog. This fix solves the problem for both components.
Testing notes: Just make sure that the dialog is always displayed on top of the overlay in every browser, inside or outside a tabset component.