In this scenario in Chrome and Firefox (not IE11) when an ace:dialog is shown the modal layer is show in front of the dialog, making it impossible to close or interact with.

The root cause of this issue is the addition of some custom CSS which is applied to a div that is a parent/container of the dialog content. For example:

<div style="position: fixed;">
     <ace:dialog id="testDialog"
            modal="true"
            closable="true"
            header="Test Dialog"
            styleClass="dialog"
            draggable="false">
                        <h:outputText value="Test Dialog"/>
     </ace:dialog>
</div>

Testing with the latest code base (EE 3.3.x and 4.x) it looks like this issue has been resolved but another behavior is shown. The modal layer is placed correctly behind the dialog content but the modal layer only is shown for the content in the <div style="position: fixed;">, anything outside of this is not effected by the modal layer.