I don't think this is the way things are supposed to work. Like most listeners, the rowEditListener and the ace:ajax listeners are invoked in the Invoke Application phase, which happens after the Validation phase. Marking a component as invalid in the Invoke Application phase won't have the desired effect, because the application already changed its state (that of the table and that of other components submitted in the same form). When validation fails (at the end of the Validation phase), the lifecyle skips the Update Model Values phase and the Invoke Application phase, jumping directly to the Render phase in order to render the page without having changed the state of the application and displaying validation messages. It's in these phases where we update the row state map of the table, as well as various state settings. So, if a component in the table is marked as invalid in the Validation phase, the row editor will stay in edit (input) mode, because it's state will be unchanged. In contrast, when a component is marked as invalid in the Invoke Application phase, the state of the table and other components will have changed already, and the row editor will be in output mode now.
Ideally, input components inside ace:cellEditor, like any other components, should process their validations in the Validation phase, using a custom validator if no standard validator satisfies their purpose.
If it's really imperative for the customer to validate components in the Invoke Application phase, we could add some code specific to the ace:rowEditor component to revert it's state if validation failed after the Validation phase, so it stays in its current state. However, this may not have an effect on other components submitted in the same form; they will be rendered as if validation hadn't failed and their states will have changed accordingly.