This is actually a limitation in HTML. Table column widths can't be narrower than the width necessary to display the column contents. Resizing columns works by increasing or decreasing the width of a column by explicitly setting the style attribute of the corresponding <th> element to the desired width. The browser then automatically resizes other columns to fit the column resized by the user. This is done without modifying the total width of the table. So, if a given column is made wider by the user, and all other columns can't be made narrower, because otherwise they couldn't display their contents, then no actual resizing will occur. When a table has too many columns that the viewport width isn't enough, then no column can be made narrower because all columns are already at their minimum width. Likewise, no column can be made wider, because the other columns can't be made narrower.
This can be easily seen on the following test page. Just try to set style="width: 1px;" on any of the <th> elements. The column won't go below certain width.
Even if an ace:dataTable component has fewer columns, if you try to resize any column, you will see that it can't be made wider than certain limit nor narrower than certain other limit.
The only thing that maybe could be done in these cases for making columns wider, is to also increase the width of the entire table to accommodate for this extra width. The problem is that it's not easy to detect when all the columns have reached their minimum width as set by the browsers specifically to fit the contents of such columns.
The other option is to introduce a different resizing mode to always increase or decrease the total table width when a column is resized and make this configurable. This could be specifically set for tables that have too many columns.