The problem is that when you group by a date column, it uses the full underlying date time, regardless of which display format you chose. Due Dates with time become separate groups because they have a different time than the date only due dates (which are actually midnight).
Create a Math column, put in X, replace it with the Date column value, then format it to date only. Then create a template column, put in X, then replace it with the math column value. This will lock in the formatting, so it’s using the forward facing date, instead of the underlying date timestamp. Finally group my that template column value.