It won’t work if the underlying times are different…even by a few milliseconds. The underlying values have to match exactly, regardless of any visual formatting you may have on the date column.
Something along the lines of creating a math column to create a stable date value will work out much better when using it for relations.