Is there any particular reason you are sorting in the sheet? The Row ID should be preventing this issue, but sorting can be done in the Glide builder.
Sounds similar to this thread, but if you watch the RowIDs, do they ever revert to their original value after you sort in the sheet? For example, they will move when you sort, but jump back a couple of seconds later? It could be a weird sync issue with a large amount of data moving around.