Correct. User Specific column values are only visible to the user they belong to.
Side note: when you have computed columns that reference User Specific Columns, then the computed values in those columns also become User Specific. Obvious when you think about it, but sometimes catches people out.