It is standard in all programming languages to always stop processing once the first true condition has been met and processed.
In your case I would add a third condition in front of the other two. The first condition would check if Phone AND Address are empty, then set all necessary columns. Else, let it run through the other conditions. It’s all about the correct heirarchy.
I absolutely agree that the If Then logic needs work in glide.
@Robert_Petitto created a feature request that asks for exactly what you want.
I have a feature request that’s a little more generic and asks for better and more consistent IF Then logic throughout all of glide.