Yes there is, and it happens in other countries (yes, they do really exist) where there are no connotations of racism with that phrase.
Also, you don’t have to take the stairs in that situation, but it would be considerate if people did.
“Why should we do xyz which is no trouble to us, so as not to bother each other” is quite the deplorable, isn’t it?