This is a great point, if it comes to a head, check your lease and make sure this is accurate and then you can say:
Karen,
I have carefully reviewed my lease to confirm that I am not breaking any rules. If there is an issue between my lease and the HOA rules, you’ll have to take that up with my landlord. My lease was signed on DATE, and is a legally binding contract, not a set of suggested guidelines.
That said, I will be sure to be careful not to admit any criminals into the building, and to continue being a safe and responsible neighbor.