"It's our inherent individual right to act according to what we feel is in our own best interests." "Whatever path people take is right for them." "Your right to violent behavior stops when your fist meets my nose." "A private negotiation between two individuals is no one else's business." Have you heard these, or something like them?
Is there ever a time when my actions have no effect on "innocent bystanders"? Even my thoughts have a high probability of eventually effecting my actions, therefore others, therefore everyone. That entity we call "humanity", and the "C words", community, cooperation, communication and consciousness are also important to me. My relationships, which I have to some degree with each and every sentient entity, affect these values for others as well as for myself. But of course the further away the effects get from us in time, space and steps, the less predictable is the effect. A distant "relationship" is effected by many others. As my effect gets more distant, it fades into the chaos of "white effect" (like in white light and white noise, where many kinds mix to become indistinguishable.
To the extent that we each put only our immediate individual interests first, we are all potential enemies, since your individual interests will sometimes conflict with mine. If you want inherently limited funds of whatever kind to go to your projects and friends, and I want it to go to my projects and friends, how can we ever agree? By finding joint project and principle we can agree on, such as the "C words".
So is it ever valid to say, "It's none of your business"? Our right to privacy compromises the above values.