I have met many people like them who will call me "selfish" when I can't provide what they asked me for. I believe that they have a problem with emotional issues. Sometimes we just need to smile at them and say we're sorry for not able to share them at the moment.
When I first read this I though you wrote 'smite' them (instead of smile) and thought, hmm, not a bad suggestion! =-)
But yes, knowing your limits (setting boundaries?) isn't selfish at all.