No, it’s NOT “just a saying”. It’s an insult stemming from prejudice. Using that word in such a way continues that prejudice.
No, it’s NOT “just a joke”. Using it continues the abuse of disabled people by ridiculing and Othering them.
No, people DO care — I’m a person, and I care. I’m not the only one who cares. How else would I take it, impersonally?
“Don’t say that word. It still hurts. I don’t like it. STOP.”
http://qw88nb88.wordpress.com/2008/04/16/dont-say-that-word/