There are very few occasions, at least amongst English speakers, when one misspeaks, using a series of words which convey a meaning contrary to that which they intend, where the speaker can plead that the audience is just being too sensitive. Usually if someone says one thing but means another, they apologize or correct themselves for clarity, not complain about Political Correctness and the loss of Free Speech.
If I go to a restaurant and order the chicken and then complain to the waiter that they did not bring me steak, it would not be my right to accuse them of policing my language.
If I have to look over my shoulder before I speak because I want to make sure there are no people of colour around who may not be aware that I’m only being ironically racist, maybe I shouldn’t be talking.
If the words I use genuinely bother a group of people by evoking historical intolerance, hatred and oppression; or their own experiences with bigotry both casual and overt; then what does it matter if that wasn’t my intention.
If the word I’m using conveys what I mean, but also conveys some other meaning very different from what I intend, and I want to speak articulately, then I could I use another word. There are over one million of them available.
The backlash against so-called Political Correctness is, at worst, a defence of hatred and bigotry; and at best, it is advocacy for miscommunication and misunderstanding.