Don't we have a duty in society to protect the speech that we hate?

We have no problem with that. What we have a problem with is the physical act of desecrating the flag. Say whatever you will, but don't defile the flag of our nation.