You definitely should fight it. What the hell are they expecting you to smell like? Roses? You're the freaking GARBAGE man! These are the types of situations where I think going to court would be justified if they refuse to give you your job back.
If there ever is garbage man that smells like roses, then I would just go gay for that guy. I mean, imagine how great he smells if he didn't handle garbage.
It could possibly get costly OP, but if you win (which I don't see how you wouldn't), then you would be reimbursed. If I were you I would see if I could get a sum of money for my trouble to hold me over until I find a new job because I wouldn't want to work for those people after that.
EDIT: I'm not sure how the system works in Canada as I live in the US. So it would probably do you good to do some research on that.
Small claims court doesn't have to be costly. It's usually just you, the offender, and a judge. My wife almost took her old boss to small claims, then she found her new kick ass job and we decided to drop her old boss. Not worth the fight.