Tsk tsk, some people, just tell them straight that you are indeed British and such.
Not your fault that you talk the way you do, some people like to notice the smallest things.
I hope you are not self conscious about the way you talk..
Good thing it was not worse though, they could have commented the way you walk or something
Or the way you breathe