I'm guessing the other couples in the company are not boss/subordinate pairings. There's a big difference between 2 people on the same level (or from different departments) dating and someone dating the person who's in charge of them. And yes, your boss's boss is still your boss too, even if you don't work directly with them. Assuming you really are just a dedicated worker and your success has nothing to do with your office romance, it does suck that you can't escape the suspicion of favoritism. But any reasonable person would have seen this coming. You should have already known what the consequences of dating your boss would be.

Jealous assholes will always find a reason to talk about you. It would be best to just brush them off. Don't worry about their approval.

If you really think this relationship is going to get serious/long term, I recommend looking for a job outside the company. Not rushing it, obviously, so you don't have to take a career setback. And not because of the coworkers either, but so the job doesn't become a factor in future arguments or something to blackmail you into compliance relationship wise. Talk to your partner, maybe they know of opening positions in other companies (business people talk - he might know something), and also so that he understands your reasons and doesn't get wrong ideas about why you are looking elsewhere.