Second hand smoke is much worse for the baby. Hopefully things work out.

You can have up to 200mg of caffeine per day while pregnant or breastfeeding.

Medical health professionals trump relatives and "concerned citizens." EVERY. SINGLE. TIME. You have every right to tell anyone who tries to tell you something contrary to what a doctor told you to go f*** themselves, doesn't matter who it is. If they have an issue, that's theirs. Your first priority is you and your child's health.

I'm not saying to jump straight to dumping him but this isn't the basis for a healthy long term relationship. If his mother is this much of an interference because he refuses to think for himself now, he's likely going to get worse. That's excluding his mother deciding to be a more direct nuisance. Just... If this keeps happening breaking up with his and filing for child support might be something to consider. It also might be a good idea to keep him away from the baby when is smoking or just recently has- second hand smoke, even just the trace amounts on clothing, is way more poisonous than what smokers inhale, due to the filter. It's a risk for causing allergies, asthma, and even cancer.