Chemicals known as pigments make up the material composition that causes certain types of light to reflect. So probably a reaction to a common pigment in paints, dyes, stains, etc.

Food and clothing dyes

Artificial colors. A lot of food uses food dyes.

It's bull shit to say allergic to "colors" since that is not possible. Sure, you could be allergic to something naturally colored a certain way (pigments and dyes as some people mentioned), but I'll bet that OP is not allergic to inert materials like concrete, asphalt, and metal. Your immune system has to attack a compound that it sees as a threat. This means people aren't allergic to inorganic materials. Polymers are different due to the possible used chemicals. For OP to find this out so late is suspicious too. If they're old enough to type an fml, they've lived long enough interacting with these things. I'm being an asshole, but it can't be serious allergic reactions if it was found out this late. I know allergies can come from nowhere, but not this many and not such a ridiculous list of allergens.

Concrete, asphalt, and metals are not inert. An inert material does not react to anything.

colors of mold

Unless they end up becoming allergic to glass as well.

So, OP clearly says "avoid the entire planet and everything man-made." I'm fairly certain the entire planet includes trees and soil.