I've often had thoughts much similar to yours. Another way to put them is to say that most people misunderstand what an 'appeal to authority' is, and hurl that accusation improperly. It's perfectly legit to believe in an authority, if they're really an authority, and we do something similar every time we believe a country we've never visited exists.

I tend to think that logic gives us absolute truth (could it ever be false to say that something can't both be and not be at the same time?) But how we go from the basic axioms of logic to those of mathematics and then to science is a far trickier matter.