I'm not intolerant, but I do have a set of morals. If you're a pedophile in the psychological sense and don't act on it, then I say "who cares". But that's not what one normally means by "pedophile" in normal conversation.
This is something that I care nothing about, and in fact would rather not...