You're not alone, but there are reasons to avoid hosting what could reasonably be deemed pornography.
Aside from potential legal issues, sites that allow pornography can be classified differently by search engines, leading to less traffic. Some networks blacklist sites that contain pornography, which is another annoyance if someone wants to visit a site during a work break, or if they're visiting in the company of others. I'm sure there are other reasons that allowing porn isn't worth the trouble.