Robots.txt Tester

Fetch and parse robots.txt files. Test which paths are allowed or blocked for any user-agent.