rbt_prs
rbt_prs is a robots.txt parser that could be useful if you want to respect others when making your bot. It's written in PHP and doesn't have any external dependencies. Note: fopen() should be allowed to download things.
rbt_prs is a robots.txt parser that could be useful if you want to respect others when making your bot. It's written in PHP and doesn't have any external dependencies. Note: fopen() should be allowed to download things.