Using norobots-rfc

1. Import the class import org.apache.http.norobots.NoRobotClient;
2. Create an instance for your user-agent NoRobotClient nrc = NoRobotClient("googlebot");
3. Parse a robots.txt at a site nrc.parse( new URL( "http://www.apache.org/" ) );
4. Ask if a url is allowed boolean test = nrc.isUrlAllowed( new URL( "http://www.apache.org/index.html" ) );