public interface RobotRules
Modifier and Type | Method and Description |
---|---|
long |
getCrawlDelay()
Get Crawl-Delay, in milliseconds.
|
long |
getExpireTime()
Get expire time
|
boolean |
isAllowed(URL url)
Returns
false if the robots.txt file prohibits us
from accessing the given url , or true otherwise. |
long getExpireTime()
long getCrawlDelay()
boolean isAllowed(URL url)
false
if the robots.txt
file prohibits us
from accessing the given url
, or true
otherwise.Copyright © 2015 The Apache Software Foundation