public class Http extends HttpBase
Modifier and Type | Field and Description |
---|---|
static org.slf4j.Logger |
LOG |
accept, acceptLanguage, BUFFER_SIZE, maxContent, proxyHost, proxyPort, responseTime, timeout, tlsPreferredCipherSuites, tlsPreferredProtocols, useHttp11, useProxy, userAgent
CHECK_BLOCKING, CHECK_ROBOTS, X_POINT_ID
Constructor and Description |
---|
Http()
Constructs this plugin.
|
Modifier and Type | Method and Description |
---|---|
Collection<WebPage.Field> |
getFields() |
protected Response |
getResponse(URL url,
WebPage page,
boolean redirect)
Fetches the
url with a configured HTTP client and gets the
response. |
static void |
main(String[] args)
Main method.
|
void |
setConf(Configuration conf)
Reads the configuration from the Nutch configuration files and sets the
configuration.
|
getAccept, getAcceptLanguage, getConf, getMaxContent, getProtocolOutput, getProxyHost, getProxyPort, getRobotRules, getTimeout, getTlsPreferredCipherSuites, getTlsPreferredProtocols, getUseHttp11, getUserAgent, logConf, main, processDeflateEncoded, processGzipEncoded, useProxy
public Collection<WebPage.Field> getFields()
public void setConf(Configuration conf)
setConf
in interface Configurable
setConf
in class HttpBase
conf
- Configurationpublic static void main(String[] args) throws Exception
args
- Command line argumentsException
protected Response getResponse(URL url, WebPage page, boolean redirect) throws ProtocolException, IOException
url
with a configured HTTP client and gets the
response.getResponse
in class HttpBase
url
- URL to be fetcheddatum
- Crawl dataredirect
- Follow redirects if and only if trueProtocolException
IOException
Copyright © 2015 The Apache Software Foundation