Norconex HTTP Collector

Release History

Version Date Description
2.9.1-SNAPSHOT 2020-??-?? Maintenance release
2.9.0 2019-12-22 Feature release
2.8.1 2018-08-17 Maintenance release
2.8.0 2017-11-26 Feature release
2.7.1 2017-05-26 Bugfix release
2.7.0 2017-04-26 Feature release
2.6.2 2017-01-06 Maintenance release
2.6.1 2016-12-14 Maintenance release
2.6.0 2016-08-25 Feature release
2.5.1 2016-07-27 Maintenance release
2.5.0 2016-06-03 Feature release
2.4.0 2016-02-28 Feature release
2.3.0 2015-11-06 Feature release
2.2.1 2015-08-07 Bug fix release
2.2.0 2015-07-22 Feature release
2.1.0 2015-04-08 Feature release
2.0.2 2015-02-04 Bug fix release
2.0.1 2014-12-03 Bug fix release
2.0.0 2014-11-26 Major release.
1.3.4 2014-08-24 Maintenance release.
1.3.3 2014-08-07 Bug fix release.
1.3.2 2014-06-17 Bug fix release.
1.3.1 2014-04-10 Bug fix release.
1.3.0 2014-03-24 Feature release.
1.2.0 2014-01-11 Feature release.
1.1.1 2013-10-03 Bug fix release.
1.1.0 2013-08-21 Feature release.
1.0.2 2013-07-11 Bug fix release.
1.0.1 2013-06-26 Bug fix release.
1.0.0 2013-06-05 Open Source release

2.9.1-SNAPSHOT Maintenance release Download 2020-??-??

This release is currently in development and the following information may change.
Can now decide to re-process headers as part of document fetching when metadata fetcher fails. #655

2.9.0 Feature release Download 2019-12-22

From Collector Core update, added "unmanaged" attribute to "logsDir" configuration option to prevent the collector from managing its own file-based logging. #593
Added new normalization rules to GenericURLNormalizer: removeQueryString, lowerCase, lowerCasePath, lowerCaseQuery, lowerCaseQueryParameterNames, lowerCaseQueryParameterValues. #479
Added "includeSubdomains" to start URL group to consider subdomains as being the same as any URL domain when "stayOnDomain" is true. #563
GenericLinkExtractor now supports using "selector-syntax" to identify portions of an HTML document to include or exclude from URL extraction.
Switch default cookie specs from DEFAULT to STANDARD and can now be configured. #525
Dependency updates: Norconex Collector Core 1.10.0, Norconex Commons Lang 1.15.1.
Stateful support added for auth sessions (e.g., NTLM). #526
Fixed GenericCanonicalLinkDetector not handling properly HTTP Response "Link" Header with multiple values. #646
Fixed GenericRecrawlableResolver large frequency units resulting in those minimum frequencies not being respected. #597
Fix quoted mediatype from content type. #578
Fixed exceptions in URLCrawlScopeStrategy#isInScope(...) ending the crawler execution. #540
Discovered canonical URLs are now rejected if they are out of scope (not matching the stayOnXXX directives). #527
Fixed good redirected URLs being committed twice in same crawl session. #533
TikaLinkExtractor now extracts URL with spaces. It should also support other URLs typically considered "bad". #628

2.8.1 Maintenance release Download 2018-08-17

Dependency updates: Norconex Collector Core 1.9.1, Norconex Commons Lang 1.15.0.
GenericDocumentFetcher now logs documents it could not fetch as INFO. #475
Fixed XMLFeedLinkExtractor sometimes splitting URLs. #487
Fixed invalid validation error when XML configuration for PhantomJSDocumentFetcher does not contain "screenshotStorageDiskDir". #473
Fixed NullPointerException when using PhantomJSDocumentFetcher without specifying any "screenshotDimensions".
Added validation support for new "authFormParams" option in GenericHttpClientFactory. #434
Fixed PhantomJSDocumentFetcher sending invalid scheme to PhantomJS proxy (when used with HttpClientProxyCollectorListener). #437

2.8.0 Feature release Download 2017-11-26

New FeaturedImageProcessor class to extract the "main" image from a web page.
New screenshot storage options for PhantomJSDocumentFetcher, like specifying image format, store inline (base64), and exact size.
New "authFormParams" option on GenericHttpClientFactory for adding arbitrary parameter to authentication forms.
New metadata field "collector.redirect-trail" for storing all redirect source URLs with the target URL metadata. #397
New "authPreemptive" configuration option on GenericHttpClientFactory to use preemptive authentication with BASIC authentication. #420
GenericLinkExtractor can now limit link extraction to specified portion of text only.
It is now possible to store as metadata URLs that are not "in-scope" with a new flag "keepOutOfScopeLinks". #360
New GenericHttpClientFactory#removeRequestHeader(String) method.
New "resourceTimeout" option on PhantomJSDocumentFetcher to prevent individual page items to hang PhantomJS for a long time. #383
Now handles circular references between a non-canonical page pointing to a URL redirecting back to that non-canonical page. When this loop is encountered, it will now process the non-canonical page (used to reject both). #162
GenericLinkExtractor now extracts unquoted URLs. #423
*.http.doc.IHttpDocumentProcessor now deprecated in favor of *.http.processor.IHttpDocumentProcessor.
Crawler event REJECTED_CANONICAL now deprecated in favor of REJECTED_NONCANONICAL.
RegexLinkExtractor now supports replacement values for regex matches.
GenericURLNormalizer XML configuration can now accepts empty normalizations tag to blank default normalizations.
Dependency updates: Norconex Collector Core 1.9.0, Norconex Commons Lang 1.14.0.
PhantomJSDocumentFetcher will now try to write a document for processing even if its status from PhantomJS is not "success" (for cases where valid content could be extracted nonetheless). #383
Blank start URLs are now silently ignored.
Fixed blank URLs extracted by RegexLinkExtractorin throwing NullPointerException. #422
Fixed blank lines in start URL seed files throwing NullPointerException. #396
Orphans being reprocessed no longer triggers the "RecrawlableResolverStage". #390
Fixed improper saving as XML of GenericDelayResolver.
Fixed GenericDelayResolver not taking into account ranges that overlap into the next cycle (e.g. 22:00 to 5:00).
Fixed setting proxy realm having no effect on GenericHttpClientFactory.
Fixed invalid validation error for "tempDir" element on StandardSitemapResolverFactory.
Fixed PhantomJSDocumentFetcher sometimes throwing a NullPointerException when content-type could not be established. #408
Removed these deprecated classes: HtmlLinkExtractor, HttpMetadataChecksummer, and deprecated methods: HttpCrawler#getUrlsFiles, HttpCrawler#setUrlsFiles.

2.7.1 Bugfix release Download 2017-05-26

Dependency updates: Norconex Collector Core 1.8.2.
Fixed "caseSensitive" flag sometimes having no effect in GenericRecrawlableResolver.

2.7.0 Feature release Download 2017-04-26

New PhantomJSDocumentFetcher which relies on an external installation of PhantomJS to fetch documents. This provides a way to crawl web pages with JavaScript-generated content. #95
New XMLFeedLinkExtractor to extract links out of RSS or Atom feeds. #319
New RegexLinkExtractor to extract links out of any text file using regular expressions. #236
Added schema-based XML configuration validation which can be trigged on command prompt with this new flag: -k or --checkcfg
Can now provide start URLs dynamically with new IStartURLsProvider. #331
Added "removeTrailingHash" to GenericURLNormalizer. #331
New "detectContentType" and "detectCharset" options on GenericDocumentFetcher for ignoring the content type and character encoding obtained from the HTTP response headers and detect them instead.
New features from dependency updates. Collector Core: ICollectorLifeCycleListener. Importer: MergeTagger, ExternalTransformer.
MongoCrawlDataStoreFactory now accepts encrypted passwords.
Now distributed with utility scripts.
XML configuration entries expecting millisecond durations can now be provided in human-readable format (e.g., "5 minutes and 30 seconds" or "5m30s").
Dependency updates: Norconex Collection Core 1.8.0, Norconex Commons Lang 1.13.0, Jetty 9.2.20.
Modified Javadoc to include an XML usage example for all XML-configurable classes.
Fixed OverlappingFileLockException when loading the a sitemap store more than once per JVM instance. #336
Fixed NullPointerException from GenericLinkExtractor when encountering HTML files with an empty href in their base tag.
Fixed minor errors in writing IXMLConfigurable classes to XML.
Removed MapDB and Apache Derby crawlstore dependencies/implementations which were deprecated in previous versions.

2.6.2 Maintenance release Download 2017-01-06

New "sslProtocols" configuration option on GenericHttpClientFactory to specify which SSL/TLS protocols are supported (e.g., SSLv3, TLSv1, TLSv1.1, TLSv1.2). Useful to enable TLSv1.2 on Java 7 when encountering SSL handshake problems on site supporting TLSv1.2 only.
Corrected documentation in collector-http-config-reference.xml to indicate default orphan strategy is now PROCESS.

2.6.1 Maintenance release Download 2016-12-14

New "addDomainTrailingSlash" normalization rule for GenericURLNormalizer.
Dependency updates: Norconex Commons Lang 1.12.3, JJ2000 5.3, Norconex Collection Core 1.7.0.
Fixed canonical URLs defined in content with single quote in them being truncated. #111
Fixed StandardRobotsTxtProivder wrongfully rejecting documents when encountering a blank "Disallow:" directive.
Fixed GenericLinkExtractor wrongfully resolving relative URLs when an HTML page had a "base href" being a relative page.

2.6.0 Feature release Download 2016-08-25

New normalization rule for GenericURLNormalizer: removeTrailingSlash. #290
Can now specify "notFoundStatusCodes" on GenericMetadataFetcher.
Specifying an empty "path" tag in XML config or, setting a null or empty string array on StandardSitemapResolverFactory#setSitemapPaths(...) method will now prevent trying to locate sitemaps using default locations and will strictly rely on sitemap URLs specified as start URLs or found in robots.txt (if enabled). #292
GenericLinkExtractor no longer extracts URL from HTML/XML comments by default. To re-enable this behavior a new "setCommentsEnabled(boolean)" method has been added. #291
Normalization rule "addTrailingSlash" in GenericURLNormalizer has been renamed "addDirectoryTrailingSlash". #290
SitemapStore now uses MVStore instead of MapDB for storing the list of processed sitemaps.
Referrer data is now always stored for GenericLinkExtractor (default) and TikaLinkExtractor.
Redirects encountered by IHttpMetadataFetcher implementations are now followed by default. #281
Dependency updates: Norconex Collector Core 1.6.0.
Renamed constant HttpMetadata.COLLECTOR_REFERNCED_URLS to HttpMetadata.COLLECTOR_REFERENCED_URLS (was misspelled).
API break: method signature changed for IHttpMetadataFetcher from Properties fetchHTTPHeaders(HttpClient httpClient, String url) to HttpFetchResponse fetchHTTPHeaders( HttpClient httpClient, String url, Properties headers)
The subject for the crawler event HttpCrawlerEvent.DOCUMENT_METADATA_FETCHED is now an instance of HttpFetchResponse.
Extracted canonical URLs now have their referrer reference stored with them.
Extracted URLs are now stored in crawl store.
Now using MVStore for sitemap store (instead of MapDB).
Fixed GenericRecrawlableResolver#MinFrequency() constructor with arguments not setting patterns correctly. #289
Fixed documents wrongfully being considered orphans when referrer was skipped for being unmodified or premature, or was in temporary error, before its URLs could be extracted. This could cause valid documents to be deleted/ignored (depending on orphan strategy used). Now those "child" URLs will be queued for processing as if they were extracted from referrer page. #278
Canonical URLs extracted are now normalized before being compared to their containing page URL, which were already normalized.

2.5.1 Maintenance release Download 2016-07-27

Updated references to test pages URLs to now be "https" in "minimum" and "complex" example configurations. #252
Dependency updates: Joda Time 2.9.4, Apache HttpClient 4.5.2, JSoup 1.8.3.

2.5.0 Feature release Download 2016-06-03

Now supports sitemap "lastmod" and "changefreq", and also allows for specifying custom minimum recrawl frequency for documents. Default implementation is GenericRecrawlableResolver (implements new IRecrawlableResolver).
New ReferenceDelayResolver that introduces delays between downloads based on matching document reference (URL) patterns. #246
Document "crawl date" and content type can be added to the crawl data and will be stored in the crawl data store (affects all ICrawlDataStoreFactory implementations).
MVStore is now the default URL crawl store.
Dependency updates: Norconex Collector Core 1.5.0.
StandardSitemapResolver now parses sitemaps with content type application/gzip (in addition to application/x-gzip).
Fixed "stayOnDomain" being true not being honored for extracted URLs with no scheme (// #243
Fixed AbstractDelay wrongfully computing seconds when logging debug statement.

2.4.0 Feature release Download 2016-02-28

New configurable IRedirectURLProvider with default implementation GenericRedirectURLProvider now better handling redirect character encoding and offering encoding options. #199
HTTP authentication password and proxy password can now be encrypted when using GenericHttpClientFactory. #178
The "" file has been moved from classes to the installation root directory.
GenericLinkExtractor now only supports these URI schemes: http, https, ftp. It is possible to overwrite these default with #setSchemes(String[]). #212
GenericLinkExtractor now attempts by default to detect the character encoding of documents before extracting links and referrer information. In addition, it is now possible to explicitly set which charset to use via the #setCharset(String) method. #194
Derby unit tests are now much faster after settings its system durability for testing to "test".
Added more logging to FORM-based authentication to help troubleshoot authentication issues.
Dependency updates: Apache HttpClient 4.5.1, Norconex Collector Core 1.4.0, Joda Time 2.9.2.
Robot rules now escapes dots and other character having special meaning in regex, before converting robot rules to regex. #200
GenericLinkExtractor now only strips the inside of script tags in HTML pages before extracting links, so that the src links can be followed if configured to do so. #232
GenericLinkExtractor is now reading characters as opposed to bytes to fix breaking some characters. #219
Robots.txt "Allow" directive is now supported properly as defined by Google bot usage. #213
URLStatusCrawlerEventListener no longer prefixes file names with "null" when no file name prefix is explicitly set. #206
Fixed both GenericLinkExtractor and TikaLinkExtractor which were not able to extract link titles when keepReferrerData was true. #204
Text is now extracted when HTML is found in an anchor body (when keepReferrerData is true). #204
GenericHttpClientFactory#trustAllSSLCertificates now disables hostname verification and algorithm constraints. #196
Fixed "lenient" flag having no effect on "sitemapResolverFactory". #193
Fixed "base href" HTML tag not being recognized when resolving relative URLs in HTML pages. #188
When the "class" attribute is not specified in "sitemapResolverFactory" for XML config, the default resolver instance is now used (as opposed to null, which was causing some configs not to be loaded properly). #187
GenericHttpClientFactory#trustAllSSLCertificates will now disable SSL SNI Extension, and unable unsafe cert algorithms that are now disabled by default since Java 7. #181

2.3.0 Feature release Download 2015-11-06

GenericHttpClientFactory now allows you to set HTTP request headers on every HTTP calls a crawler will make. #175
New crawler configuration options: stayOnProtocol, stayOnDomain, and stayOnPort. These new settings can be applied as attributes to the startURLs tag in an XML configuration, or on the object returned by HttpCrawlerConfig#getURLCrawlScopeStrategy(). This addition affects or replace previous implementations discussed in #138 , #135 , #131 , #17 , and possibly others. #144
It is now possible to specify one or more sitemap URLs as "start URLs". New HttpCrawlerConfig#[set|get]StartSitemapURLs(...) methods. #128
GenericURLNormalizer now has the few normalizations by default, as described in its Javadoc.
New StandardSitemapFactory#setPaths(...) method to specify where to look for sitemap files for each URLs processed (relative to URL root).
The "sitemap" tag used to set the ISitemapResolverFactory implementation has been renamed to "sitemapResolverFactory" to avoid confusion with the "sitemap" tag that can now be set as a start URL.
URL normalization now always takes place by default, using GenericURLNormalizer. Can be turned off by either setting it to null in the crawler configuration, or invoking GenericURLNormalizer#setDisabled(true). #145
URLs extracted from a document are now stored in "collector.referenced-urls" after they have been normalized. #145
URL redirects are now logged as REJECTED_REDIRECTED (log level INFO).
HtmlLinkExtractor has been deprecated in favor of GenericLinkExtractor.
HttpCrawlerConfig#[set|get]UrlsFiles(...) has been deprecated in favor of HttpCrawlerConfig#[set|get]StartUrlsFiles(...)
StandardSitemapFactory#setLocations(...) now deprecated in favor of being able to specify sitemaps as start URLs.
ISitemapResolver#resolveSitemaps(...) as a new argument to specify whether the sitemap locations provided were defined as "start URLs" or not.
Now logs User-Agent upon startup (log level INFO).
Dependency updates: Norconex Collector Core 1.3.0, Norconex Commons Lang 1.8.0.
Added new event types loggers to
Corrected typos and improved documentation.
Saved and loaded configuration-related classes are now equal. Methods equals/hashCode/toString for those classes are now implemented uniformly and where added where missing.
Fixed some configuration classes not always being saved to XML properly or giving errors.
Relative redirect URLs are now converted to absolute. #169
Fixed robots.txt being fetched before reference filters were executed. Robots.txt are no longer fetched for rejected reference. #168
GenericLinkExtractor now unescapes HTML entities in URLs.
Fixed ClassCastException in HttpCrawlerRedirectStrategy when using an HTTP Proxy. #167
GenericCanonicalLinkDetector now supports links that are escaped (HTML-entities). They are now unescaped before they are processed. #164
Fixed circular redirect exception. #146
GenericLinkExtractor and TikaLinkExtractor now extract meta http-equiv refresh properly when "refresh" is without quotes or not lowercase. #141
Fixed duplicate commits when multiple URL redirects are pointing to the same target URL. #135
Fixed possible URISyntaxException in GenericURLNormalizer (fixed by updated Norconex Commons Lang URLNormalizer dependency). #132

2.2.1 Bug fix release Download 2015-08-07

Dependency updates: Norconex Collector Core 1.2.1.
Fixed NullPointerException occurring inconsistently when using "thread" scope in GenericDelayResolver.
GenericDocumentFetcher now handles more special characters in URLs. Also saves itself as XML properly now. Created new unit tests for it.
StandardRobotsTxtProvider now handles empty "Disallow: " properly." #129
Fixed NullPointerException when configuring an extractor tag in XML without specifying the class attribute (now defaults to HtmlLinkExtractor).

2.2.0 Feature release Download 2015-07-22

Added support for canonical links defined in both HTTP Headers or as a link tag in an HTML document head tag. Canonical links detection is always performed unless explicitly disabled. #79
New URLStatusCrawlerEventListener class for producing reports of fetched URLs and their status. Useful for finding broken links or else.
Added three new configuration options to GenericHttpClientFactory to better deal with HTTP connectivity issues (like timeouts): "maxConnectionsPerRoute", "maxConnectionIdleTime", and "maxConnectionInactiveTime". #118
New LastModifiedMetadataChecksummer that uses Last-Modified HTTP header value for checksum purposes, replacing HttpMetadataChecksummer as the default implementation. For choosing one or more fields of your choice to create a checksum, you can now use the new GenericMetadataChecksummer from the Collector Core dependency.
New CurrentDateTagger, DateMetadataFilter, NumericMetadataFilter, TextPatternTagger, GenericSpoiledReferenceStrategizer and more new features introduced by dependency upgrades.
New method GenericDocumentFetcher#setNotFoundStatusCodes(int...) to specify one or several custom "Not Found" HTTP codes. Default is 404.
GenericHttpClientFactory default maximum connection was increased from 20 to 200 and default maximum connections per route was increased from 2 to 20. #118
New HttpFetchResponse class now passed to crawl event listeners after a document fetch instead of the IHttpDocumentFetcher used. This adds the ability to listen for specific HTTP response status code. As a consequence, IHttpDocumentFetcher now returns a HttpFetchResponse.
HttpMetadataChecksummer has been deprecated in favor of LastModifiedMetadataChecksummer.
HtmlLinkExtractor now supports specifying tags without an attribute for detecting URLs.
HtmlLinkExtractor now ignores whatever is found between "script" tags so that JavaScript-generated URLs can no longer cause trouble. #119
Dependency updates: Norconex Collector Core 1.2.0, Joda Time 2.8.1, Apache HTTP Client 4.5, Jetty Webapp 9.2.11.v20150529, Apache Ant 1.9.5.
Jar manifest now includes implementation entries and specifications entries (matching Maven pom.xml).
Improved/fixed javadoc.
HttpCrawlState#NOT_FOUND was migrated to Norconex Collector Core CrawlState#NOT_FOUND.
Fixed HTML documents being skipped when HtmlLinkExtractor found a URL of invalid format. Now a warning is thrown for each bad URLs instead and the document is processed anyway, and good URLs are extracted. #119
Fixed MongoDB stage/depth compound index. #97
Fixed MongoCrawlDataSerializer "referrerLinkText" link attribute/metadata having the same value as "referrerLinkTag". #82
HtmlLinkExtractor now decodes encoded ampersands present in URLs. #88
Both HtmlLinkExtractor and TikaLinkExtractor no longer extract empty href links. Added this use case to corresponding unit test. #87
HtmlLinkExtractor now strips leading spaces.
Fixed "trustAllSSLCertificates" configuration option on GenericHttpClientFactory not being recognized in XML config. #100
Fixed exceptions thrown in StandardRobotsTxtProvider when robots.txt contained rules ending with ? or when the referrer URL was starting with a space.
Several invalid characters are now supported in URLs (e.g., commas).
Fixed GenericDelayResolver not saving in XML properly and fixed its javadoc as well, which did not mention how to pass the delay in XML config.
TargetURLRedirectStrategy no longer throws an exception when redirects are disabled for a request. #124
Fixed "sitemapLocations" and "lenient" not being applied to StandardSitemapFactory. #120

2.1.0 Feature release Download 2015-04-08

Several new features, updates and fixes were added by upgrading Norconex Collector Core ( and Norconex Importer ( dependencies. Those include support for ORC, translation, a title generator, new content type parsing, and more. Refer to dependency release notes for more details.
New methods and configuration attribute to disable checksum creation in HttpMetadataChecksummer.
Sitemap resolving pipeline stage is now always invoked (but won't do anything if disabled).
Dependency updates: Norconex Collector Core 1.1.0, JUnit 4.12, Joda-Time 2.7, Apache HTTP Components 4.4, Jetty Webapp 9.2.10.v20150310, Fongo 1.6.2.
Added Sonatype repository to pom.xml for snapshot releases.
Updated several maven plugins and added SonarQube maven plugin.
Improvements on character encoding detection from HTTP headers. from ./classes/ now properly loaded by #59
Improved javadoc.
Added many unit tests for testing start vs resume vs stop vs deleted vs modified, in different JVM instances.
"minimum" and "complex" configuration examples now ignore sitemap.xml files.
Fixed link extractor not fetching link text properly when keepReferrerData is true on HtmlLinkExtractor. #56
Robot meta data found in HTML pages will no longer be extracted if found within an HTML comment. Robot metadata detection is also more robust. #60
Fixed NPE in HttpImporterPipelineUtil#enhanceHTTPHeaders when content type from HTTP header is not defined.
Fixed log4j log levels incorrectly ending with a semi-colon.

2.0.2 Bug fix release Download 2015-02-04

Fixed the collector "stop" action having no effect. #49
Fixed crawl data wrongfully applied as metadata after the import phase.
Fixed NullPointerException when sitemap support is disabled.
Fixed incorrect deletion behavior for embedded orphan documents.
Improved logging options for crawler events.
Upgraded Norconex Collector Core dependency to 1.0.2.

2.0.1 Bug fix release Download 2014-12-03

From collector-core-1.0.1: When keepDownloads is true, saved files and directories are now prefixed with "f." and "d." respectively to avoid collisions. #44
Fixed errors in example configuration files.

2.0.0 Major release. Download 2014-11-26

Upgraded Norconex Importer to version 2.0.0, which brings to Norconex HTTP Collector a lot of new features, such as: Document content splitting, splitting of embedded documents into individual documents, new taggers for language detection, changing character case, parsing and formatting dates, providing content statistics, and more. Please read the Norconex Importer release notes for a complete list of changes at:
Can now supplied a "pathsFile" as part of the startPaths, acting as a seed list.
New fast MVStore database implementation for URL database (from Norconex Collector Core).
New H2 database implementation for URL database (crawl data store).
Now keeps track of parent references (for embedded/split documents).
More unit tests, with the addition of an embedded Jetty Web server rendering test pages for some unit tests.
New JMX/MBean support added on crawlers.
IUrlExtractor is now ILinkExtractor and both their implementing classes (HtmlLinkExtractor and TikaLinkExtractor) now support also extracting a link title and text, and they also support the "nofollow" robot rule. #23
It is now possible to configure multiple link extraction classes, each taking effect on particular URLs and/or content-types.
IHtmlLinkExtractor can be configured to use specified HTML tags and attributes to find URLs.
Now licensed under The Apache License, Version 2.0.
Replaced the configuration option "deleteOrphans(true|false)" with "orphansStrategy(DELETE|PROCESS|IGNORE)".
The collector now references document content as reusable InputStream with memory caching instead of relying only on files. This saves a great deal of disk I/O and improves performance in most cases.
Refactored to use the new Norconex Collector Core library. A significant portion of the Norconex HTTP Collector code has been moved to that core library. Some of the moved classes are (base package being com.norconex.collector.*): http.checksum.IHttpDocumentChecksummer to core.checksum.IDocumentChecksummer, http.checksum.IHttpHeadersChecksummer to core.checksum.IMetadataChecksummer, http.checksum.DefaultHttpDocumentChecksummer to core.checksum.impl.MD5DocumentChecksummer, http.filter.IURLFilter to core.filter.IReferenceFilter http.filter.IHttpHeadersFilter to core.filter.IMetadataFilter http.filter.IHttpDocumentFilter to core.filter.IDocumentFilter http.filter.impl.ExtensionURLFilter to core.filter.impl.ExtensionURLFilter http.filter.impl.RegexHeaderFilter to core.filter.impl.RegexMetadataFilter http.filter.impl.RegexURLFilter to core.filter.impl.RegexReferenceFilter
Amongst others, the following classes were renamed (within com.norconex.collector.http.*): checksum.impl.DefaultHttpHeadersChecksummer to checksum.impl.HttpMetadataChecksummer, client.impl.DefaultHttpClientFactory to client.impl.GenericHttpClientFactory, delay.impl.DefaultDelayResolver to delay.impl.GenericDelayResolver, fetch.impl.DefaultDocumentFetcher to fetch.impl.GenericDocumentFetcher, fetch.impl.SimpleHttpHeadersFetcher to fetch.impl.GenericHttpHeadersFetcher, robot.impl.DefaultRobotsMetaProvider to robot.impl.StandardRobotsMetaProvider, robot.impl.DefaultRobotsTxtProvider to robot.impl.StandardRobotsTxtProvider, sitemap.impl.DefaultSitemapResolver to sitemap.impl.StandardSitemapResolver, url.impl.DefaultURLExtractor to url.impl.GenericURLExtractor
Several references to "url" were changed to "reference".
New and more scalable crawler event model along with new listeners.
Refactored to use JEF 4.0.0 which makes the HTTP Collector easier to monitor.
Other libray upgrades: Norconex Committer to 2.0.0 and Norconex Commons Lang to 1.5.0.
Removed previously deprecated classes.
Crawled sitemap details now has its own store (no longer mixed with the crawl data store).
ISiteMapResolver now needs an ISiteMapResolverFactory.
Sitemap resolution now stops when a stop request is issued. #38
Now checks if crawler is running before attempting to stop it. #37

1.3.4 Maintenance release. Download 2014-08-24

MongoCrawlURLDatabase now supports user authentication.
Now requires Java 7 or higher.
Fixed DefaultRobotsTxtProvider failing to parse some robots.txt patterns. #36
Fixed DefaultRobotsTxtProvider failing to parse some robots.txt patterns. #36

1.3.3 Bug fix release. Download 2014-08-07

Upgraded JEF to 3.0.1 to fix stop action not working.
Fixed NullPointerException in robots.txt resolution under some circonstances.

1.3.2 Bug fix release. Download 2014-06-17

DefaultURLExtractor no longer treat empty href as being a URL ending with a double-quote.
Renamed HttpMetadata key "collector.http.dept" to "collector.http.depth" (typo fix).
Upgraded Norconex Commons Lang to 1.3.2
GenericURLNormallizer no longer rejects URLs with spaces in them. It now logs a warning instead (thanks to Norconex Commons Lang upgrade).

1.3.1 Bug fix release. Download 2014-04-10

Header and document checksum value are no longer added by default to prevent the issue described in github ticket #24 . Instead, adding checksum is now an optional feature of DefaultHttpDocumentChecksummer and DefaultHttpHeadersChecksummer. #24

1.3.0 Feature release. Download 2014-03-24

Now supports NTLM authentication. SPNEGO and Kerberos were also added but are experimental (see DefaultHttpClientFactory).
Can now specify character set of HTTP connections and authentication forms.
Can now set custom timeout values on HTTP connection-related activities.
New option to trust all SSL certificates of sites being crawled (see DefaultHttpClientFactory).
Can now specify a maximum number of HTTP connections for each crawler independently of configured number of threads (see DefaultHttpClientFactory).
DefaultHttpClientFactory introduces additional configuration options: proxy scheme, 'Expect: 100-continue' handshake, maximum HTTP redirects, local address, stale connection checks
HTTP header checksum and document checksum are now added to the document metadata as HttpMetadata#CHECKSUM_HEADER and HttpMetadata#CHECKSUM_DOC.
The empty sub-folders contained under the "download" folder are now periodically deleted. This speeds up directory scanning and increases performance on large crawls.
The userAgent is now a crawler configuration option (previously was an option of DefaultHttpClientInitializer )
API change: IRobotsTxtProvider#getRobotsTxt(...) method signature has changed to accept the User-Agent.
IHttpClientInitializer is now deprecated in favor of IHttpClientFactory, giving you more control over HttpClient creation.
API change: Methods previously accepting DefaultHttpClient instances now have their signature accepting parent interface HttpClient instead.
More logging to help resolve crawler issues with DEBUG log level.
HttpCrawler more lenient upon encountering some errors that were previous aborting entire execution.
Library upgrades. Updated default crawl url database (MapDB) to version 0.9.10, Norconex Commons Lang to 1.3.0, Norconex Committer to 1.2.0, Norconex Importer to 1.2.0, and Apache HttpClient to 4.3.2.
Now ensures that robots.txt agent matching gives priority to the most specific match (as opposed to the first match). Sitemaps detected in robots.txt are also preserved for sitemap resolving.
Removed classes deprecated since 1.1.

1.2.0 Feature release. Download 2014-01-11

New optional Mongo URL Database implementation.
New TikaURLExtractor class providing an alternate IURLExtractor implementation based on Apache Tika HTMLParser.
New SegmentCountURLFilter class for filtering URLs having a specified number of segments (can check duplicate segments too).
New unit tests.
MapDB URL Database classes moved to its own "mapdb" package. DefaultCrawlURLDatabaseFactory still exists, but is just a pointer to MapDBCrawlURLDatabaseFactory.
Example configurations now point to Norconex test pages to ensure their stability.
Upgraded dependent libraries: Norconex Committer 1.1.0, Norconex Commons Lang 1.2.0, MapDB 0.9.8 and other thrid party libraries.
Improved Javadoc.

1.1.1 Bug fix release. Download 2013-10-03

Fixed not being able to extract the "href" attribute when it starts a new line. #18
Fixed HTTP redirects not storing final target URL but the source URL instead. #17
Upgraded dependent libraries to Norconex Importer 1.1.0 and Norconex Commons Lang 1.1.0.

1.1.0 Feature release. Download 2013-08-21

Crawler now fires additional events. Added "documentRobotsMetaRejected" and "documentImportRejected" methods to IHttpCrawlerEventListener.
DefaultCrawlURLDatabase now uses a MapDB-based implementation for faster performance. The derby implementation has been kept for those with a preference for it.
Now support sitemap.xml and sitemap index (plain or gzip). URLs from sitemaps will have the sitemap information as metadata.
BASIC and DIGEST authentication now supported.
Now supports in-page robot instructions. Via "ROBOTS" meta tag first, or "X-Robots-Tag" tag if present in HTTP header.
"ftp" protocol now supported.
It is now possible to specify the scope of each delay between URL download with DefaultDelayResolver (per crawler, site, or thread). "crawler" is default.
Javadoc Jar and Source Jar are now also deployed to Maven repository.
Deprecation of *.handler.* package. Classes have been moved to more intuitive packages.
IDelayResolver are no longer systematically synchronized (i.e. accessible only one thread a a time). This is a decision left up to each implementation.
Reduced the number of calls to the crawl database to improve performance (URL filtering but be successfully passed for a document to get queued for processing).
Upgraded to Norconex Importer from 1.0.0 to 1.0.2.
Upgraded to Norconex Commons Lang from 1.0.0 to 1.0.1.
Connection timeout now 30 seconds in DefaultHttpClientInitializer.
When deleting downloaded files, the text file from the importer output is now also deleted as URLs are being processed, greatly reducing the number of files on disk during the crawling process.
The application now prints the full path of an error file when an error occurs on startup (error-*.log).
When reprocessed, orphan URLs are processed/filtered in case there has been a configuration change. For instance, if if the maxDepth was reduced, former URLs that were once good may now be invalid due to their depth being too big. Only "valid" URLs are now considered for detecting orphans (OK or UNMODIFIED statuses). #16
Fixed when cloning HttpCrawlerConfig (when is used).

1.0.2 Bug fix release. Download 2013-07-11

DefaultURLExtractor now handle URLs starting with ? properly and those prefixed with "URL=" (e.g. meta http-equiv="refresh")

1.0.1 Bug fix release. Download 2013-06-26

Relative URLs starting with two slashes (//) are now extracted properly. #15
Replacements now working for GenericURLNormalizer. #15

1.0.0 Open Source release Download 2013-06-05

Starting with this release, Norconex HTTP Collector is open-source under GPL.

Copyright © 2013-2020 Norconex Inc.