HP Autonomy users, take control over your web crawling. Norconex recently released an HP Autonomy IDOL Committer module for its open-source web crawler, Norconex HTTP Collector. You can now enjoy the features of Norconex crawler and experience the freedom of open-source when crawling your sites for indexing into IDOL.
We have published the 100% Java source code for this piece of software on Github, and it is freely available to anyone who wants to download it. We encourage people to use it, report issues, and contribute to the project by helping with the documentation and providing patches.
Most key features of HP Autonomy HTTP Connector are available in Norconex HTTP Collector, including document changes detection on incremental crawls and purging documents from IDOL for deleted web pages. New ones are introduced, such as having different hit intervals at different times of the day and the ability to overwrite pretty much every part of the web crawling flow with your own implementation logic. The IDOL Committer has been tested on diverse public and internal web sites with great performance.
If you have not used the Norconex HTTP Collector before, get familiar with it first. The following section focuses on the configuration related to the Norconex IDOL Committer module (version 1.0 as of this writing).
IDOL Committer Configuration Explained
The IDOL Committer will automatically populate key fields, such as DREREFERENCE and DRECONTENT, as well as any other extracted fields you wish to keep. While you can configure the Committer programmatically, most users will prefer to rely on well-documented XML configuration.
Below is the XML template for the IDOL Committer, that is to be inserted in the Norconex HTTP Collector configuration. This snippet can also be found in the IDOL Committer Javadoc.
(Host to IDOL.)
(Port to IDOL ACI)
(Port to IDOL Index.)
(IDOL Database Name where to store documents.)
(parameter value)
(parameter value)
(Name of source field that will be mapped to the IDOL "DREREFERENCE"
field or whatever "idTargetField" specified.
Default is the document reference metadata field:
"document.reference". Once re-mapped, the metadata source field is
deleted, unless "keep" is set to true.)
(Name of IDOL target field where to store a document unique
identifier (idSourceField). If not specified, default
is "DREREFERENCE".)
