During a recent client project, I was required to crawl several websites with specific requirements for each. For example, one of the websites required: to have a meta tag content…
Blog
Notes from the team on crawling, search, and building data pipelines that hold up in production.
HP Autonomy users, take control over your web crawling. Norconex recently released an HP Autonomy IDOL Committer module for its open-source web crawler, Norconex HTTP Collector.…
GATINEAU, QC, CANADA – Thursday, August 25, 2014 – Norconex is announcing the launch of Norconex Filesystem Collector, providing organizations with a free “universal” filesystem…
Norconex Committer and all its current concrete implementations (Solr, Elasticsearch, IDOL) have been upgraded and have seen a redesign of their web sites. Committers are…
Release 1.3.0 of Norconex Importer is now available. Release overview: Now stores the content “family” for each document as “importer.contentFamily”. New SplitTagger: Split values…
Norconex Importer 1.2.0 was just released along with a new website for it. New features: Now support text extraction from WordPerfect documents. New transformer to reduce…
