Norconex Products

OPEN SOURCE CRAWLERS

Full-featured, flexible and extensible. Run on any platform. Crawl what you want, how you want.

transparent-dots

Norconex Products

OPEN SOURCE CRAWLERS

Full-featured, flexible and extensible. Run on any platform. Crawl what you want, how you want.

Available Crawlers

HTTP Collector

Collect content from websites for your search engine or any other data repository. This full-featured collector can run independently or embed it within your own application.

Filesystem Collector

Norconex Filesystem Collector is a flexible crawler for collecting, parsing and manipulating data ranging from local hard drives to network locations into various data repositories such as search engines.

transparent-circle
transparent-dots

Features

  • Universal Crawlers
  • Easy for developers to extend
  • Commercially supported
  • Modify document metadata
  • Easy to run
  • Embeddable
  • Ease of maintenance
  • Resumable upon system failure
  • Modular design
  • Cross-platform
  • Portable
  • Open Source
  • Powerful
  • Easy to use
  • Good documentation
  • Event listeners
  • Logs are meaningful and verbose
  • Flexible