Introduction
Norconex Web Crawler is a full-featured, open-source web crawling solution meticulously crafted to parse, extract, and index web content. The Crawler is flexible, adaptable and user-friendly, making it a top-notch selection for extracting data from the web.
As the volume and complexity of web crawling tasks increase, organizations face challenges in efficiently scaling the Crawler to meet organizational needs. Scaling effectively involves addressing issues related to configuration management, resource allocation, and the handling of large data sets to enable seamless scalability while maintaining data quality and integrity.
In this blog post you will learn how to handle configuration management for medium to large Crawler installations.
The Problem
Norconex Web Crawler only needs to be installed once, no matter how many sites you’re crawling. If you need to crawl different websites requiring different configuration options, you will likely need multiple configuration files. And as Crawling needs further grow, yet more configuration files will be needed. Some parts of these configuration files will inevitably have common elements as well. How can you minimize the duplication between configs?
The Solution: Apache Velocity Templates
Norconex Web Crawler configuration is not a plain XML file, but rather, a Apache Velocity template. Broadly speaking, the configuration file is interpreted by the Velocity Engine before being applied to the Crawler. You can leverage the Velocity Engine to dynamically provide the appropriate values. The following sections walk you through exactly how to do so.
Variables
To keep things simple, consider a crawling solution that contains just 2 configuration files; one for siteA and one for siteB.
Note: This scenario is for demonstration purposes only. If you only have 2 sites to crawl, the following approach is not recommended.
Default configurations
The configurations for the 2 sites may look as follows.
siteA configuration
