Introduction
In the era of data-driven decision-making, efficient data indexing is pivotal in empowering businesses to extract valuable insights from vast amounts of information. Elasticsearch, a powerful and scalable search and analytics service, has become popular for organizations seeking to implement robust search functionality. Norconex Web Crawler offers a seamless and effective solution for indexing web data to Elasticsearch.
In this blog post, you will learn how to utilize Norconex Web Crawler to index data to Elasticsearch and enhance your organization’s search capabilities.
Understanding Norconex Web Crawler
Norconex Web Crawler is an open-source web crawler designed to extract, parse, and index content from the web. The crawler’s flexibility and ease of use make it an excellent choice for extracting data from the web. Plus, Norconex offers a range of committers that index data to various repositories. See https://opensource.norconex.com/committers/ for a complete list of supported target repositories. If the provided committers do not meet your organizational requirements, you can extend the Committer Core and create a custom committer.
This blog post will focus on indexing data to Elasticsearch.
Prerequisites
Elasticsearch
To keep things simple, we will rely on Docker to stand up an Elasticsearch container locally. If you don’t have Docker installed, follow the installation instructions on their website. Once Docker is installed, open a command prompt and run the following command.
docker run -d -p 9200:9200 -p 9600:9600 -e "discovery.type=single-node" -e "xpack.security.enabled=false" elasticsearch:7.17.10
This command does the following
- requests version 7.17.10 of Elasticsearch
- maps ports 9200 and 9600
- sets the discovery type to “single-node”
- disables the security plugin
- Starts the Elasticsearch container
Once the container is up, browse to http://localhost:9200 in your favourite browser. You will get a response that looks like this:
{
"name" : "c6ce36ceee17",
"cluster_name" : "docker-cluster",
"cluster_uuid" : "gGbNNtDHTKCSJnYaycuWzQ",
"version" : {
"number" : "7.17.10",
"build_flavor" : "default",
"build_type" : "docker",
"build_hash" : "fecd68e3150eda0c307ab9a9d7557f5d5fd71349",
"build_date" : "2023-04-23T05:33:18.138275597Z",
"build_snapshot" : false,
"lucene_version" : "8.11.1",
"minimum_wire_compatibility_version" : "6.8.0",
"minimum_index_compatibility_version" : "6.0.0-beta1"
},
"tagline" : "You Know, for Search"
}
Elasticsearch container is now up and running!
Norconex Web Crawler
Download the latest version of the Web Crawler from Norconex’s website. At the time of this writing, version 3.0.2 is the most recent version.
Download the latest version of Elasticsearch Committer. At the time of this writing, version 5.0.0 is the most recent version.
Follow the automated installation instructions to install the Elasticsearch Committer libraries into the Crawler.
Crawler Configuration
We will use the following Crawler configuration for this test. Place this configuration in the root folder of your Crawler installation, with the filename my-config.xml.
