Introduction
Azure Cognitive Search is a robust cloud-based service that enables organizations to build sophisticated search experiences. In this blog post, you will learn how to utilize Norconex Web Crawler to index data into Azure Cognitive Search and enhance your organization’s search capabilities.
Understanding Norconex Web Crawler
Norconex Web Crawler is an open-source web crawler designed to extract, parse, and index content from the web. The crawler’s flexibility and ease of use make it an excellent choice for extracting data from the web. Plus, Norconex offers a range of committers that index data to various repositories. See https://opensource.norconex.com/committers/ for a complete list of supported target repositories. If the provided committers do not meet your organizational requirements, you can extend the Committer Core and create a custom committer.
This blog post will focus on indexing data to Microsoft Azure Cognitive Search.
Prerequisites
Azure Cognitive Search
Before getting started, make sure you’ve already set up an Azure Cognitive Search service instance through your Azure portal. Consult the official Microsoft documentation for guidance on setting up this service. After completing the setup, create an Index where you will index/commit your data. Then configure the index with the following fields:
Note: For this exercise, the English – Lucene analyzer will be used for the title, description, and content fields.
Note that the following 3 items are required to configure the Norconex Azure Cognitive Search Committer:
- URL (listed on the Overview page of your Azure Cognitive Search portal)
- Admin API key (listed under Settings -> Keys)
- Index name
Norconex Web Crawler
Download the latest version of the Web Crawler from Norconex’s website. At the time of this writing, version 3.0.2 is the most recent version.
Download the latest version of Azure Search Committer. At the time of this writing, version 2.0.0 is the most recent version.
Follow the Automated Install instructions to install the Azure Search Committer libraries into the Crawler.
Crawler Configuration
We will use the following Crawler configuration for this test. Place this configuration in the root folder of your Crawler installation, with the filename my-config.xml.
