---
title: 'Open-Source Crawler'
url: 'https://norconex.com/products/open-source-crawler'
markdown: 'https://norconex.com/products/open-source-crawler.md'
date: '2026-07-27'
description: 'The free, open-source Norconex Crawler — Web and File System variants of one engine — turns any content source into clean, structured data for search or AI.'
---

# Norconex Crawler

One open-source engine, two purpose-built variants — **Web Crawler** and **File System Crawler** — for collecting content from anywhere and turning it into clean, structured data ready for search, AI, or agents. Free, open source, and built to run at any scale.

**Norconex Crawler V4 is here, in beta** — a rebuilt core for horizontal scalability and clustering, plus a new visual configurator. [Meet the Crawler](https://crawler.norconex.com) · [Try the Visual Configurator](https://configurator.norconex.com)

Looking for a production-ready release today? [V3 is the current stable version](https://opensource.norconex.com/crawlers/).

![](https://norconex.com/user/pages/02.products/open-source-crawler/01._hero/nexie-side.svg)

## One Engine, Two Variants

Besides the fetchers and a few source-specific pieces, both variants share the same crawl → process → commit core. Pick the one that matches where your content lives.

### Web Crawler

Best for websites, portals, and authenticated web experiences.

- Crawl modern JavaScript-heavy sites with full browser rendering
- Handle authenticated areas and logged-in user journeys
- Respect sitemaps, redirects, canonical URLs, and site rules
- Tune requests with custom headers, cookies, and user agents
- Skip unchanged content on repeat crawls

[Web Quick Start](https://crawler.norconex.com/docs/getting-started/quick-start-web)

### File System Crawler

Best for drives, shares, repositories, and internal file estates.

- Crawl local disks, network shares, and mounted repositories
- Connect to SharePoint, Nextcloud, object storage, and HDFS
- Reach remote content over FTP, SFTP, and WebDAV
- Traverse large file estates with path and depth controls
- Capture content and metadata consistently across every source

[File System Quick Start](https://crawler.norconex.com/docs/getting-started/quick-start-fs)

## Features

### Universal Output

Send crawled content to search engines, databases, queues, and custom targets without locking into one destination.

### Built to Extend

Adapt crawler behavior in plain Java when the out-of-the-box pipeline isn't enough.

### Embeddable or Standalone

Run it from the command line or embed the same engine inside your own Java applications.

### Metadata & Content Control

Enrich, filter, rename, and transform content and metadata before anything is sent downstream.

### Portable Operations

Reuse configurations, isolate environment-specific settings, and promote the same crawl setup across environments.

### Resumable & Observable

Resume after interruptions, tap into crawler events, and diagnose failures with detailed logs.

[See All Features](https://crawler.norconex.com/features)

## Need Help At Scale?

Our team offers configuration, deployment, and support packages for teams running the crawler in production — from a single-site setup to enterprise deployments across multiple sources. [Contact us for a quote](https://norconex.com/contact-us).

## Get Started

The crawler is free and open source — download the stable release and run your first crawl today, or preview what's coming in V4. Need a hand at scale? We're here.

 [Download V3 (Stable)](https://opensource.norconex.com/crawlers/) [Try V4 (Beta)](https://crawler.norconex.com/docs/getting-started) [Talk to Us](https://norconex.com/contact-us)

---

## Navigation

- Parent: [Products & Services](https://norconex.com/products.md)
- Previous: [Data & Content Enrichment](https://norconex.com/products/data-content-enrichment.md)
- Next: [Search & AI Consulting](https://norconex.com/products/search-ai-consulting.md)
