Norconex just made it easier to understand the inner-workings of its crawlers by creating clickable flow diagrams. Those diagrams are now available as part of both the Norconex HTTP Collector and Norconex Filesystem Collector websites.
Clicking on a shape will bring up relevant information and offer links to the corresponding documentation in the Collector configuration page.
While not all features are represented in those diagrams, there should be enough to improve your overall understanding and help you better configure your crawling solution.
Have a look now:
Pascal Essiembre
Pascal Essiembre has been a successful Enterprise Application Developer for several years before founding Norconex in 2007 and remaining its president to this day. Pascal has been responsible for several successful Norconex enterprise search projects across North America. Pascal is also heading the Product Division of Norconex and leading Norconex Open-Source initiatives.