Despite all the “noise” on social media sites, we can’t deny how valuable information found on social media networks can be for some organizations. Somewhat less obvious is how to harvest that information for your own use. You can find many posts online asking about the best ways to crawl this or that social media service: Shall I write a custom web scraper? Should I purchase a product to do so?
This article will show you how to crawl Facebook posts (more…)
GATINEAU, QC, CANADA – Monday, December 1, 2014 – Norconex announces the launch of its Google Search Appliance (GSA) Committer module for its Norconex Collectors Crawler Suite. Enterprise search developers and enthusiasts now have a flexible and extensible option for feeding documents to their GSA infrastructure. GSA is a target repository for crawled documents released by Norconex HTTP Collector, Norconex Filesystem Collector, and any future Collector released by Norconex . These Collectors can reside on any server (like remote filesystems) and send discovered documents across the network to a GSA installation. The GSA Committer is the latest addition to the growing list of Committers already available to Norconex Collector users: Apache Solr, Elasticsearch, HP IDOL, and Lucidworks.
“The increasing popularity of our universal crawlers motivates us to provide support for more search engines. Search engines come and go in an organization, but your investment in your crawling infrastructure can be protected by having re-usable crawler setups that can outlast any search engine installation,” said Norconex President Pascal Essiembre.
GSA Committer is part of Norconex’s commitment to delivering quality open-source products backed by community or commercial support. GSA Committer is available for immediate download at /collectors/committer-gsa.
Founded in 2007, Norconex is a leader in enterprise search and data discovery. The company offers a wide range of products and services designed to help process and analyze structured and unstructured data.
For more information on GSA Committer:
GSA Committer Website: /collectors/committer-gsa
Norconex Collectors: /collectors
Email: info@norconex.com
Norconex just released major upgrades to all its Norconex Collectors and related projects. That is, Norconex HTTP Collector and Norconex Filesystem Collector, along with the Norconex Importer module and all available committers (Solr, Elasticsearch, HP IDOL, etc), were all upgraded to version 2.0.0.
With these major product upgrades come a new website that makes it easier to get all the software you need in one location: the Norconex Collectors website. At a quick glance you can find all Norconex Collectors and Committers available for download.
Among the new features added to your crawling arsenal you will find:
- Can now split a document into multiple documents.
- Can now treat embedded documents as individual documents (like documents found in zip files or in other documents such as Word files).
- Language detection (50+ languages).
- Parsing and formatting of dates from/to any format.
- Character case modifiers.
- Can now index basic content statistics with each documents (word count, average word length, average words per sentences, etc).
- Can now supply a “seed file” for listing start URLs or start paths to your crawler.
- Document content reads and writes are now performed in memory up to a configurable maximum size, after which the filesystem gets used. This reduces I/O and improves performance.
- New event model where listeners can listen for any type of crawler events.
- Can now ignore parsing of specific content types.
- Can filter documents based on arbitrary regular expressions performed on the document content.
- Enhanced debugging options, where you can print out specific field content as they are being processed.
- HTTP Collector: Can add link names to the document the links are pointing to (e.g. to create cleaner titles).
- More…
Another significant change is all Norconex open-source projects are now licensed under The Apache License 2.0. We hope this will facilitate adoption with third party commercial offerings.
It is important to note version 2.0.0 are not compatible with their previous 1.x version. The configuration options changed in many areas so do not expect to run your existing configuration under 2.0.0. Please refer to the latest documentation for new and modified configuration options.
Visit to the new Norconex Collectors website now.
HP Autonomy users, take control over your web crawling. Norconex recently released an HP Autonomy IDOL Committer module for its open-source web crawler, Norconex HTTP Collector. You can now enjoy the features of Norconex crawler and experience the freedom of open-source when crawling your sites for indexing into IDOL. (more…)
Norconex just released version 1.2 of Norconex HTTP Collector, its open-source web crawler. Along with it comes a complete product web site redesign and a new logo: a lovely web crawling spider wearing a Norconex hat.
Some changes in this feature release: (more…)
2013/06/05
Say hello to Norconex HTTP Collector! At Norconex, we have always recognized the value open-source brings to software development, and to a greater extent, the world. It benefits us when building custom solutions for our customers and ourselves. As long-time consumers of open-source, it is time for us to give back.
As a result, Norconex is proud to announce open-sourcing of a handful of its libraries and products, so that the community can save time and money like it did for us. The Norconex HTTP Collector is an HTTP Crawler meant to give the greatest flexibility possible for developers and integrators. (more…)