Norconex Elasticsearch Committer
Configuration

When used with a Norconex Collector, you can use the following XML to configure Elasticsearch as the <committer> section of your Norconex Collector configuration:

<committer class="com.norconex.committer.elasticsearch.ElasticsearchCommitter">
    <nodes>...</nodes>
    <indexName>...</indexName>
    <typeName>...</typeName>
    <ignoreResponseErrors>[false|true]</ignoreResponseErrors>
    <discoverNodes>[false|true]</discoverNodes>
    <dotReplacement>...</dotReplacement>
    <jsonFieldsPattern>...</jsonFieldsPattern>
    <connectionTimeout>(milliseconds)</connectionTimeout>
    <socketTimeout>(milliseconds)</socketTimeout>
    <maxRetryTimeout>(milliseconds)</maxRetryTimeout>
    <fixBadIds>[false|true]</fixBadIds>
    <username>...</username>
    <password>...</password>
    <passwordKey>...</passwordKey>
    <passwordKeySource>[key|file|environment|property]</passwordKeySource>
    <sourceReferenceField keep="[false|true]">...</sourceReferenceField>
    <sourceContentField keep="[false|true]">...</sourceContentField>
    <targetContentField>...</targetContentField>
    <queueDir>...</queueDir>
    <queueSize>...</queueSize>
    <commitBatchSize>...</commitBatchSize>
    <maxRetries>...</maxRetries>
    <maxRetryWait>...</maxRetryWait>
</committer>

Tag descriptions:

Tag Description
nodes Comma delimited list of host URLs to connect to join the cluster. Default is http://localhost:9200.
indexName Index name to use when committing documents to Elasticsearch.
typeName Type name to use when committing documents to Elasticsearch.
ignoreResponseErrors Optionally ignore errors in Elasticsearch response. When ignored, errors are logged instead of throwning an exception. Default is false.
discoverNodes Optionally enable automatic discovery of cluster nodes beyond the configured ones. Default is false.
dotReplacement Optionally replace dots in field names with any value. Default is null (does not replace dots).
jsonFieldsPattern Optional regular expression to identify fields containing JSON objects instead of regular strings.
connectionTimeout Elasticsearch connection timeout (default 1 second).
socketTimeout Elasticsearch socket timeout (default 30 seconds).
maxRetryTimeout Maximum amount of time to wait before retrying a failing Elasticsearch host (default 30 seconds).
fixBadIds Flag to fix ids not matching Elasticsearch ID limitations.
username Basic authentication user name.
password Basic authentication password.
passwordKey Reference to password key (or actual key) for encrypted passwords. See javadoc for encryption instructions.
passwordKeySource Source of password key for encrypted passwords. See javadoc for encryption instructions.
sourceReferenceField Name of source field that will be mapped to the Elasticsearch id field. Default is the document reference the Committer stores as document.reference. The metadata source field is deleted, unless keep is set to true.
sourceContentField Source field name for a document content/body. Default is not a field, but rather the document body content. Once re-mapped, the metadata source field is deleted, unless keep is set to true.
targetContentField Target field name for a document content/body. Default is: content.
queueDir Optional path where to queue files before sending them to Elasticsearch. Default is: ./committer-queue.
queueSize Optional maximum queue size before sending document to Elasticsearch. Default is: 1000.
commitBatchSize Optional maximum of documents to send to Elasticsearch at once. Default is: 100.
maxRetries Maximum retries upon commit failures. Default is 0 (no retry).
maxRetryWait Maximum delay (millisecond) between retries. Default is 0 (no delay).

Copyright © 2013-2020 Norconex Inc.