Elasticsearch

On this page

Elasticsearch is a Warehouse output connector. Distributed search and analytics engine.

IP Allowlists

You might need to add the Monad IP range of 34.210.32.104/32 to your Elasticsearch allowlist, so Monad can connect to it.

NOTE: This target is not backwards compatible with versions prior to elasticsearch 8.

Identify some Configurations of your Elasticsearch Instance

There are a number of configuration settings you will need to know about your elasticsearch instance.

  1. Scheme: http or https.
  2. Host: The hostname of your elasticsearch instance. If you are an admin of elastic cloud, you can obtain this value by navigating to https://cloud.elastic.co/home. Then click the Manage option for your deployment. From there, click the Copy endpoint link next to elasticsearch to obtain the full url. Be sure to trim off the scheme.
  3. Port: 9200 by default. 443 default for elastic cloud.
  4. SSL CA File: The path to SSL CA Certificate file. An optional parameter to be used with https. Details on how to obtain the certificate

You will also need one of the following types of authentication:

  • Username and Password: Consult your elasticsearch instance administrator for this value.
  • Bearer token
  • API Key, API Key ID
  • Encoded API Key: Go into your elasticsearch instance. Then via searchbar, navigate to Stack Management. From there go to API Key. There you can create an Encoded API Key.

Required Privileges:

  • The user or associated Personal Access Key must have certain privileges. If you have admin permissions, you can check this by going into your elasticsearch instance. Then via searchbar, navigate to Stack Management. From there go to Users. Your user’s roles must include the following privileges.
    • Cluster Privileges: Monitor
    • Index Privileges: create_index, create_doc, write

NOTE: To navigate into your elasticsearch instance in elastic cloud, first navigate to https://cloud.elastic.co/home. Click on your deployment name.

Additionally there are a few more optional fields which are reserved mainly for advanced usage. You can specify the name of your indices via the index format field. By default it is set to **, or the name of the mom tables. For more information on how to use Index Format, Index Schema Fields, or Metadata Fields, consult the settings section here

Setting up an Elasticsearch output connector in Monad

To set up an Elasticsearch connector, you will need to know the above mentioned configuration details. Once you have this information, you set up the connector:

  1. Log in to your Monad account, and click Add connector.
  2. Select the Elasticsearch connector.
  3. Optionally, change the default name for the connector. This name serves as a label for the connector in the Monad app, and you can change it later.
  4. Select a method of authentication and click Connection Settings.
  5. Fill in the required fields. Note that if you are using https for your scheme, you will need to supply an ssl ca cert file.
  6. Click Test Connection and standby while Monad to confirms the credentials are valid.
  7. In the Models to Export field, specify which data models you wish to export to your elasticsearch instance.
  8. Click Connect.

Monad then tests the connection to elasticsearch, and if successful, begins syncing data from your Monad account into your elasticsearch instance.

NOTE: The elasticsearch connector will export ALL specified data on each run. So if you run the connector multiple times you may end up with duplicate data. To avoid duplicate data within an index, you can store exports in different indices by modifying the index format field to add a prefix or suffix to an export name.

This page was last modified: 29 Nov 2023