S3

Amazon S3 is a Storage output connector. Export raw JSONL data to S3 buckets.

IP Allowlists

You might need to add the Monad IP range of 34.210.32.104/32 to your S3 allowlist, so Monad can connect to it.

Summary

After setting up your S3 connector, every time you synchronize an input connector, the unprocessed output data in JSONL format will be saved to your S3 buckets and compressed using gzip. The location of each file will be saved using a path that follows this format: {bucket}/{prefix}/{connector_type}/{connector_id}/{flow_run_id}/{stream_name}/{file.gz}.

It is important to note that there are different streams available, each representing an endpoint from which we gather data. While some connectors may have only one stream, others may have many. These streams represent related types of data within the data sets returned from a security product API.

Setup Instructions:

AWS Permissions

To enable the connector to write to your S3 bucket, you will first need to set up a trusted role in your AWS account. Here are the steps to follow:

  1. Go to the AWS console and navigate to IAM, then select Roles.
  2. Click on the “Create role” button located in the top-right corner of the page.
  3. For “Trusted entity type,” choose “AWS account,” then select “Another AWS account.” Input 271127603659 and click Next.
  4. To grant permissions, click “Create policy” in the top-right corner of the page. Here, you can either use the visual aid to grant S3: Put object permission to the bucket of your choice or use the JSON option and paste the following code. NOTE: This grants access to ALL buckets. To limit access, define the bucket in Resource by passing the ARN of the bucket you wish to use. Then click Next. 
    {
 "Version": "2012-10-17",
 "Statement": [
     {
         "Sid": "VisualEditor0",
         "Effect": "Allow",
         "Action": "s3:PutObject",
         "Resource": "*"
     }
 ]
}
  5. Add any necessary tags as defined by your organization, and click Next.
  6. Give the role a name and description, then click “Create policy.”
  7. Navigate back to the previous tab to continue creating the role. Click “Refresh” next to “Create policy” and search for the role you just created. Select it and click Next.
  8. In Step 1: Select trusted entities click “Edit” then copy and paste the following JSON 
    {
 "Version": "2012-10-17",
 "Statement": [
     {
         "Effect": "Allow",
         "Principal": {
             "AWS": "arn:aws:iam::271127603659:role/monad-app"
         },
         "Action": "sts:AssumeRole"
     }
 ]
}

    This allows Monad’s monad-app role to assume the role you’ve created, which will give access to write to the S3 bucket.

  9. Give the role a name, and then click “Create role.”
  10. Search for the role you just created under “Roles,” select it, and copy the ARN. It should look something like arn:aws:iam::{account-id}:role/{role-you-created}. You will need this when creating your S3 connector.

Monad Connector

  1. In the Monad UI, click Add connector and select S3.
  2. Input the bucket name you wish to save your files to. If you have not, create a bucket do so.
  3. Set a prefix for the files to be saved under. For example /connector-syncs/ this will save all the files in the bucket under the connector-syncs path.
  4. Paste the role ARN you copied from the permissions steps in role ARN.
  5. Click Connect.

Monad then tests the connection to S3, and if successful, begins syncing data from your Monad account into your S3 bucket.

This page was last modified: 9 Oct 2023