Recorded Future Collective Insights for Elastic

Introduction

Recorded Future's Collective Insights is a new type of Recorded Future analytics that provides a complete view of the threats that matter to your organization. Collective Insights lets you analyze detected incidents to create an intelligence resource in two ways:

• Collective Insights provides a comprehensive view of detections across their infrastructure and controls.
• Anonymized data can be used to create visualizations and analytics comparing threat vectors and the entire threat landscape for your enterprise anonymously compared to specific industries and geographies.

For more information on Recorded Future’s Collective Insights, see Getting Started with Collective Insights.

Getting Started

This is a script-based solution that ingests documents from the Elasticsearch alerts index generated from Indicator Detection Rules. The script runs an initial search based on the 'lookback' -L command line argument, specifying the time in hours to search the Elasticsearch alerts index (or 1 hour by default). Once the script executes for the first time, a timestamp is saved to file that is then referenced in future script executions, depending on the alerts returned. If alerts are returned then the timestamp saved is the timestamp of the latest Elasticsearch alert. Otherwise, the timestamp saved is the script execution time. Following executions add one millisecond to this reference timestamp when searching the next round of documents to send to Collective Insights.

• To overwrite the saved timestamp with the command line argument lookback you must first delete the file at config/latest_timestamp.txt and then run the python script.

By default the Elasticsearch instance major version number determines which Alert index is queried by the script:

• v8.0.0 or later: .alerts-security.alerts-*
• Before v8.0.0: .siem-signals-*

You can override the default alerts index if needed within config/settings.ini.

Recorded Future collects the following fields from each incident:

• _id - Unique Elasticsearch document ID
• @timestamp - Alert Timestamp in ECS format. Can be overwritten for custom field.
• hostname - Elasticsearch document hostname in ECS format. Can be overwritten for custom field.
• threat.enrichments.matched.atomic - Indicator value written back to Recorded Future. Recorded Future only collects fields matching the types in ip, domain, hash, and url.
• kibana.alert.rule.name - Name of the Indicator Match Rule that triggered the alert.
• threat.tactic.id - Specific MITRE ATT&CK techniques and tactics used in a particular security incident.

Prerequisites
The following items needs to be installed/gathered before setting up the integration script:

• Python v3.9 
• Access to the Elasticsearch API
• An environment to run the script locally with access to the internet
  • Server/Workstation for script to run on schedule
  • Internet Access to Recorded Future API & Elasticsearch API
  • Recommend whitelisting api.recordedfuture.com
• A Recorded Future API Token with access to Collective Insights API
  • This can be provided by your account team
• The following from Elasticsearch:
  • Elasticsearch Host: Cloud ID, Hostname(s)
  • Authentication Method (one of the following):
    • API Token ID & Token Secret (API Token Authentication)
    • Username & Password (Basic Authentication)
    • CA, Client Cert, and Client Key files (Certificate Authentication)
  • read index privilege to read data and search data in the alerts index

Installation
The Elasticsearch Collective Insights python script is available as an attachment to this support article.

Once the package has been provided and downloaded to the machine where the installation will run, use the following instructions to configure and run the script for the first time.

Before running the script, inspect config/settings.ini and adjust the parameters as needed. You may opt to store client credentials/secrets as environment variables instead of passing them as parameters to the script. If you choose to do so, you will need to set the following environment variables:

• Collective Insights API: RF_API_KEY
• Elasticsearch User Authentication:
  • ELASTIC_USER | ELASTIC_PASSWORD

Setup Instructions

• Set up a new virtual environment to install dependencies and run the script from: python3 -m venv venv
• Activate new virtual environment: source venv/bin/activate
• Install dependencies from requirements.txt: pip3 install -r requirements.txt
• Run python script to confirm successful: python3 run_elk_ci.py -h
• Setup script to run on schedule daily to ingest events and send to Collective Insights:
  • Example cron schedule to run at minute 0 every hour: 0 * * * * <FILE_DIR>/venv/bin/python3 <FILE_DIR>/run_elk_ci.py -k RF_API_KEY

Troubleshooting

• Script is failing due to modules not installed
  • Confirm the requirements.txt was installed properly with pip3
  • Confirm the virtual environment where the python packages are installed is activated
• Not authorized to submit to Recorded Future Collective Insights API
  • Confirm that the Recorded Future API token has the correct Collective Insights API permissions activated
• Why am I ingesting more alerts from Elasticsearch than are being submitted to Collective Insights?
  • Collective Insights has filtering for unique indicators when submitting via the API