Recorded Future Collective Insights for Elastic

Introduction

Recorded Future's Collective Insights is a new type of Recorded Future analytics, providing clients with a complete view of what threats matter to an organization. Collective Insights enables Recorded Future’s clients to analyze detected incidents to create an intelligence resource for use in two ways:

• Collective Insights provides clients with a comprehensive view of detections across their infrastructure and controls.
• Anonymized data can be used to create visualizations and analytics comparing threat vectors and the entire threat landscape for your enterprise anonymously compared to specific industries and geographies.

For more information on Recorded Future’s Collective Insights, please see the Getting Started with Collective Insights page for more information.

Getting Started

This is a script based solution which will ingest documents from the Elasticsearch alerts index generated from Indicator Detection Rules. The script runs an intial search based on the 'lookback' -L command line argument specifying the time in hours to search the Elasticsearch alerts index (or 1 hour by default). Once the script executes for the first time a timestamp is saved to file that is then referenced in future script executions depending on the alerts returned. If alerts are returned then the timestamp saved is the timestamp of the latest Elasticsearch alert. Otherwise the timestamp saved will be the script execution time. Following executions add one millisecond to this reference timestamp when searching the next round of documents to send to Collective Insights.

• To overwrite the saved timestamp with the command line argument lookback you must first delete the file at config/latest_timestamp.txt and then run the python script.

By Default the Elasticsearch instance major version number determines which Alert index is queried by the script:

• v8.0.0 or later: .alerts-security.alerts-*
• Before v8.0.0: .siem-signals-*

Users can override the default alerts index if needed within config/settings.ini.

Fields Collected from each incident by Recorded Future:

• _id - Unique Elasticsearch document ID
• @timestamp - Alert Timestamp in ECS format. Can be overwritten for custom field.
• hostname - Elasticsearch document hostname in ECS format. Can be overwritten for custom field.
• threat.enrichments.matched.atomic - Indicator value written back to Recorded Future. Recorded Future only collects fields matching the types in ip, domain, hash, and url.
• kibana.alert.rule.name - Name of the Indicator Match Rule that triggered the alert.
• threat.tactic.id - Specific MITRE ATT&CK techniques and tactics used in a particular security incident.

Prerequisites
The following items needs to be installed/gathered before the setup of the integration script:

• Python v3.9 or greater must be installed
• Client must have access to the Elasticsearch API
• Client must be able to provide an environment to run the script locally with access to the internet.
  • Server/Workstation for script to run on schedule
  • Internet Access to Recorded Future API & Elasticsearch API
  • Recommend whitelisting api.recordedfuture.com
• Client must have Recorded Future API Token with access to Collective Insights API
  • This can be provided by your IntServ Consultant or PS Engineer
• Client must have the following from Elasticsearch:
  • Elasticsearch Host: Cloud ID, Hostname(s)
  • Authentication Method, one of:
    • API Token ID & Token Secret (API Token Authentication)
    • Username & Password (Basic Authentication)
    • CA, Client Cert, and Client Key files (Certificate Authentication)
  • read index privilege to read data and search data in the alerts index

Installation
The Elasticsearch Collective Insights python script is available as an attachment to this support article.

Once package has been provided and downloaded to the machine where installation will be ran the following steps can be taken to configure and run the script for the first time.

Before running the script, inspect config/settings.ini and adjust the parameters as needed. You may opt to store client credentials/secrets as environment variables instead of passing them as parameters to the script. If you choose to do so, you will need to set the following environment variables:

• Collective Insights API: RF_API_KEY
• Elasticsearch User Authentication:
  • ELASTIC_USER | ELASTIC_PASSWORD

Set-up Instructions

• Setup a new virtual environment to install dependencies and run the script from: python3 -m venv venv
• Activate new virtual environment: source venv/bin/activate
• Install dependencies from requirements.txt: pip3 install -r requirements.txt
• Run python script to confirm successful: python3 run_elk_ci.py -h
• Setup script to run on schedule daily to ingest events and send to collective insights:
  • Example cron schedule to run at minute 0 every hour: 0 * * * * <FILE_DIR>/venv/bin/python3 <FILE_DIR>/run_elk_ci.py -k RF_API_KEY

Troubleshooting
The below section is for providing assistance with troubleshooting when having issues running the script.

• Script is failing due to modules not installed.
  • Confirm the requirements.txt was installed properly with pip3
  • Confirm the virtual environment where the python packages are installed is activated
• Not authorized to submit to Recorded Future Collective Insights API
  • Confirm that the Recorded Future API token has the correct Collective Insights API permissions activated
• Why am I ingesting more alerts from Elasticsearch than are being submitted to Collective Insights?
  • Collective Insights has filtering for unique indicators when submitting via the api