Recorded Future Collective Insights for Logrhythm

Introduction

Recorded Future's Collective Insights is a new type of Recorded Future analytics, providing clients with a complete view of what threats matter to an organization. Collective Insights enables Recorded Future’s clients to analyze detected incidents to create an intelligence resource for use in two ways:

• Collective Insights provides clients with a comprehensive view of detections across their infrastructure and controls.
• Anonymized data can be used to create visualizations and analytics comparing threat vectors and the entire threat landscape for your enterprise anonymously compared to specific industries and geographies.

For more information on Recorded Future’s Collective Insights, please see the Getting Started with Collective Insights page for more information.

Overview

This is a script based solution that ingests Alarms from LogRhythm created within the previous 24 hours. This timeframe can be adjusted to backfill any additional collective insights. The script also allows filtering of Alarms based on Risk Based Priority. Additionally, users can choose to send only Recorded Future generated Alarms or exclude Alarms by providing an exclusion list of rule names.

Fields collected from Alarms by Recorded Future are defined below:

• alarmId
• alarmRuleName
• dateInserted
• impactedHost
• impactedIP
• originHost
• originIP
• domain
• url
• cve

Getting Started

The following prerequisites are required to run the integration script

1. Python >=3.8, <=3.10
2. LogRhythm >=7.4.10
3. A server that has the following requirements (i.e., Platform Manager):
   • Access to the LogRhythm Alarms API
   • Access to api.recordedfuture.com
4. Recorded Future API key with Collective Insights enabled
5. LogRhythm Third Party Application API Key
   • See Register Third-Party Applications to Use the API for instructions on generating an API key

Installation:

1. Create environment variables for both API keys and the API URL
   • Open Run (Windows + R) and enter sysdm.cpl
   • Go to the Advanced tab → Environment Variables
   • Add three new User Environment Variables:
     • RF_API_KEY
     • LOGRHYTHM_API_KEY
     • LOGRHYTHM_API_URL
2. Unzip the logrhythm_collective_insights.zip package. These files should be present:
   • logrhythm_collective_insights.py
   • psengine-<version>.whl
   • requirements.txt
3. Open a command prompt and navigate to the LogRhythm Collective Insights folder
   • cd C:\<directory-path>\logrhythm_collective_insights
4. Install dependencies from the requirements.txt file
   
   • C:\<python-path>\Scripts\pip.exe install -r requirements.txt
5. Run the script manually with the -d flag (debug mode, detections not submitted) and verify that it finishes successfully:
   
   • C:\<python-path>\python.exe logrhythm_collective_insights.py -d
6. Set the script to run on a schedule. Open Task Scheduler → Create Task:
   • General
     • Name: LogRhythm Collective Insights
     • Description: Send Alarm data to Recorded Future's Collective Insights API
     • Security options: Run whether user is logged on or not
   • Triggers
     • Begin the task: On a schedule
     • Settings: Daily
   • Actions:
     • Action: Start a program
       • Program/script: C:\<python-path>\python.exe
       • Arguments: C:\<directory-path>\logrhythm_collective_insights\logrhythm_collective_insights.py
   • Conditions: Default, or your preference
   • Settings: Default, or your preference
   • Run the task manually, and confirm that it finishes successfully. Detections should be available in Collective Insights shortly after.

(Optional) Backfill any additional Alarms that you want to submit to Collective Insights by running the script manually and including the -lb flag. This flag takes in a positive integer representing the number of days back to search for Alarms.

FAQ

1. Why is the script is failing due to “modules not installed”?

Confirm that all of the required modules are installed from the requirements.txt file. This can be done by running pip freeze in a command prompt.

2. Why am I getting 403s when submitting to the Recorded Future Collective Insights API?

Confirm with your account team that your Recorded Future API token has the correct permissions needed for the Collective Insights API.

3. Where do I find my LogRhythm API URL?

The default LogRhythm API URL is available at http://localhost:8505/ on the Platform Manager for on-premise deployments. For cloud deployments, reach out to LogRhythm for your API URL.

4. How do I submit only Recorded Future generated Alarms?

This can be done by adding the -rf flag to the script. This will look for any Alarms that have “Recorded Future” in the Alarm Rule Name, and only include those Alarms.

5. How do I submit Alarms based on their Risk Based Priority?

This can be done by adding the -rbp flag to the script. This flag takes in an integer from [0, 100] and will use that to filter out any Alarms that have a lower Risk Based Priority score.

6. Are there other ways to exclude Alarms from being sent?

Alarms can be excluded based on their name. This can be done by creating an exclusions text file that has the Rule Names to be excluded. This file should be a text file with each Alarm Rule Name on a single line. The script will then filter out any Alarms matching those names. When running the script, add the -ef flag and the name of the file as the flag argument.

7. Why are there multiple detections with the same Alarm ID and Alarm Name?

There can be multiple events that can contribute to an Alarm. The script will look at each event in an Alarm and extract any possible IOCs from the event fields. These events are all sent with the same incident ID (Alarm ID) and will have the same description (Alarm ID: Alarm Rule Name).

8. Why are there only 20 indicator submissions when 40 Alarms triggered?

Collective Insights has filtering for unique indicators when submitting via the API. Additionally, LogRhythm Alarms are only collected if they have a valid IP, domain, or URL value in the following Alarm fields. Note that RFC-1918 IP Addresses are excluded from the submission.

• impactedHost
• impactedIP
• originHost
• originIP
• domain
• url
• cve

Finally, the LogRhythm Alarms Search API is also only capable of returning 100 Alarms per request. There is no way to paginate through the results if there are more than 100 Alarms. Due to this restriction, if more than 100 Alarms have triggered within one hour, not all Alarms will be sent to Collective Insights.

9. What should I do if I’m still having issues with the script?

The script keeps a rotating log of all runs in the logrhythm_collective_insights\logs directory. By default, the logging level is set to INFO. Run the script again and change the logging level to DEBUG by adding -l DEBUG. If you are unable to resolve the issue, please contact support@recordedfuture.com and send the latest log file.

Happy Hunting !!