Introduction

Recorded Future collective insights is a new type of analytic, providing clients a complete view of what threats matters to an organization. Collective insights can aggregate detections across all client integrations to show trends across all detections to help TI and SecOps users better prevent and protect client networks by prioritizing their actions based on which detections and TTPs are most common across their networks. This article will walk through how to setup collective insights for Carbon Black using the Recorded Future collective insights API

Prerequisites for Collective Insights with Carbon Black

Attached to this article is a script (CB-Collective_Insights.py) that maps Carbon Black detections to Recorded Future's collective insights data model.  The script will make API requests to build JSON objects for each IOC from Carbon Black with supporting evidence to be sent to the Recorded Future Collective Insights API.  You will need the following to run this script:

• Carbon Black Cloud
  • Need the Organization ID/Organization Key
  • API ID/API Secret
    • these need to have CREATE+READ permissions on the Search Categories:
    • An access level can be created with these permissions and assigned to the API Key
    • More info about Carbon Black API access available here: https://developer.carbonblack.com/reference/carbon-black-cloud/platform/latest/alerts-api#authentication
  • NOTE: Carbon Black EDR (on-prem, formerly known as CB Response) is not supported
• a local machine with Python v3.8.18 or greater installed with internet access
  • Server/Workstation that can be set up to run the script on a schedule
  • Internet access to Recorded Future API and Carbon Black Cloud API
  • Recommend whitelisting api.recordedfuture.com
• A valid Recorded Future collective insights API Token (email support@recordedfuture.com to request this)

Installation

Once package has been provided and downloaded to the machine where installation will be ran the following steps can be taken to configure and run the script for the first time.

1. OPTIONAL: Create environment variables in .bashrc file
   • # CB Integration
   • export RF_API_KEY=<Recorded Future API KEY>
   • export CB_ORG_ID=<Carbon Black Org ID>
   • export CB_HOSTNAME=<Carbon Black Hostname>
   • export CB_SECRET=<Carbon Black API Secret>
   • export CB_API_ID=<Carbon Black API ID>
2. Setup a new virtual environment to install dependencies and run the script from:
   • python3 venv venv
3. Activate new virtual environment:
   • source venv/bin/activate
4. Install dependencies from requirements.txt (attached to this article):
   • pip3 install -r requirements.txt
5. Run python script to confirm successful:
   • python3 CB-Collective_Insights.py -h
   • usage: CB-Collective_Insights.py [-h] [-k RF_API_KEY] [-co CB_ORG_ID] [-ch CB_HOSTNAME] [-cs CB_SECRET] [-cid CB_API_ID] [--debug] [-l {DEBUG,INFO,WARNING,ERROR,CRITICAL}] [-ef EXCLUSION_FILE]
6. Setup script to run on schedule daily to ingest events and send to collective insights:
   • Example cron schedule to run at 00:15 every night:
   • 15 0 * * * <FILE_DIR>/venv/bin/python3 <FILE_DIR>/CB-Collective_Insights.py -k RF_API_KEY -co CB_ORG_ID -ch CB_HOSTNAME -cs CB_SECRET -cid CB_ID

Results

Once the script has run at least once, collective insights results can be viewed in the Recorded Future UI in the SecOps Dashboard. The SecOps Dashboard panels in the UI will be populated displaying Detection Trends .

Click into the detection trends and detection activity panels to see which detections are driving each analytic to gain insights to common attributes between all detections coming from Carbon Black (note, screenshot below shows detections from SentinelOne, but otherwise should look similar for Carbon Black). 

Troubleshooting

The below section is for providing assistance with troubleshooting when having issues running the script.

• Script is failing due to modules not installed.
  • Try confirming either the requirements.txt was installed properly with pip3
  • Confirm that the virtual environment where the python packages were installed is activated
• Not authorized to submit to Recorded Future Collective Insights API
  • Confirm that the Recorded Future API token has the correct Collective Insights API permissions activated
• Not authorized to collect events from Carbon Black Cloud
  • Confirm that the correct Access Levels & Permissions are enabled for the API Key (for ref, see Alerts API - Carbon Black Developer Network)
  • Confirm that the API ID & API Secret(Key) are in the correct parameter locations
• Why am I ingesting ~100 different hashes but only ~80 are being sent to Collective Insights?
  • You are most likely using the -ef option to exclude hashes from your list to be sent back.
  • Remove this option or remove the desired hashes from the Exclusion File
• Why am I ingesting 800 events from Carbon Black but the log says I’ve only submitted 200 hashes?
  • Collective Insights has filtering for unique indicators when submitting via the api.

Related Articles

Getting Started with Collective Insights

Collective Insights API

SecOps Dashboard Overview