Recorded Future Collective Insights for Carbon Black

To enable the Recorded Future integration for VMware Carbon Black, navigate to the Integration Center in the left-hand menu. 

Click the VMware Carbon Black tile. 

You will see additional details and resources for the integration. Click the blue Set up button.

Note: You must be an administrator to see the Set up button. 

Enter the requested information in the setup modal that displays.

• Connector Name: Collective Insights Connector For VMware Carbon Black
• VMware Carbon Black Authentication 
  • Server URL: Carbon Black Server URL
  • API ID
  • API Secret Key
  • Organization Key
• Detection Parameters 
  • Alert Determination:
  • Connector Update Frequency: The Update Frequency refers to the duration of time that Recorded Future will pause between updates. This can be set to hours, minutes, or days. Note that Recorded Future will poll for updates based on this frequency, but for all new events since the last time queried. The default (suggested) frequency is every 30 minutes.
• Initial Import
  • Detections Created Last: The duration of historical information that Recorded Future will pull based on the initial setup. The default range is 1 day; ranges longer than the previous 24 hours may cause delays in the setup process

Click Activate.

Steps to deploy Scripted Solution (Advanced)

Attached to this article is a script (CB-Collective_Insights.py) that maps Carbon Black detections to Recorded Future's collective insights data model.  The script will make API requests to build JSON objects for each IOC from Carbon Black with supporting evidence to be sent to the Recorded Future Collective Insights API.  You will need the following to run this script:

• Carbon Black Cloud
  • Need the Organization ID/Organization Key
  • API ID/API Secret
  • NOTE: Carbon Black EDR (on-prem, formerly known as CB Response) is not supported
• a local machine with Python v3.8.18 or greater installed with internet access
  • Server/Workstation that can be set up to run the script on a schedule
  • Internet access to Recorded Future API and Carbon Black Cloud API
  • Recommend whitelisting api.recordedfuture.com
• A valid Recorded Future collective insights API Token (email support@recordedfuture.com to request this)

Installation

Once package has been provided and downloaded to the machine where installation will be ran the following steps can be taken to configure and run the script for the first time.

1. OPTIONAL: Create environment variables in .bashrc file
   • # CB Integration
   • export RF_API_KEY=<Recorded Future API KEY>
   • export CB_ORG_ID=<Carbon Black Org ID>
   • export CB_HOSTNAME=<Carbon Black Hostname>
   • export CB_SECRET=<Carbon Black API Secret>
   • export CB_API_ID=<Carbon Black API ID>
2. Setup a new virtual environment to install dependencies and run the script from:
   • python3 venv venv
3. Activate new virtual environment:
   • source venv/bin/activate
4. Install dependencies from requirements.txt (attached to this article):
   • pip3 install -r requirements.txt
5. Run python script to confirm successful:
   • python3 CB-Collective_Insights.py -h
   • usage: CB-Collective_Insights.py [-h] [-k RF_API_KEY] [-co CB_ORG_ID] [-ch CB_HOSTNAME] [-cs CB_SECRET] [-cid CB_API_ID] [--debug] [-l {DEBUG,INFO,WARNING,ERROR,CRITICAL}] [-ef EXCLUSION_FILE]
6. Setup script to run on schedule daily to ingest events and send to collective insights:
   • Example cron schedule to run at 00:15 every night:
   • 15 0 * * * <FILE_DIR>/venv/bin/python3 <FILE_DIR>/CB-Collective_Insights.py -k RF_API_KEY -co CB_ORG_ID -ch CB_HOSTNAME -cs CB_SECRET -cid CB_ID

Troubleshooting

The below section is for providing assistance with troubleshooting when having issues running the script.

• Script is failing due to modules not installed.
  • Try confirming either the requirements.txt was installed properly with pip3
  • Confirm that the virtual environment where the python packages were installed is activated
• Not authorized to submit to Recorded Future Collective Insights API
  • Confirm that the Recorded Future API token has the correct Collective Insights API permissions activated
• Not authorized to collect events from Carbon Black Cloud
  • Confirm that the correct Access Levels & Permissions are enabled for the API Key (for ref, see Alerts API - Carbon Black Developer Network)
  • Confirm that the API ID & API Secret(Key) are in the correct parameter locations
• Why am I ingesting ~100 different hashes but only ~80 are being sent to Collective Insights?
  • You are most likely using the -ef option to exclude hashes from your list to be sent back.
  • Remove this option or remove the desired hashes from the Exclusion File
• Why am I ingesting 800 events from Carbon Black but the log says I’ve only submitted 200 hashes?
  • Collective Insights has filtering for unique indicators when submitting via the api.

FAQ

1. How to generate API ID and Secret in Carbon Black for Collective Insights?

Step 1: Creating a custom access level 

When logged into Carbon Black Cloud console, navigate to Settings -> API Access -> Access Levels. From there, you can view existing access levels. Click on "Add Access Level". Give an appropriate name and description and  select the required permissions. After you click Save, the Access Level gets created.

Step 2: Creating API Key

After saving the Access Level, navigate to Settings -> API Access -> API Keys. From there, you can view existing API keys. Click on "Add API Key". Give an appropriate name and description. Select Access Level Type as "Custom" and Custom Access Level as the Name of Access Level created in Step 1. After you click Save, the API Key gets created and you will see the API ID and Secret Key generated. Please copy and save the API ID and Secret to refer them.

2. What permissions are required as part of Access Level for Collective Insights?

The API key should have READ access to Tags, ThreatMetadata, Notes, ThreatHunt and General Information under Alerts category of Access Levels.

3. Why don't i see any alerts or mismatch in the number of alerts between Carbon Black and Detection Trends dashboard ?

Not all the alerts from Carbon Black make it to Collective Insights. The following criteria has to match in order for an alert in Carbon Black to be sent to Collective Insights:

a. There should be a non-null value for SHA256 hash (process_sha256) under an alert in Carbon Black.

b. The determination of the alert should be either TRUE_POSITIVE or FALSE_POSITIVE. (Note: Any alerts with determination value being None will not be ingested into Collective Insights)

4. Why don't i see any MITRE codes related to my Carbon Black alerts in the Detection Activity dashboard ?

Currently, there are no MITRE ATT&CK codes available from the Carbon Black alerts due to which they are not reflected under Detection Activity dashboard.

5. What are the IP addresses which needs to be whitelisted to allow communication from the hosted service to Carbon Black?

The traffic from the following IP addresses from AWS which are dedicated to Recorded Future needs to be whitelisted to allow communication from the hosted service:

• 52.204.27.85
• 54.198.55.229
• 54.156.251.192
• 34.235.48.77

6. How to setup connector under a suborg in case of a Multiorg environment?

In order to setup a connector under a specific suborg, please switch as Enterprise admin to that particular suborg and create a connector following the same process mentioned here.