## **Overview**

Recorded Future Collective Insights is a new type of analytic, providing clients a complete view of what threats matters to an organization. Collective Insights aggregates detections across client integrations, confirming indicators related to malicious behavior or high-risk activity. This helps TI and SecOps users better prevent and protect client networks by prioritizing their actions based on which detections and TTPs are most common across their networks. This article will walk through how to setup Collective Insights for Carbon Black using the [Recorded Future collective insights API](https://support.recordedfuture.com/hc/en-us/articles/15847735339923-Collective-Insights-API).

The following script will ingest events from Carbon Black Cloud where the field `blocked_sha256` or `blocked_md5` or `ioc_hit` is present for the last 24 hours. These events can then be filtered (if desired) against a known set of legitimate process hashes to only include a set of malicious sha256 values that will then be sent off to Recorded Future’s Collective Insights API.

Fields Collected from each incident by Recorded Future:

- `process_sha256` - sha256 hash of the process that was detected
- `id` - event id
- `detection_timestamp` - timestamp of the detected event

## **Prerequisites**

The following items be installed/gathered before the setup of the integration script

1. Python v3.8.18 or greater must be installed
2. Client must have Carbon Black Cloud
   1. Currently not supported for Carbon Black EDR (on-prem) (formerly CB Response)
3. Client must be able to provide an environment to run the script locally with access to the internet.
   1. Server/Workstation for script to run on schedule
   2. Internet Access to Recorded Future API & Carbon Black Cloud API
   3. Recommend whitelisting [api.recordedfuture.com](http://api.recordedfuture.com/)
4. Client must have Recorded Future API Token with access to Collective Insights API
   1. This can be provided by your IntServ Consultant or PS Engineer
5. Client must have the following from Carbon Black Cloud:

   1. Organization ID/Organization Key
   2. API ID/API Secret

      1. The above needs CREATE+READ permissions on the Search categories
      2. An access level can be created with these permissions and assigned to the API Key

      3. [Alerts API - Carbon Black Developer Network](https://developer.carbonblack.com/reference/carbon-black-cloud/platform/latest/alerts-api#authentication)

         ![https://developer.carbonblack.com/images/favicons/favicon-16x16.png](https://developer.carbonblack.com/images/favicons/favicon-16x16.png)

# **Installation**

The Carbon Black Collective Insights python script can be provided via the assigned Professional Services engineer.

Once package has been provided and downloaded to the machine where installation will be ran the following steps can be taken to configure and run the script for the first time.

1. OPTIONAL: Create environment vairables in `.bashrc`

```
# CB Integration
export RF_API_KEY=<Recorded Future API KEY>
export CB_ORG_ID=<Carbon Black Org ID>
export CB_HOSTNAME=<Carbon Black Hostname>
export CB_SECRET=<Carbon Black API Secret>
export CB_API_ID=<Carbon Black API ID>
```

2. Setup a new virtual environment to install dependencies and run the script from:`python3 venv venv`
3. Activate new virtual environment:`source venv/bin/activate`
4. Install dependencies from requirements.txt: `pip3 install -r requirements.txt`
5. Run python script to confirm successful: `python3 CB-Collective_Insights.py -h`
   ```
   usage: CB-Collective_Insights.py [-h] [-k RF_API_KEY] [-co CB_ORG_ID] [-ch CB_HOSTNAME] [-cs CB_SECRET] [-cid CB_API_ID] [--debug] [-l {DEBUG,INFO,WARNING,ERROR,CRITICAL}] [-ef EXCLUSION_FILE]
   ```
6. Setup script to run on schedule daily to ingest events and send to collective insights:
   1. Example cron schedule to run at 00:15 every night:
   ```
   15 0 * * * <FILE_DIR>/venv/bin/python3 <FILE_DIR>/CB-Collective_Insights.py -k RF_API_KEY -co CB_ORG_ID -ch CB_HOSTNAME -cs CB_SECRET -cid CB_ID
   ```

# **Troubleshooting**

The below section is for providing assistance with troubleshooting when having issues running the script.

- Script is failing due to modules not installed.
  - Try confirming either the requirements.txt was installed properly with pip3
  - Confirm that the virtual environment where the python packages were installed is activated
- Not authorized to submit to Recorded Future Collective Insights API
  - Confirm that the Recorded Future API token has the correct Collective Insights API permissions activated
- Not authorized to collect events from Carbon Black Cloud
  - Confirm that the correct Access Levels & Permissions are enabled for the API Key
    - [https://developer.carbonblack.com/reference/carbon-black-cloud/platform/latest/alerts-api#authentication](https://developer.carbonblack.com/reference/carbon-black-cloud/platform/latest/alerts-api#authentication)
  - Confirm that the API ID & API Secret(Key) are in the correct parameter locations
- Why am I ingesting ~100 different hashes but only ~80 are being sent to Collective Insights?
  - You are most likely using the `ef` option to exclude hashes from your list to be sent back.
  - Remove this option or remove the desired hashes from the Exclusion File
- Why am I ingesting 800 events from Carbon Black but the log says I’ve only submitted 200 hashes?
  - Collective Insights has filtering for unique indicators when submitting via the api.