Introduction
Recorded Future's Collective Insights is a new type of Recorded Future analytics, providing clients with a complete view of what threats matter to an organization. Collective Insights enables Recorded Future’s clients to analyze detected incidents to create an intelligence resource for use in two ways:
- Collective Insights provides clients with a comprehensive view of detections across their infrastructure and controls.
- Anonymized data can be used to create visualizations and analytics comparing threat vectors and the entire threat landscape for your enterprise anonymously compared to specific industries and geographies.
For more information on Recorded Future’s Collective Insights, please see the Getting Started with Collective Insights page for more information.
Overview
Recorded Future Collective Insights connector for Proofpoint ingests threats related to issues from Proofpoint via SIEM API into Collective Insights. Specifically, the following threats are collected from Proofpoint:
1. Blocked or permitted clicks to threats recognized by URL Defense
2. Blocked or delivered messages that contain threats recognized by URL Defense or Attachment Defense
The connector collects URL threats from click issues, and domain, URL, and hash threats from message issues.
Message threats are scored for the following 4 threat categories:
- Phish score
- Malware score
- Impostor score
- Spam score
Scores are from 0 to 100. Higher scores indicate higher certainty. Users have the ability to set score filters. Score filters are enabled in groups: a message must meet all score thresholds from 1 group to be collected. See Message Filter Logic section for more details.
The following data points are ingested from Proofpoint into Collective Insights:
Click Issues:
- threatTime
- url
- threatID
- classification
Message Issues:
- threatsInfoMap/threatTime
- threatsInfoMap/threatType
- threatsInfoMap/threat
- threatsInfoMap/threatID
- threatsInfoMap/classification
Getting Started
To enable the Recorded Future integration for Proofpoint, navigate to the Integration Center in the left-hand menu.
Click the Proofpoint tile.
You will see additional details and resources for the integration. Click the blue Set up button.
Note: You must be an administrator to see the Set up button.
Enter the requested information in the modal that displays.
- Connector Name: Collective Insights Connector For Proofpoint
-
Proofpoint Authentication
- Principal: Parameter for SIEM API authentication, located in Proofpoint, under 'Threat Insight Dashboard > settings > Connected Apps'
- Secret: Parameter for SIEM API authentication, located in Proofpoint, under 'Threat Insight Dashboard > settings > Connected Apps'
-
Detection Parameters
- Fetch Blocked Click Events: Fetch events for clicks on blocked malicious URLs
- Fetch Permitted Click Events: Fetch events for clicks on malicious URLs not blocked
- Fetch Blocked Messages: Fetch events for emails with blocked threats
- Fetch Delivered Messages: Fetch events for emails with threats not blocked
-
The following filters can be selected only if one of either "Fetch Blocked Messages" or "Fetched Delivered Messages" is enabled:
- Fetch Malware Phish Messages and Events: Fetch Messages and Events with Phishing score 100 and Malware score at least 1. Score Ranges: 1-100
- Fetch Malware Spam Messages and Events: Fetch Messages and Events with Spam score 100 and Malware score at least 1. Score Ranges: 1-100
- Fetch Spam Phishing Messages and Events: Fetch Messages and Events and Events with Phishing score 100 and Spam score at least 100. Score Ranges: 1-100
- Connector Update Frequency: The Update Frequency refers to the duration of time that Recorded Future will pause between updates. This can be set to hours, minutes, or days. Note that Recorded Future will poll for updates based on this frequency, but for all new events since the last time queried. The default (suggested) frequency is every 30 minutes.
-
Initial Import
- Detections Created Last: The duration of historical information that Recorded Future will pull based on the initial setup. The default range is 1 day; ranges longer than the previous 24 hours may cause delays in the setup process
Click Activate.
- By default we only collect messages matching one of the following filters from the configuration
- Fetch Malware Phish Messages and Events
- Fetch Malware Spam Messages and Events
- Fetch Spam Phishing Messages and Events
- We recommend enabling the three default filter threshold groups (“Fetch messages with positive malware and phish scores”, “Fetch messages with positive malware and spam scores”, and “Fetch messages with phish and spam scores of 100”)
- These three groups reduced collected threats from messages by about 90% on a test enterprise
- If enabling a custom filter threshold group, we recommend setting at least a malware threshold of 100
- In testing on a client enterprise, we saw that the 50th percentile of malware scores for messages classified as malware is 100 - meaning at least half of messages classified as malware had a score of 100.
- Malware: 50
- Phish: 0
- Spam: 50
- Impostor: 0
- Fetch messages with positive malware and phish scores
- Fetch messages with positive malware and spam scores
- The message does not meet all thresholds from “Fetch messages with positive malware and phish scores”
- Malware: 50 >= Malware: 1
- Phish: 0 < Phish: 1
- The message does meet all thresholds from “Fetch messages with positive malware and spam scores”
- Malware: 50 >= Malware: 1
- Spam: 50 >= Spam: 1
FAQ
1. Are there any filters applied to the threats in Proofpoint before sending to Collective Insights?
The connector filters messages from SIEM API in the following way:
- Disabled issue types are filtered
- Invalid and unsupported IOCs are filtered
- The following examples are considered invalid, as they are not accepted by Collective Insights API or it is not clear what indicator should be sent to Collective Insights API:
- URLs without a scheme
- URLs with backslashes
- URLs with spaces
- URLs or domains with “@” in the domain
- Email addresses
- The following examples are considered invalid, as they are not accepted by Collective Insights API or it is not clear what indicator should be sent to Collective Insights API:
- IOCs below threshold settings (only applicable for message issues)
2. How to get the Service Principal and Secret values from Proofpoint?
The service principal and secret are used to authenticate to the SIEM API. To generate TAP Service Credentials please follow the following steps.
- Log in to the TAP dashboard.
- Navigate to Settings > Connected Applications.
- Click Create New Credential.
- Name the new credential set and click Generate.
- Copy the Service Principal and Secret and save them for later use.
3. How to setup connector under a suborg in case of a Multiorg environment?
In order to setup a connector under a specific suborg, please switch as Enterprise admin to that particular suborg and create a connector following the same process mentioned under 'Getting Started'.
Known Limitations
- The SIEM API has a limit of 1800 requests per hour for requests the /v2/issues/all endpoint used by the connector.
- The connector makes requests for issues using 1-hour intervals (the largest allowed by the API). Even using the largest initial fetch of 168 hours of issues, the connector should not exceed the throttle limit set by the SIEM API.
- Malware from Proofpoint Threats API is collected when available and written to Collective Insights. However, malware names from Proofpoint which do not have an equivalent name in Recorded Future are ignored by Collective Insights API.
- Techniques from Proofpoint Threats API which are not T-Codes are ignored by the connector.
Happy Hunting !!