Recorded Future Collective Insights for Microsoft Defender XDR

Introduction

Recorded Future's Collective Insights is a new type of Recorded Future analytics, providing clients with a complete view of what threats matter to an organization. Collective Insights enables Recorded Future’s clients to analyze detected incidents to create an intelligence resource for use in two ways:

• Collective Insights provides clients with a comprehensive view of detections across their infrastructure and controls.
• Anonymized data can be used to create visualizations and analytics comparing threat vectors and the entire threat landscape for your enterprise anonymously compared to specific industries and geographies.

For more information on Recorded Future’s Collective Insights, please see the Getting Started with Collective Insights page for more information.

Recorded Future can collect information on incidents from Defender 365 into Collective Insights allowing you to track indicators and TTPs from incidents across platforms. Specifically, Recorded Future can collect from the following Defender Security tools:

• Microsoft Defender For Endpoint
• Microsoft Defender For Identity
• Microsoft Defender For CloudApps,
• Microsoft Defender For Office365
• Microsoft 365Defender
• AzureAd Identity Protection
• Microsoft App Governance
• Microsoft Defender For Cloud

The following fields are collected from each incident by Recorded Future: 

• createdDateTime - Time the incident was created
• title - Title of the Incident
• Id - unique ID of the incident
• serviceSource - Defender tool (from list above) the incident came from
• detectionSource - Detection technology or sensor that identified the notable component or activity
• mitreTechniques - TTPs associated with the incident
• IOC type
• IOC value

Getting Started

There are 3 different deployment modes for setting up Collective Insights for Microsoft 365 Defender:

Hosted service - In this mode, the connector for ingesting incidents from Microsoft 365 Defender is hosted at Recorded Future which pulls incidents from Microsoft 365 Defender.

Scripted solution - In this mode, the connector for ingesting incidents from Microsoft 365 Defender is deployed as a script at customer's environment which pushes incidents from Microsoft 365 Defender.

Logic app - In this mode, the logic app is deployed in Microsoft Azure which pushes incidents from Microsoft 365 Defender.

Install via the Integration Center (Preferred)

To enable the Recorded Future integration for Microsoft Defender XDR, navigate to the Integration Center in the left-hand menu. 

Click the Microsoft Defender XDR tile. 

You will see additional details and resources for the integration. Click the blue Set up button.

Note: You must be an administrator to see the Set up button. 

Enter the requested information in the modal that displays.

• Connector Name: Collective Insights Connector For Microsoft Defender XDR
• Microsoft Defender XDR Authentication 
  • Client ID: To look up 'Client ID', go to 'App Registration > App > Overview > Application (Client) ID'
  • Secret: To look up 'Secret', go to 'Microsoft Entra ID > App Registration > App > Overview > Certificates & Secrets > Secret'
  • Tenant ID: To look up 'Tenant ID', go to 'Microsoft Entra ID > Overview > Tenant ID'
• Detection Parameters
  • Connector Update Frequency: The Update Frequency refers to the duration of time that Recorded Future will pause between updates. This can be set to hours, minutes, or days. Note that Recorded Future will poll for updates based on this frequency, but for all new events since the last time queried. The default (suggested) frequency is every 30 minutes.
• Initial Import
  • Detections Created Last: The duration of historical information that Recorded Future will pull based on the initial setup. The default range is 1 day; ranges longer than the previous 24 hours may cause delays in the setup process

Click Activate.

Steps to deploy Scripted solution (Advanced)

Before you run this script, you must:

• Have python3.8 installed
• Have the python module requests installed (pip install requests)

Secrets can be passed to the script as environment variables or command line arguments.

Environment variables:

• RF_API_KEY: Recorded Future API Key
• MS_CLIENT_ID: Azure AD Client ID
• MS_CLIENT_SECRET: Azure AD Client Secret
• MS_TENANT: Azure Tenant ID

Alternatively, as command line arguments:

• -k: Recorded Future API Key
• -cid: Azure AD Client ID
• -cs: Azure AD Client Secret
• -t: Azure Tenant ID

The script also supports additional command line arguments to configure the integration:

• -l: The number of days to lookback to fetch incidents from. If running on a daily cadence, should set to 1
• -fs: filter string. Add a custom ODATA filter string to filter which incidents will be ingested into Collective Insights
• -ll: Log level of the integration

Example usage: python3 365_collective_insights.py -k=<Recorded Future API Token> -cid=< Client ID> -cs=<Client Secret> -t=<tenant ID>   -lb 1 -fs="serviceSource eq 'microsoftDefenderForEndpoint'"

Logic App

You can install the logic app using the attached ARM template by following the steps provided here.

FAQ

1. How do i generate the Client ID and Secret in Microsoft 365 Defender for Collective Insights?

Step 1: Generation of Client ID

Navigate to Microsoft Entra ID > App Registrations > New Registration > Input a name for the app > Select the Account type you want > Click register > In the overview of the app registration the id can be found in the Application (Client) ID field.

Step 2: Generation of Secret

Navigate to Microsoft Entra ID > App Registrations > Select your app > In the overview of the app > Click “Add a certificate or secret” > New Client Secret > Enter description > Select Expiry date > Click Add > The value field is the secret.

Step 3: Add permissions

Navigate to Microsoft Entra ID > App Registrations > Select your app > API permissions > Add a Permission > Click Microsoft Graph > Select "Application Permissions" > Scroll to SecurityAlert > Add the SecurityAlert.Read.All Permission > Add permissions > Ask a Global admin to approve the permission change.

Note: Delegated user permissions will not work.

2. Where do i find my tenant ID ?

Navigate to Microsoft Entra ID > In the overview panel there is a field that says Tenant ID, this is the tenant id that is needed.

3. Why don't i see any alerts or mismatch in the number of alerts between Microsoft 365 Defender and Detection Trends dashboard ?

Not all the alerts from Microsoft 365 Defender make it to Collective Insights. The following criteria has to match in order for an alert in Microsoft 365 Defender to be sent to Collective Insights:

a. The evidence in the alert should be one of the following: microsoft.graph.security.urlEvidence, microsoft.graph.security.ipEvidence, microsoft.graph.security.fileEvidence

b. There should be a non-null value for url, IP Address (ipAddress) and sha256 fields corresponding to microsoft.graph.security.urlEvidence, microsoft.graph.security.ipEvidence, microsoft.graph.security.fileEvidence type of alerts respectively.

4. Why don't i see any MITRE codes related to my Microsoft 365 Defender alerts in the Detection Activity dashboard ?

The following criteria has to match in order for MITRE codes in Microsoft 365 Defender alerts to be reflected under Detection Activity dashboard:

a. The MITRE Techniques (mitreTechniques) field in the alert should not be null.

5. What are the IP addresses which needs to be whitelisted to allow communication from the hosted service to Microsoft 365 Defender?

Collective insights for defender 365 shouldn't require any whitelisting, since we're only communicating with Microsoft's Graph API, not directly with the Defender in the client environment.

6. How to setup connector under a suborg in case of a Multiorg environment?

In order to setup a connector under a specific suborg, please switch as Enterprise admin to that particular suborg and create a connector following the same process mentioned here.

7. I am currently integrated with Microsoft Azure Sentinel. Do i still need to activate Collective Insights connector for Microsoft Defender XDR ?

If you are not pumping your Defender telemetry into your Sentinel or have not activated Collective Insights in Sentinel, it is recommended to activate Collective Insights connector for Microsoft Defender XDR.

Happy Hunting !!