Identity Intelligence Module Splunk Integration

Table of Contents

• Use Case
  • Solutions 
  • Issues 
• Technical Requirements 
• Installation Instructions 
  • Installing python package
  • Deploying Search & Dashboard
  • Troubleshooting 

Use Case 

The Recorded Future Identity Intelligence module provides visibility into billions of compromised credentials with attached context and analysis that provides actionable intelligence. The module focuses on two use cases: “Workforce” (aka employee compromise) and “External” (aka customer compromise). Pulling down exposed credentials can be used for automating tasks, analyst notification, and historical knowledge. 

Solutions 

• Automated user password change upon notification of a user’s leaked credential finding 
• Detecting users whose credentials are frequently found in breaches 
• Analyze breached password properties to determine new password requirements for the organization 

The integration starts with a dashboard summary of your exposures:

In the below example, Splunk is used to query our Identity API to discover compromised identities. In this case, it would search for exposed credentials where the login domain was norsegods[.]online. The total number of identities, total number of credentials, and the top identities are returned. 

The results in the next panels include a compromised credentials timeline, top FQDNs, top malware, and top technologies exposed. 

The results in the following panels display the list of identities found from the search criteria used at the top of the dashboard with some general information about the credentials.

Click on a specific subject to drill down into the details of that identity’s leaked credentials. For example, you can see the account name, malware type used, date of exfiltration, date of last first and last download, password properties, cookie information, and associated dumps and breaches. 

Issues 

• Inability to view exposed credentials found by Recorded Future outside of the Platform and API 
• Inability to perform automated actions off of exposed credentials found by Recorded Future 
• Inability to store exposed credentials locally for historical knowledge 

Technical Requirements 

The integration has the following technical requirements: 

• Recorded Future App for Splunk 

>= v.1.1.9 - Recommended for clients who do NOT  license an integration with Recorded Future for Splunk 

>= v2.0 - Recommended for clients who license an integration with Recorded Future for Splunk 

• A Python script and several conf configuration files that are added to the Recorded Future App for Splunk
• Two indexes created to house the Recorded Future identities details and log data
• A saved search that is used to index identity data
• A dashboard to search for identities and view results 

This solution requires the Identity Intelligence module. There are separate instructions for clients who own the Recorded Future for Splunk integration and for those without the integration. This integration is intended for use as-is. Ongoing support is provided by Integration or Premium Support. Further requests or enhancements require scoping by Professional Services. 

Installation Instructions 

Installing python package

1. Install the latest 1.1.x version of the Recorded Future App for Splunk from splunkbase. 

• Currently v1.1.9 as of 2023-10-17 

2. Navigate to the Recorded Future App for Splunk and enter your API Token provided by your account team into the configuration page.

• Splunk Cloud clients will deploy this custom package on a Heavy Forwarder 
• Splunk on-prem clients can choose to install on a Search Head or a Heavy Forwarder
• Universal Forwarders and IDMs are not supported

3. Create two new indexes that will house the identity data from Recorded Future within Splunk:

• rfidentities: Stores the identity data
• rfidentities_log: Stores the log event from script runs

4. After downloading the Recorded Future Identity package for Splunk, copy the zip file over to your Splunk server for configuration via CLI.

5. Copy the following files into their respective directories. 

a. recordedfuture_identities.py

  i. cp recordedfuture_identities.py /opt/splunk/etc/apps/TA-recordedfuture/bin/ 

If you are using a proxy, you will need to use the proxy version of the script. Run the below command instead:

ii. cp recordedfuture_identities_proxy.py /opt/splunk/etc/apps/TA-recordedfuture/bin/

For the proxy version of the script, modify lines 76 and 77 to reflect your proxy information.

b. splunklib/

cp -R splunklib /opt/splunk/etc/apps/TA-recordedfuture/bin/

c. Note: If you are an existing Recorded Future Splunk client, some of these configuration files may already exist. Do not copy these files over and instead append the contents to the existing files.

commands.conf, inputs.conf, props.conf, savedsearches.conf, tags.conf

cp *.conf /opt/splunk/etc/apps/TA-recordedfuture/local

The TA-recordedfuture directory should look like the following: 

6. Change permissions of the files for Splunk to operate properly.

sudo chown -R splunk:splunk /opt/splunk/etc/apps/TA-recordedfuture
sudo chmod 755 /opt/splunk/etc/apps/TA-recordedfuture/bin/recordedfuture_identities.py

7. Restart Splunk for changes to take effect.

2. Save your changes

3. Run the “RecordedFutureIdentities” search manually for the first time. This will index credentials based on the saved search you edited above. You can opt to use a longer timeframe for lookback days if you’d like to backfill any credentials.

4. Navigate to the Settings → User Interface → Views → + Add new to create the Recorded Future Identities dashboard.

5. Use “recorded_future_identities” for the view name. Paste in the XML from the recorded_future_identities.xml file from the downloaded package and click Save. 

Repeat this by adding an additional view named "recorded_future_identities_overview" with content from "recorded_future_identities_overview.xml"

6. Add a drop-down option to the app navigation bar by navigating to Settings → User Interface → Navigation Menus → Open the TA recordedfuture menu. 

7. Paste in the following collection: 

<collection label="Custom Views">  
<view name="recorded_future_identities"/>  
<view name="recorded_future_identities_overview" />
</collection>

8. Save your changes.

9. Setup is now complete and you can access the dashboard from the Recorded Future App for Splunk. Credentials will be indexed on an hourly basis from the “RecordedFutureIdentities” search and credentials can be viewed from the Recorded Future Identities dashboard. 

Troubleshooting 

• The savedsearches.conf file will instruct Splunk on when to run the script. You can also add additional command line arguments to the search command based on what properties you want for returned credentials.
• If the script fails, Splunk will not show a stack trace in the Web GUI. To troubleshoot the issue, review the logs in the dashboard panel located at the bottom of the dashboard. If the most recent log results are not appearing, refresh the search. Log results from the script are indexed at rfidentities_log.
• A successful invocation will produce a file named rfidentities_[timestamp].json in the app's local directory (/opt/splunk/etc/apps/TA-recordedfuture/local). These credentials are indexed at rfidentities and will be displayed in the first dashboard panel.
• If the script has run successfully and no output file was produced, no credentials were found or the credentials were written to disk previously within the last 7 days. Check the log results for verification.
• If an output file was produced but the credentials are not displaying in the first dashboard panel, refresh the dashboard. It may take some time to index the credentials depending on how many credentials were retrieved and written to disk.
• Splunk clients installing on a heavy forwarder may experience issues with duplicate field indexing. To prevent this from occurring at ingest time, copy the content of the props.conf file into the search head local/folder of the Recorded Future app for Splunk. For Splunk Cloud clients, this would require the use of the Configuration REST API. Another option is to create a custom Splunk app that uses the configuration files instead of the Recorded Future app.
  • At search time, the duplicate fields can be removed with the use of mvdedup. Additional SPL found in the mvdedup_spl.txt attachment can be added to the “RecordedFutureIdentitiesDashboard” saved search before the final rexand dedupcommands.

• For all other errors, run the script manually from the command line to debug. You will need to provide the API key if running from the command line. See the script details section for example usage.
• File contents are used to ensure identity credentials are not duplicated on disk.
• Files are retained on disk for 1 week. 

For any other questions, please reach out to your account team or support@recordedfuture.com