Recorded Future Identity Module Installation and Configuration for Splunk:

- Before you begin notes:
---- These instructions assume a default Linux installation in the /opt/splunk/ folder
---- For Windows users, the default install directory is C:\Program Files\Splunk\
---- If you are using a search head cluster or Splunk Cloud, run our app from the heavy 
     forwarder and configure the script below 
-------- If running on a heavy forwarder, in the Recorded Future App for Splunk, disable
         all Risk Lists within the app configuration tab once the API token has 
         been configured
-------- If running on a heavy forwarder, copy props.conf into the search head local/
         folder for the Recorded Future App for Splunk (prevents duplicate indexing of 
         JSON fields)
---- If you only have one search head it can be installed on that one search head 



<---------- Setting up the indexes and dashboard ---------->

- Login to the Splunk Web GUI and create two indexes:
  "rfidentities": Stores the identity data
  "rfidentities_log": Stores the log events from script runs

- Add the custom identity dashboard to the Recorded Future App for Splunk
  1. Copy the XML source content of "recorded_future_identities.xml"
  2. In the Splunk Web GUI, navigate to Settings -> User interface -> Views -> New View
  3. Add a View name: "recorded_future_identities"
  4. Paste the XML source into the View field, and click Save
  5. In the Splunk Web GUI, navigate to Settings -> User interface -> Navigation menus
  6. Find the row with App: TA-recordedfuture and click the "default" link to edit it
  7. Add a new collection called "Custom Views", and add the new view as a collection element.
     Note that this collection may exist as the result of migrating views from 1.x --> 2.0.

<collection label="Custom Views"> 
<view name="recorded_future_identities"/> 
</collection>, and click Save

  8. Click Save

  To view the new dashboard, in the Recorded Future App for Splunk, navigate to
  Custom Views -> Recorded Future Identities. Note that no data has been ingested yet.



<---------- Setting up the script and configuration files ---------->

- Login via CLI to the Splunk Search head that has the Recorded Future app installed 

- Copy recordedfuture_identities.py and splunklib/ into the TA-recordedfuture/bin folder:

	$ cp recordedfuture_identities.py /opt/splunk/etc/apps/TA-recordedfuture/bin
  $ cp -R splunklib /opt/splunk/etc/apps/TA-recordedfuture/bin

The directory structure should look like this:

TA-recordedfuture/
└── bin/
    ├── recordedfuture_identities.py
    └── splunklib/
        ├── __init__.py
        ├── binding.py
        ├── client.py
        ├── data.py
        └── six.py
    ├── ...

- Change the owner and group of the script via:
	$ chown splunk:splunk recordedfuture_identities.py

- Modify the permissions of the recordedfuture_identities.py script via: 
  $ chmod 755 recordedfuture_identities.py


- In /opt/splunk/etc/apps/TA-recordedfuture/local, create the following files.
  If the .conf files already exist, append the stanzas to the existing files.

commands.conf
  [recordedfuture_identities]
  filename = recordedfuture_identities.py
  passauth = true
  enableheaders = true
  python.version = python3

inputs.conf
  [monitor:///opt/splunk/etc/apps/TA-recordedfuture/local/rfidentities_*.json]
  disabled = false
  host = IdentityAPIRecordedFuture
  index = rfidentities
  sourcetype = rf_identity_json
  crcSalt = <SOURCE>

  [monitor:///opt/splunk/etc/apps/TA-recordedfuture/local/recordedfuture_identitydetails.log]
  disabled = false
  host = IdentityAPIRecordedFuture
  index = rfidentities_log
  sourcetype = rf_identity_log

props.conf
  [rf_identity_json]
  TRUNCATE = 0
  KV_MODE = none
  AUTO_KV_JSON = false
  detect_trailing_nulls = auto
  NO_BINARY_CHECK = true
  SHOULD_LINEMERGE = false
  INDEXED_EXTRACTIONS = JSON
  DATETIME_CONFIG = CURRENT
  FIELDALIAS-rf_identity_subject = "subject" AS src_user
  FIELDALIAS-rf_identity_domain = "authorization_service.domain" AS src_user_domain
  FIELDALIAS-rf_identity_url = "authorization_service.url" AS url

  [rf_identity_log]
  TRUNCATE = 0
  KV_MODE = none
  AUTO_KV_JSON = false
  detect_trailing_nulls = auto
  NO_BINARY_CHECK = true
  SHOULD_LINEMERGE = false
  TZ = UTC
  TIME_PREFIX = /(?:[01]\d|2[0-3]):(?:[0-5]\d):(?:[0-5]\d)/

tags.conf
  email = enabled

savedsearches.conf
  [RecordedFutureIdentities]
  enableSched = 1
  cron_schedule = 0 */1 * * *
  dispatch.earliest_time = -1h
  search = | script recordedfuture_identities -d [DOMAINS] -ld [LOOKBACK DAYS]

  [RecordedFutureIdentitiesDashboard]
  enableSched = 0
  search = [SEE FILE FOR SEARCH STRING]

  [Recorded Future Leaked Credentials]
  enableSched = 0
  ...[This file contains a search for producing Splunk Alerts and Notable Events]...

  ** You will have to change the list of domains and the lookback days
     in the "RecordedFutureIdentities" stanza. 
     Lookback days correspond to the latest downloaded date. 

  ** It is recommended to use a lookback of at least 72 hours (-ld 3) to ensure that any new credential data 
     that is obtained by Recorded Future is captured by the integration. The integration will handle any 
     deduplication of credentials that are already on disk.

     Example: | script recordedfuture_identities -d google.com gmail.com drive.google.com -ld 7

  ** If you wish to enable Splunk Alerts and Notable Events for credentials that are indexed,
     set enableSched = 1 in the "Recorded Future Leaked Credentials" stanza.


If using Windows, use the following monitor path in inputs.conf:
  [monitor://C:\Program Files\Splunk\etc\apps\TA-recordedfuture\local\rfidentities_*.json]
  [monitor://C:\Program Files\Splunk\etc\apps\TA-recordedfuture\local\recordedfuture_identitydetails.log]



<---------- Running the script and using the dashboard ---------->

- Restart or perform a debug refresh of the Search Head (or rolling restart)

- Invoke the script for the first time manually, from a Splunk search.
  For this first search, it's often useful to use a larger timeframe:

	| script recordedfuture_identities -d [DOMAINS] -ld [LOOKBACK DAYS]

- You can view the raw results with this search:
  index="rfidentities" sourcetype="rf_identity_json"


Troubleshooting:

- The savedsearches.conf file will instruct Splunk on when to run the script.
  You can also add additional command line arguments to the search command based on 
  what properties you want for returned credentials.

- If the script fails, Splunk will not show a stack trace in the Web GUI
  To troubleshoot the issue, review the logs in the dashboard panel located
  at the bottom of the dashboard. If the most recent log results are not appearing,
  refresh the search. Log results from the script are indexed at "rfidentities_log".

- A successful invocation will produce a file named "rfidentities_[timestamp].json" in the app's
  local directory (/opt/splunk/etc/apps/TA-recordedfuture/local). These credentials are indexed at
  "rfidentities", and will be displayed in the first dashboard panel.

- If the script has run successfully and no output file was produced, no credentials 
  were found or the credentials were written to disk previously within the last 7 days.
  Check the log results for verification.

- If an output file was produced but the credentials are not displaying in the first
  dashboard panel, refresh the dashboard. It may take some time to index the 
  credentials depending on how many credentials were retrieved and written to disk.

- For all other errors, run the script manually from the command line to debug.
  You will need to provide the API key if running from the command line.
  See the script details section for example usage.

- File contents are used to ensure identity credentials are not duplicated on disk.

- Files are retained on disk for 1 week.



<---------- Script Details ---------->

Usage:

python recordedfuture_identities.py -h for usage output.
Use "--" with the full argument name, or "-" with the one letter abbreviation

---- Required Arguments

token:         Recorded Future API token for Identity Module

lookback_days: Number of days since credential was last downloaded

domains:       Domains to search exposed credentials for


---- Optional Arguments

domain_types:         'Authorization' (External) or 'Email' (Internal)
                      default: None (includes both)

password_properties:  Password properties of the exposed credentials
                      [Letter, Number, Symbol, UpperCase, LowerCase, MixedCase,
                      AtLeast8Characters, AtLeast12Characters, AtLeast16Characters,
                      Atleast24Characters]
                      default: []

exfiltration_days:    Number of days since credential was exfiltrated from host machine
                      default: None

breach_name:          Name of the breach where the exposed credential was found
                      default: None

breach_lookback_days: Number of days since the specified breach
                      default: lookback_days

dump_name:            Name of the dump where the exposed credential was found
                      default: None

dump_lookback_days:   Number of days since the specified dump
                      default: lookback_days         

log:                  log level, options are debug, info, warn, error, critical.
                      default: info

dir:                  target directory for output files. 
                      default: ../local/

no-verify:            disable SSL verification (to be used with proxy configurations)
	                    default: False


---- Usage Example
To invoke recordedfuture_identities to find exposed credentials for `recordedfuture.com` downloaded
within the last 30 days, with log level info, and writing to output to ../local/:

	python3 recordedfuture_identities.py -t [API_KEY] -d recordedfuture.com -ld 30



<---------- Logging Information ---------->

Logs:

recordedfuture_identities produces logs in the ../local/ directory by default.
Logs are created using a rotating file handler with a max file size of 2MB for up 
to 5 total files. Log filenames begin with recordedfuture_identitydetails.log. When
the maximum file size is reached, this file is renamed
recordedfuture_identitydetails.log.1. Then, .1 becomes .2, and so on up to .4.
The most current log records will always be in recordedfuture_identitydetails.log.

Logs record the number of identities and credentials retrieved, the number of
new credentials, output files written, and any errors encountered.
