[this is for v4.0.x of the Recorded Future App for Splunk Enterprise]

Adapt searches

Correlation searches

Recorded Future has a compilation of common log formats and how to correlate these with our risklists on our support web: Common Splunk Search Strings for Recorded Future Risk Lists

There is also a feature in Splunk called "Workflow actions" which allows for context dependent menu entries for search results.

It's easy to add more such actions for fields not covered in the default set of actions: Pivoting to Recorded Future Enrichment Data in Splunk.

Base search

The base search for the correlation dashboards look almost the same. This is the default base search for the IP Log Correlation Dashboard:

    index=main sourcetype="netscreen:firewall" earliest=-24h 
   | eval Name=dst
   | join [|inputlookup rf_ip_risklist.csv | table * ]
   | search Risk != "" 
   | sort -Risk

The base search is used, as the name suggests, as a base for more specific searches and is used to be able to both optimise but also to make it easier to write queries since you only have to write the first part of the search once. This also lowers the risk of copy-paste errors.

The first row selects the index 'main', sourcetype 'netscreen:firewall' and time scope 'last 24 hours' for the search.

The second row we rename the field 'dst' in the logs to 'Name' by using the eval statement.

In the third row we join the results from the prior search with a table containing all fields available in the ip_risklist.csv lookup file. In older versions of this dashboard we used the lookup command instead. Lookup if, for our purposes, much slower and will not remove any non-matching events from the result, which is why we moved to use join and a subsearch with inputlookup instead.

In the forth row we remove all results that does not have a Risk assigned to them, thus removing any results with no matches in the risk list, by searching for results there the field 'Risk' is not empty.

In the last row we sort the results descending, by adding a '-' before the field name, with regards to the Risk, thus making sure that the high risk results end up at the top.

Typical changes here would be to use another list to match the data with, such as a custom risk list. In the Recorded Future risk lists, the main element to match on is called Name, for example the domain in a domain risk list and the CVE in a vulnerability risk list. If a custom risk list is used, make sure to change the second row to match the field names of both the data in Splunk and the risk list.

Sub searches

This is the search that determins the amount of unique rows in the base search. The first statement does a statistical distinct count, thus only counting the unique values in the field 'Name' and saves the result in the field 'count'. This value is then colored by using the rangemap statement.

    | stats dc(Name) as count
    | rangemap field=count low=0-0 evelated=1-1 default=severe

In this subsearch, which is the last table in the Correlation Dashboards, we start by counting the occurrances of each value in the field 'Name'. The difference between stats and eventstats is that with eventstats a new field is added to each row in the results. In this case the field 'Count'. We then remove all duplicate rows in regards to 'Name'. 'format_evidence' is a macro created by Recorded Future to parse the JSON object in the field 'EvidenceDetails', thus populating new fields with the data contained there. We then sort, descending, by and remove all decimals from Risk before creating a table showing the final information.

    | eventstats Count BY Name
    | dedup Name
    | `format_evidence`
    | sort -Risk
    | eval Risk = round(Risk,0)
    | table Risk, Name, Count, RiskString, Rule, Evidence

Enrichment searches

The enrichment dashboard uses a REST endpoint to get information from the Recorded Future Connect API to get more information about the entered entity. Please note that this REST endpoint, and thus the enrichment dashboards uses API credits.

Base search

The base search for each of the enrichment dashboards contains a REST call to get the information about the entity entered. Here it is possible to customise which kind of information you want to get back from the API. In our case we have selected to get all available fields to be able to present as much data as possible to the user. The available fields can be found at the Recoded Future API page. The reason for the CDATA encapsulation, in other words <![CDATA[before and ]]> after the search, is due to Splunks handling of the characters in the regular expression in this search. The spath statement is used to parse JSON objects to be able to use the information in searches. The last two statements are to convert the timestamps to a much more readable format for humans.

    <![CDATA[
    | rest /services/TA_recordedfuture-cyber/lookup_ip/$name$ fields="counts,risk,analystNotes,location,sightings,entity,timestamps,metrics,relatedEntities,riskyCIDRIPs,intelCard,threatLists" output_mode=json
    | `unpack_metrics`
    | rex field=risk.riskString ".*/(?<rulesMax>.+)"
    | rename rulesMax TO risk.rulesMax
    | spath input=timestamps 
    | eval FirstSeen=strftime(strptime('timestamps.firstSeen', "%Y-%m-%dT%H:%M:%S.%NZ"), "%b %d, %Y") 
    | eval LastSeen=strftime(strptime('timestamps.lastSeen', "%Y-%m-%dT%H:%M:%S.%NZ"), "%b %d, %Y")
    ]]>

Sub searches

A lot of the sub searches in the enrichment dashboards are quite similar, especially when it comes to the related entities tables. One example is the one below for related domains:

    | `unpack_relatedEntities(RelatedInternetDomainName)`
    | sort -count, name
    | rename name as Domain
    | rename count as References
    | table Domain, References

The first thing we do is to call the macro 'unpack_relatedEntities' which parses the JSON object in the field sent to the macro. We use a macro here to not have to write the same code over and over and it also makes sure that we do it in the same way for all the related entity tables. Then we sort and rename some fields to get the headers and order we want. The last last row creates the table that we see in the dashboard, showing the fields 'Domain' and 'References'.

Further help

Your Recorded Future Intelligence Services consultant would be happy to help you with additional questions and advice.  If you do not know who that is, you can also contact support@recordedfuture.com.

Please do not contact Splunk support about "Recorded Future for Splunk Enterprise".