This document is organized as follows:
- General Overview
- Available metadata types
- Examples using dynamic labels
- Entity metadata types
- Styling
- Multiline text
General Overview
The metadata file describes the output json of the extension, and how it should look in the Recorded Future UI. It is needed for resolving entities returned in the extension response and for the Intelligence card rendering of the response data. All key / values returned in the extension response must be mapped in the metadata file. Metadata is a superset of all keys that could be returned in the extension response. If an extension response contains unmapped keys an error message will be shown in the intelligence card.
For reference, attached are three files that illustrate a finished metadata.json file, the expected response from the python code, and a screenshot of the extension; theyare separate from the examples below.
Example of a metadata.json file
{
"entries": [
{
"key": "ip_address",
"label": "IP Address",
"type": "IpAddress"
},
{
"key": "ip_other_domains",
"label": "Co-Hosting",
"type": "text"
},
{
"key": "registrant",
"label": "Registrant",
"type": "text"
},
{
"key": "registrant_domains",
"label": "Registered Domains",
"type": "text"
},
{
"key": "website",
"label": "Website",
"type": "dict",
"entries": [
{
"key": "title",
"label": "Website Title",
"type": "text"
},
{
"key": "response_code",
"label": "HTTP Response Code",
"type": "integer"
},
{
"key": "server_type",
"label": "Web Server Type",
"type": "text"
}
]
},
{
"key": "created",
"label": "Created on",
"type": "date"
},
{
"key": "recently_created",
"label": "Recently Created",
"type": "text"
},
{
"key": "registrar",
"label": "Registrar",
"type": "text"
},
{
"key": "history",
"label": "History",
"type": "dict",
"entries": [
{
"key": "hosting_history",
"label": "Hosting History",
"type": "text"
},
{
"key": "ip_history",
"label": "IP History",
"type": "text"
},
{
"key": "ip_history_details",
"label": "Historic IPs",
"type": "list",
"item": {
"type": "IpAddress"
}
},
{
"key": "registrar_history",
"label": "Registrar History",
"type": "text"
},
{
"key": "whois_history",
"label": "Whois History",
"type": "text"
}
]
},
{
"key": "email",
"label": "Contact Email",
"type": "list",
"item": {
"type": "EmailAddress"
}
},
{
"key": "name_servers",
"label": "Name servers",
"type": "list",
"item": {
"type": "InternetDomainName"
}
},
{
"key": "registrant_address",
"label": "Registrant Address",
"type": "text"
},
{
"key": "screenshot_history",
"label": "Screenshots",
"type": "link"
},
{
"key": "host_domain_count",
"label": "Total Domains for Host",
"type": "integer"
},
{
"key": "host_domain_names",
"label": "Domains Hosted at IP",
"type": "list",
"item": {
"type": "InternetDomainName"
}
},
{
"key": "community_link",
"label": "Browse",
"type": "link"
}
]
}
Available metadata types
The following types are available:
- text - used for labels or plain text values
- integer - used for count values
- date - will be displayed in the intelligence card on the format [month day, year]
- datetime - will be displayed in the intelligence card on the format [month day, year, hh:mm:ss]
- To create a datetime value from a timestamp in the correct format use the utility function create_datetime
- list - displays a list of values that are expandable. A top level list will start expanded and have a zebra pattern.
- dict - displays a dictionary with keys and values that are expandable. A top level dict will start expanded.
- link - the link type is a dictionary containing the keys url and name in the response data
{
"rf_link": {
"url": "www.recordedfuture.com",
"name": "Recorded Future",
}
}
{
"key": "rf_link",
"label": "Try it out",
"type": "link"
}The list type is followed by a key item that describes the type of the elements in the list. The initial state of a list (expanded or collapsed) can be specified with the key collapse.
{
"key": "a_records",
"label": "Resource record type A",
"type": "list",
"collapse": true,
"item": {
"type": "dict",
"entries": [
{
"key": "bailiwick",
"type": "bailiwick"
},
{
"key": "first_seen",
"type": "first_seen"
},
...
{
"key": "name_server",
"label": "Name server",
"type": "InternetDomainName"
}
]
}
}
The dict type is followed by a key entries which is a list to state the order of the elements in the dict.
{
"key": "dates",
"label": "Dates",
"type": "dict",
"entries": [
{
"key": "created",
"label": "Created on",
"type": "date"
},
{
"key": "expires",
"label": "Expires on",
"type": "date"
},
{
"key": "updated",
"label": "Updated on",
"type": "date"
}
]
}
The dict type supports dynamic labels. Dynamic labels are useful when the data determines the labels you want to show and can't be specified statistically in the metadata.
Examples using dynamic labels
JSON response from extension
{
"records": [
{
"year": "2010",
"values": [
{
"start": "2010-03-07T04:09:54",
"stop": "2010-05-21T05:14:28",
"host": "ip:1.2.3.4"
}
]
},
{
"year": "2011",
"values": [
{
"start": "2011-03-28T05:12:28",
"stop": "2011-04-21T05:14:28",
"host": "ip:8.8.8.8"
},
{
"start": "2011-05-28T05:12:28",
"stop": "2016-09-21T05:14:28",
"host": "ip:8.8.8.8"
}
]
}
]
}
Extension metadata
{
"entries": [
{
"key": "records",
"label": "Records",
"type": "list",
"collapse": true,
"item": {
"type": "dict",
"label_key": "year",
"item": {
"key": "values",
"type": "list",
"item": {
"type": "dict",
"entries": [
{
"key": "start",
"label": "Start of event",
"type": "datetime"
},
{
"key": "stop",
"label": "Stop of event",
"type": "datetime"
},
{
"key": "last_seen",
"type": "last_seen"
},
{
"key": "host",
"label": "IP Address",
"type": "IPAddress"
}
]
}
}
}
}
]
}
Use label_key in the metadata file to specify which key should be used as label. Use key inside item to specify which key holds the data to be rendered under the dynamic label.
Entity metadata types
When returning an entity value in the extension response the metadata shall specify the type of that entity so that it can be resolved to a Recorded Future entity. The available entity types are
- IpAddress
- This type should also be used for IP ranges in the CIDR notation. Note that the /32 notation should not be used for individual IP addresses.
- InternetDomainName
- This type should be used for all DNS names (also FQDN, NS and MX DNS names)
- EmailAddress
- URL
- ASNumber
- The expected syntax for a valid ASNumber is "ASxxxxx" where "xxxxx" is the number itself. Values where the leading "AS" is omitted will cause a type mismatch and result in unexpected behavior when rendering the partner's response in the intelligence card extension.
- PaymentCardNumber
- Hash
- Malware
- Malware may have many names; the most common or widely accepted one as found in Recorded Future's Ontology will be passed to the Partner's API for search. For example, for Dridex/Bugat/Cridex, only Dridex would be passed through the Extension API for use in searching the Partner API.
- CyberVulnerability
- ThreatActor [coming soon]
Styling
It is possible to indicate a certain style for a value in the extension response data. It might be that a value should be highlighted under certain conditions. The style-key is therefore not specified in the metadata file but instead in the response data if the extension source code logic finds a special value. The syntax for this in the response data json is:
{"entity_1":{"value":"1.2.3.4", "style":"highlight"}}
As opposed to a regular unstyled element
{"entity_2":"1.2.3.4"}
Any values in the response json could be styled by returning the vale as a dictionary with keys value and style. Currently the supported value for the style key is highlight.
Multiline text
To render multiple lines of text, use an array of strings, where each record of the array represents a single line of text. For example, to show an address over multiple lines, the response data should look like this:
{ "address" : [ "363 Highland Avenue", "Suite 2", "Somerville, MA 02144", "United States" ] }and the metadata would be as follows:
{
"key" : "address",
"type" : "list",
"item" : {
"type" : "text"
}
}