Metadata-Version: 2.1
Name: psengine
Version: 1.12.1
Summary: Python 3 library for rapid integration development with Recorded Future's API
Author-email: ps@recordedfuture.com
Project-URL: Homepage, https://www.recordedfuture.com
Keywords: API,Recorded Future,Professional Services,Threat Intelligence
Requires-Python: <3.12,>=3.8
Description-Content-Type: text/x-rst
License-File: LICENSE
Requires-Dist: requests <=2.29.0,>=2.27.1
Requires-Dist: jsonpath-ng <=1.6.1,>=1.5.3
Requires-Dist: stix2 ~=3.0.1
Requires-Dist: stix2-patterns <=2.0.0,>=1.3.2
Requires-Dist: python-dateutil ~=2.8.2
Requires-Dist: more-itertools <=10.2.0,>=8.14.0
Requires-Dist: simplejson ~=3.19.2
Provides-Extra: dev
Requires-Dist: tox ==4.12.1 ; extra == 'dev'
Requires-Dist: build ==1.0.3 ; extra == 'dev'
Requires-Dist: pytest ==7.4.4 ; extra == 'dev'
Requires-Dist: pytest-vcr ==1.0.2 ; extra == 'dev'
Requires-Dist: pytest-cov ==4.1.0 ; extra == 'dev'
Requires-Dist: pytest-mock ==3.12.0 ; extra == 'dev'
Requires-Dist: ruff ==0.1.9 ; extra == 'dev'
Requires-Dist: wheel ==0.37.1 ; extra == 'dev'
Requires-Dist: setuptools ==61.0.0 ; extra == 'dev'
Requires-Dist: Sphinx ==7.1.2 ; extra == 'dev'
Requires-Dist: sphinxcontrib-confluencebuilder ==2.3.0 ; extra == 'dev'
Requires-Dist: atlassian-python-api ==3.41.4 ; extra == 'dev'
Requires-Dist: sphinx-autodoc-typehints ==1.25.2 ; extra == 'dev'

===========================================
PSEngine
===========================================
API driver from the Professional Services team, built to empower integration building. 

Configuration management, logging, and a collection of use cases available:

* Alerts
* Playbook alerts
* Risklists
* STIX2 formatted risklists
* Fusion files
* HTTP requests

===========================================
Contents
===========================================

1)    Pre-requisites

2)    Installation of Library

3)    Make Usage

4)    Project Directory Structure

5)    Library Usage

6)    Example Code Library Usage

7)    Gotchas

8)    Settings Location

9)    Settings & Logs Explained




===========================================
1. Pre-requisites
===========================================

Python version <= 3.8, >3.12 supported.

Administrator access to installation device - READ, WRITE, EXECUTE


============================================
2. Installation of Library
============================================

Build and install pip package

1) Make a new project directory ``[projectdir]``

2) After cloning, build the wheel file. From the psengine-py folder::

    $ make build
    $ cd pkg_build/psengine_[version]_[integer]
    $ cp psengine-<version>-py3-none-any.whl [projectdir]

3) Create a virtual environment in ``[projectdir]``::

    $ python3 -m venv venv
    $ source venv/bin/activate

4) Install pip module in ``[projectdir]``::

    $ pip install psengine-<version>-py3-none-any.whl

============================================
3. Make Usage
============================================
The ``make`` command has to be executed from within ``psengine-py/`` because this is where ``Makefile`` exists.

Help make commands::
    
    $ make help
    $ make debug


Commands to build the wheel file::
    
    $ make build


Commands to run before merging the code with Master::
    
    $ make black
    $ make test


============================================
4. Project Directory Structure
============================================
Below is a concrete example of how to get started using the ``get_alerts`` example application.

Once ``psengine`` has been installed in your virtual environment:

1) Copy the ``get_alerts`` folder contents into ``projectdir`` project directory::

    $ cp -R example/get_alerts projectdir

2) Edit configuration and python code as required

Your project directory should have the following structure:

::

    projectdir/
    ├── venv/ -> psengine is installed inside here
    ├── run_alerts.py
    ├── README.rst
    └── config/          
        └──settings.ini

``run_alerts.py`` imports classes from ``psengine`` and uses them.

``README.rst`` documents your code usage and outputs. Update it as your application evolves.

``config/`` contains a ``settings.ini`` for your program settings. (since PSEngine v1.6.0 this is not optional, refer to CHANGELOG for more details)

Further of this structure as well as file contents are available in
the ``examples/`` folder.

============================================
5. Library Usage
============================================

1) Ensure the following environmental variables are set::

    Linux/MacOS:
    $ export RF_API_KEY=[your API key]

    Windows:
    $ set RF_API_KEY=[your API key]

2) Initialise the built in logger & load configuration file:

.. code-block:: python

    from psengine.config import Config
    from psengine.logger import RFLogger
    
    LOG = RFLogger().get_logger()
    config = Config("config/settings.ini")



============================================
6. Example Code Library Usage
============================================
 
**Collective Insights Submission**

.. code-block:: python

    from psengine.config import Config, ConfigError
    from psengine.logger import RFLogger
    from psengine.collective_insights import (
        RFCollectiveInsights,
        RFInsight,
        DETECTION_TYPE_RULE,
        ENTITY_HASH,
        DETECTION_SUB_TYPE_SIGMA,
    )

    try:
        config = Config(join('config', 'settings.ini'))
        rci = RFCollectiveInsights(config)

    except (ConfigError, EnrichmentFieldsError) as ce:
        LOG.critical(ce, exc_info=False)
        sys.exit(1)
    
    insight1 = RFInsight(
        ioc_value="fbee00cb1d1ea4d7e0604436d9a36def71a9f3be804f1e2b8d117fd5d35aeabc",
        ioc_type=ENTITY_HASH,
        detection_type=DETECTION_TYPE_RULE,
        detection_sub_type=DETECTION_SUB_TYPE_SIGMA,
        detection_id="doc:o6_lui",
        detection_name="Instance of Alleged New Wiper Malware",
        ioc_field="hash",
        ioc_source_type="symantec",
        timestamp=now,
        incident_id="Incident 001",
        incident_name="Malware detected",
        incident_type="RF Sigma Rule",
        mitre_codes=["T1542", "T1485"],
        malwares=["Aesthetic Wiper"],
    )

    rci.submit(insight=insight1)



**Get alerts & update their status**

.. code-block:: python

    from psengine.config import Config
    from psengine.logger import RFLogger
    from psengine.legacy_alerts import RFAlertMgr

    LOG = RFLogger().get_logger()
    config = Config("config/settings.ini")
    alert_mgr = RFAlertMgr(config)
    alerts = alert_mgr.fetch_alerts()
    alert_mgr.write_file(alerts)
    # Set alert status for fetched alerts using config settings.ini
    alert_mgr.set_alert_status()



**Get a risklist**

Configure risklist settings in the configuration file and then get it by the stanza name as shown below:

.. code-block:: python

    from psengine.config import Config, ConfigError
    from psengine.logger import RFLogger
    from psengine.risklists import RFRiskListMgr

    LOG = RFLogger().get_logger()
    config = Config("config/settings.ini")
    risklist_mgr = RFRiskListMgr(config)
    risklist_content = risklist_mgr.get_risklist("risklist_ip_default")
    risklist_mgr.write_file(risklist_content, 'ip_default_risklist.csv')



**IOC Enrichment**

Configure an input and a custom output file, the connect api and/or soar api fields and run the enrichment. 
If during the execution some parameters have to be changed (ex having multiple IO files), the setter methods are available. 

.. code-block:: python

    from psengine.config import Config, ConfigError
    from psengine.logger import RFLogger
    from psengine.enirch import RFEnrichment

    try:
        config = Config(join('config', 'settings.ini'))
        rfe = RFEnrichment(config, 100)

    except (ConfigError, EnrichmentFieldsError) as ce:
        LOG.critical(ce, exc_info=False)
        sys.exit(1)
    
    rfe.soar_fields = {
        'domain': [
            'risk.score',
            'risk.rule.evidence.linkedToMalware.rule',
            'risk.context.malware.score',
        ]
    }

    LOG.info(rfe.lookup(['google.com', 'facebook']))
    data = rfe.read_file('input_file.csv')
    enriched_data = rfe.lookup(data)

    rfe.write_file(enriched_data)
    data = rfe.soar(['google.com', 'facebook.com', '203.210.192.55'])
    rfe.write_file(data)
    



============================================
7. Gotchas
============================================

Missing packages:

* If developing PSEngine do a pip install of ``requirements_dev.txt``. Refer to the `PSEngine Confluence Page <https://recordedfuture.atlassian.net/wiki/spaces/ProfServOps/pages/3049881699/PSEngine>`_ for more information.

  * This contains extra packages needed for development, like dependencies for testing and linting.

* If developing against PSEngine do a pip install of ``requirements.txt``.

  * This contains only the packages needed to run PSEngine.

============================================
8. Settings Location
============================================

**settings.ini**

To change behaviour edit ``config/settings.ini``
   

**NOTE:** ``psengine-py`` does not have a ``config`` folder of its own. These live
inside your project directory


============================================
9. Settings
============================================

**settings.ini**

[alerts]

* fetch_status = Alerts status to fetch. Options: ``unassigned|assigned|pending|dismiss|no-action|actionable|tuning``
* update_status = Enables calls to ``RFAlertMgr.set_alert_status``. Options: ``true|false``
* new_status = Update alert status to this status. Options: ``unassigned|assigned|pending|dismiss|no-action|actionable|tuning``
* format = Alert format to output. Options: ``standard|raw``
* triggered = Alerts triggered since. For example: ``2h``, ``1d``, ``7d``
* alert_rules = An array of alert rulename prefixes. ``["*"]`` fetches all alerts


[playbook_alerts]

* limit = Limit of alerts to fetch. For example: ``1``, ``5``, ``100``
* fetch_image = Fetch images. Options: ``true|false``
* panels = Panels to fetch. Options: ``["status", "action", "summary", "dns", "whois", "log"]``
* statuses = statuses to fetch. Options: ``["New", "InProgress", "Dismissed", "Resolved"]``
* category = categories to fetch. Options:  ``["domain_abuse"]``
* priority = priority to fetch. Options:  ``["High", "Moderate", "Informational"]``
* created = created date range. For example: ``2h``, ``1d``, ``7d``
* updated = updated date range. For example: ``2h``, ``1d``, ``7d``

**NOTE:**  ``created`` and ``updated``, when queried together, have an ``AND`` relationship.
           Only alerts that match both criteria will be returned.

[risklist_*]

* entity_type = ``ip|hash|domain|url|vulnerability``
* enabled = ``true|false``
* interval = Interval in seconds for risklist refresh, for example: ``3600``
* fusion_file = Path to a fusion file, for example: ``/home/risklists/elk_ecs_ip_default_risklist.csv``
* list = List parameter from the ConnectAPI

**NOTE:**   Fusion_file and list parameters cannot be specified at the same time. 
            Not specifying any of them will result in fetching the default risk list for the entity type.

[enrichment]

* input_file = Name of the input file containg the IOC to be enriched. One IOC per line with no separators. ``str``
* lookup_fields =  Fields to be extracted for Connect API ``list`` for example ``["risk.score","risk.evidenceDetails[*].rule"]``
* soar_fields = Fields to be extracted for SOAR API ``dict`` for example ``{"hash": ["risk.score"], "ip": ["risk.score", "risk.context.public.mostCriticalRule" ] }``

[output]

* alerts = directory to output alerts to. Options: ``alerts/|full path``
* risklists = directory to output risklists to. Options: ``risklists/|full path``
* enrichment = directory to output enriched IOCs. Options: ``enrichment/|full path``

[rfapi]

* proxy_enabed = ``true|false``
* proxy_servers = an array of proxies to be used
* verify_ssl = ``true|false``

============================================
10. Logging
============================================

If your project requires a logger object with sensible settings for console and file handling, use ``RFLogger``.

``logger = RFLogger().get_logger()`` will create a ``psengine`` logger instance with the following settings:

* ``RFLogger`` constructor takes output logs directory (default: ``sys.path[0] + logs/``), log level (default: ``INFO``), and ``propagate`` flag (default: ``False``)
* Creates file handler and console handler
* File handler is a rotating log handler, described below
* Uncaught exceptions will be caught and logged

Otherwise, if psengine should be logging to another logger instance, all classes use ``logging.getLogger(__name___)`` and will report log records to instances higher up the hierarchy.

``RFLogger`` produces logs in the script directory ``logs/`` or a directory of the user's choice,
created using a rotating file handler with a max file size of 20MB up to 5 total files. Log filenames begin with ``recfut.log``. 
When the maximum file size is reached, this file is renamed ``recfut.log.1``. ``.1`` becomes ``.2``, and so on up to ``.4``. 
The most current log records will always be in ``recfut.log``.

The logs written on file have a different syntax compared to the on screen logs. Every statement will show the Thread name in brackets::

    2022-10-14 14:25:59,920 [MainThread] INFO [rf_enrichment] soar:490 - SOAR API - Starting Enrichment...
    2022-10-14 14:25:59,920 [ThreadPoolExecutor-2_0] DEBUG [rf_enrichment] _soar_api:398 - SOAR API - Multithreading debug: _soar_api() assigned to thread: ThreadPoolExecutor-2_0.


============================================
11. Troubleshooting
============================================

Any uncaught exceptions get logged to the console and the log file, for instance::

    $ 2022-01-14 10:12:44,852 CRITICAL [rf_logger] log_critical_exceptions:69 - An unexpected error has occurred
    $ Traceback (most recent call last):
    $ File "/Users/ernest/workspace/ps/psengine-py/rfalerts.py", line 79, in main2()
    $ File "/Users/ernest/workspace/ps/psengine-py/rfalerts.py", line 74, in main2
    $ print(notcool)
    $ UnboundLocalError: local variable 'notcool' referenced before
