Unable to run all parsers when pre-processor detects a suitable preset

[nflexfo]

Describe the problem:

It is not possible to run all parsers when no parser filter expression is provided and pre-processor detects a specific OS.

More specifically, when running on a Windows image, it falls backs to the win7 preset which does not include the mft parser. It can be worked around with something like --parsers win7,mft, --parsers win7_slow, or even specifying the complete list. But then, what if the disk also contains data that could be parsed by spotlight_storedb (randomly chosen), or any "future" parser for that mean.

Furthermore, all parsers seems to be enable when the pre-preprocessor cannot detect a suitable preset. That is, dependending on the source format (disk or directory), different set of parsers are enable albeit no parser filter is ever set by user. This is confusing.

I understand the default behavior to use win7 instead of win7_slow suits most user. In our case, we prefer to extract everything possible and filter data afterwards by other means and tools.

What is expected is either:

a) Run all parsers unless specified otherwise
b) Add an additional magic option --parsers all (similar to --parsers list or --partitions all)
c) Add a switch to disable automatic preset detection (like --skip-preset-detection)

To Reproduce:

Plaso: 20240826
OS: Linux
Install: Sources
Data Source: The Windows 7 disk from Data Leakage Case (CFReDS)

Filter file (l2t_filter_mft.yaml):

description: File system metadata files (MFT).
type: include
path_separator: '\'
paths:
- '\\[$]MFT'

Reproducer cmd:

./scripts/log2timeline.py --partitions all --vss_stores=none --filter-file ./l2t_filter_mft.yaml --storage_file ./output/debug.plaso ./my_data/in_a_dir/cfreds.dd
./scripts/pinfo.py ./output/debug.plaso --verbose

Pinfo gives:

************************** Plaso Storage Information ***************************
            Filename : debug.plaso
      Format version : 20230327
Serialization format : json
--------------------------------------------------------------------------------

*********************************** Sessions ***********************************
5472a64d-56b3-4281-871a-e94a2a7c80a6 : 2024-12-02T14:36:27.818708+00:00
--------------------------------------------------------------------------------

**************** Session: 5472a64d-56b3-4281-871a-e94a2a7c80a6 *****************
                Start time : 2024-12-02T14:36:27.818708+00:00
           Completion time : 2024-12-02T14:36:38.659266+00:00
              Product name : plaso
           Product version : 20240826
    Command line arguments : ./scripts/log2timeline.py --partitions all
                             --vss_stores=none --filter-file
                             ./l2t_filter_mft.yaml --storage_file
                             ./output/debug.plaso ./my_data/in_a_dir/cfreds.dd
  Parser filter expression : win7
Enabled parser and plugins : bencode/bencode_transmission,
                             bencode/bencode_utorrent, binary_cookies,
                             chrome_cache, chrome_preferences,
                             custom_destinations, czip/oxml,
                             esedb/file_history, esedb/msie_webcache,
                             esedb/user_access_logging, filestat,
                             firefox_cache, java_idx, lnk, mcafee_protection,
                             msiecf, olecf/olecf_automatic_destinations,
                             olecf/olecf_default, olecf/olecf_document_summary,
                             olecf/olecf_summary, opera_global,
                             opera_typed_history, pe, plist/safari_history,
                             prefetch, recycle_bin, sqlite/chrome_17_cookies,
                             sqlite/chrome_27_history,
                             sqlite/chrome_66_cookies, sqlite/chrome_8_history,
                             sqlite/chrome_autofill,
                             sqlite/chrome_extension_activity,
                             sqlite/firefox_10_cookies,
                             sqlite/firefox_2_cookies,
                             sqlite/firefox_downloads, sqlite/firefox_history,
                             sqlite/google_drive, sqlite/safari_historydb,
                             sqlite/skype, symantec_scanlog,
                             text/gdrive_synclog, text/powershell_transcript,
                             text/sccm, text/setupapi, text/skydrive_log_v1,
                             text/skydrive_log_v2,
                             text/teamviewer_application_log,
                             text/teamviewer_connections_incoming,
                             text/teamviewer_connections_outgoing,
                             text/winfirewall, usnjrnl, winevtx, winjob,
                             winpca_db0, winpca_dic, winreg/amcache,
                             winreg/appcompatcache, winreg/bagmru, winreg/bam,
                             winreg/ccleaner, winreg/explorer_mountpoints2,
                             winreg/explorer_programscache,
                             winreg/microsoft_office_mru,
                             winreg/microsoft_outlook_mru,
                             winreg/mrulist_shell_item_list,
                             winreg/mrulist_string,
                             winreg/mrulistex_shell_item_list,
                             winreg/mrulistex_string,
                             winreg/mrulistex_string_and_shell_item,
                             winreg/mrulistex_string_and_shell_item_list,
                             winreg/msie_zone, winreg/mstsc_rdp,
                             winreg/mstsc_rdp_mru, winreg/network_drives,
                             winreg/networks, winreg/userassist,
                             winreg/windows_boot_execute,
                             winreg/windows_boot_verify, winreg/windows_run,
                             winreg/windows_sam_users, winreg/windows_services,
                             winreg/windows_shutdown,
                             winreg/windows_task_cache,
                             winreg/windows_timezone,
                             winreg/windows_typed_urls,
                             winreg/windows_usb_devices,
                             winreg/windows_usbstor_devices,
                             winreg/windows_version, winreg/winlogon,
                             winreg/winrar_mru, winreg/winreg_default
        Preferred encoding : UTF-8
       Preferred time zone : UTC
                Debug mode : False
          Artifact filters : N/A
               Filter file : N/A
--------------------------------------------------------------------------------

********** System configuration: 5472a64d-56b3-4281-871a-e94a2a7c80a6 **********
                Hostname : INFORMANT-PC
        Operating system : Windows NT
Operating system product : Windows 7 Ultimate
Operating system version : 6.1
                Language : en-US
               Code page : cp1252
         Keyboard layout : N/A
               Time zone : America/New_York
--------------------------------------------------------------------------------

...

************************* Events generated per parser **************************
Parser (plugin) name : Number of events
--------------------------------------------------------------------------------
            filestat : 8
               Total : 8
--------------------------------------------------------------------------------

So, no mft events (as it is absent in the parser list), an unprovided "Parser filter expression", and surprisingly (yet it is correctly handled), Pinfo thinks there is no "Filter file".

Now, the same run with the mft parser provided:

./scripts/log2timeline.py --partitions all --vss_stores=none --filter-file ./l2t_filter_mft.yaml --storage_file ./output/debug-with-mft.plaso ./my_data/in_a_dir/cfreds.dd --parsers mft

./scripts/pinfo.py ./output/debug-with-mft.plaso --verbose

*********************************** Sessions ***********************************
...
  Command line arguments : ./scripts/log2timeline.py --partitions all
                             --vss_stores=none --filter-file
                             ./l2t_filter_mft.yaml --storage_file
                             ./output/debug-with-mft.plaso
                             ./my_data/in_a_dir/cfreds.dd --parsers mft
  Parser filter expression : mft
  Enabled parser and plugins : mft
...

************************* Events generated per parser **************************
Parser (plugin) name : Number of events
--------------------------------------------------------------------------------
                 mft : 933089
               Total : 933089
--------------------------------------------------------------------------------

It correctly finds mft events.

Now this is what confused me, using the test_data/MFT file from Plaso's dataset (and without any parser filter):

./scripts/log2timeline.py --partitions all --vss_stores=none --filter-file ./l2t_filter_mft.yaml --storage_file ./output/file-no-mft.plaso ./plaso-20240826/test_data/MFT

And pinfo output:

**************** Session: 2197f23b-a6c2-43e3-bd04-f066b27842a1 *****************
                Start time : 2024-12-02T14:50:49.603078+00:00
           Completion time : 2024-12-02T14:50:56.436049+00:00
              Product name : plaso
           Product version : 20240826
    Command line arguments : ./scripts/log2timeline.py --partitions all
                             --vss_stores=none --filter-file
                             ./l2t_filter_mft.yaml --storage_file
                             ./output/file-no-mft.plaso
                             ./plaso-20240826/test_data/MFT
  Parser filter expression : N/A
Enabled parser and plugins : android_app_usage, asl_log, bencode,
                             bencode/bencode_transmission,
                             bencode/bencode_utorrent, binary_cookies,
                             bodyfile, bsm_log, chrome_cache,
                             chrome_preferences, cups_ipp, custom_destinations,
                             czip, czip/oxml, esedb, esedb/file_history,
                             esedb/msie_webcache, esedb/srum,
                             ...

************************* Events generated per parser **************************
Parser (plugin) name : Number of events
--------------------------------------------------------------------------------
            filestat : 3
                 mft : 126349
               Total : 126352
--------------------------------------------------------------------------------

Now this time, the mft parser is turned on by default. So, we kinda have an inconsistent behavior depending on the type of data. Or maybe, it should be made more explicit in the command line help?

I tried to add a dirty and False at the following line to disable the preset detection and I got the expected behavior.

Let me know what you think about it. I'm even fine coding the feature if you would like. Hopefully, I didn't missed any existing feature/option.

EDIT: I should add that this issue is not only about Windows or MFT. If a Linux server hosts EVTX files for archives, I want the winevtx parser to be activated even if those events don't belongs to the server.

Looking forward to read you, thanks.