Resolve #11741, bbot update based on testing using BBOT v2.1.2 #11742

colin-stubbs · 2024-11-14T13:36:55Z

Enhancement

Proposed commit message

This commit adds BBOT v2.x NDJSON format support the integration, while simultaneously supporting the existing BBOT v1.x format.

It also adds a HTTP input configuration option to support webhook/NDJSON style logging over HTTP or HTTPS via Elastic Agent inputs.

Enhancements to ingest pipelines, as well as pipeline and system tests have also been made.

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.
I have verified that any added dashboard complies with Kibana's Dashboard good practices

Author's Checklist

Pipeline and system tests complete
Manual testing from live bbot system to production system using HTTP output/input methods captured all of the same events that were logged to file

How to test this PR locally

elastic-agent test

Related issues

Closes Update bbot integration to support recent major v2.x release #11741

Screenshots

Not applicable.

+- version: "1.0.2"
+  changes:
+    - description: Add integration screenshots
+      type: enhancement
+      link: https://github.com/elastic/integrations/pull/11742
+- version: "1.0.1"
+  changes:
+    - description: Tests use BBOT v2.1.2 NDJSON output
+      type: enhancement
+      link: https://github.com/elastic/integrations/pull/11742
+    - description: Update to support BBOT v2.0.0 event format
+      type: enhancement
+      link: https://github.com/elastic/integrations/pull/11742
+    - description: Add http_endpoint input option to support BBOT HTTP output module
+      type: enhancement
+      link: https://github.com/elastic/integrations/pull/11742


Suggested change

- version: "1.0.2"

changes:

- description: Add integration screenshots

type: enhancement

link: https://github.com/elastic/integrations/pull/11742

- version: "1.0.1"

changes:

- description: Tests use BBOT v2.1.2 NDJSON output

type: enhancement

link: https://github.com/elastic/integrations/pull/11742

- description: Update to support BBOT v2.0.0 event format

type: enhancement

link: https://github.com/elastic/integrations/pull/11742

- description: Add http_endpoint input option to support BBOT HTTP output module

type: enhancement

link: https://github.com/elastic/integrations/pull/11742

- version: "1.1.0"

changes:

- description: Update to support BBOT v2.0.0 event format.

type: enhancement

link: https://github.com/elastic/integrations/pull/11742

- description: Add http_endpoint input option to support BBOT HTTP output module.

type: enhancement

link: https://github.com/elastic/integrations/pull/11742

I don't think we need to mention the addition of screenshots.

The changelog is user-facing, so we don't need to mention the testing changes.

This is an enhancement, so bump minor (semver semantics).

efd6 · 2024-11-17T22:09:42Z

packages/bbot/manifest.yml

@@ -1,7 +1,7 @@
 format_version: 3.1.2
 name: bbot
 title: "BBOT (Bighuge BLS OSINT Tool)"
-version: "1.0.0"
+version: "1.0.2"


Suggested change

version: "1.0.2"

version: "1.1.0"

efd6 · 2024-11-17T22:10:30Z

packages/bbot/_dev/build/docs/README.md

-This tool is used to enhance your external knowledge of your environment. This is done through the integration of many tools into BBOT providing a overview of your attack surface. Here is [how it works](https://www.blacklanternsecurity.com/bbot/Stable/how_it_works/).
+This integration requires the external use of BBOT. You will have to download and run the tool apart from this integration. Once your scan is complete, this integration can ingest the results into Elastic using the BBOT `json` output module or the `http` output module aka. HTTP POST/webhooks.
+
+As of version `0.3.0` of this integration both BBOT `1.x` and `2.x` are supported. However all examples provided by the module documentation now assume use of BBOT `2.x`.


efd6 · 2024-11-17T22:11:49Z

packages/bbot/_dev/build/docs/README.md

+As of version `0.3.0` of this integration both BBOT `1.x` and `2.x` are supported. However all examples provided by the module documentation now assume use of BBOT `2.x`.
+
+In order to support the conflicting BBOT NDJSON event formats the following occurs during ingest of BBOT `2.x` SCAN events:
+1. The `bbot.data.scan` field, if received as a Map aka. object, is renamed to `bbot.data.scan_config`


Suggested change

1. The `bbot.data.scan` field, if received as a Map aka. object, is renamed to `bbot.data.scan_config`

1. The `bbot.data.scan` field, if received as an object, is renamed to `bbot.data.scan_config`

efd6 · 2024-11-17T22:12:47Z

packages/bbot/_dev/deploy/docker/sample_logs/bbot-v1-webhook.log

+{"type": "DNS_NAME", "id": "DNS_NAME:f57ba0828becd7bf94faa616db081ed06f31bd3d", "data": {"DNS_NAME": "example.com"}, "scope_distance": 0, "scan": "SCAN:725368977d3a680e579707504e59428a7e3acc9d", "timestamp": 1709170932.40485, "resolved_hosts": ["1.128.0.0"], "source": "IP_ADDRESS:605bb7cd2537bcf0fd94ce6b120bfb56cf3e0753", "tags": ["resolved", "a-record", "in-scope", "subdomain"], "module": "PTR", "module_sequence": "PTR"}
+{"type": "IP_ADDRESS", "id": "IP_ADDRESS:605bb7cd2537bcf0fd94ce6b120bfb56cf3e0753", "data": {"IP_ADDRESS": "1.128.0.0"}, "scope_distance": 1, "scan": "SCAN:725368977d3a680e579707504e59428a7e3acc9d", "timestamp": 1709170932.033501, "resolved_hosts": ["1.128.0.0"], "source": "DNS_NAME:f57ba0828becd7bf94faa616db081ed06f31bd3d", "tags": ["ipv4", "resolved", "distance-1", "internal", "ptr-record"], "module": "A", "module_sequence": "A"}
+{"type": "DNS_NAME", "id": "DNS_NAME:32102fa4a314358e17370aaec3b0f32641dd9ab3", "data": {"DNS_NAME": "example.com"}, "scope_distance": 1, "scan": "SCAN:725368977d3a680e579707504e59428a7e3acc9d", "timestamp": 1709170933.584077, "resolved_hosts": ["1.128.0.0"], "source": "OPEN_TCP_PORT:62743609c2dcda87da01a4faa9ddd1b7b362939f", "tags": ["soa-record", "affiliate", "resolved", "distance-1", "domain", "a-record", "txt-record", "mx-record", "ns-record"], "module": "sslcert", "module_sequence": "sslcert"}
+{"type": "AZURE_TENANT", "id": "AZURE_TENANT:db22e942c8a51294e1abc874daa076caecccbf01", "data": {"AZURE_TENANT": {"tenant-names": ["exampletenant"], "domains": ["example.smtp.online", "example.com", "example1.com", "example.org", "example2.com", "example.gov", "example3.com"], "tenant-id": "70af547c-69ab-416d-b4a6-543b5ce52b99"}}, "scope_distance": 1, "scan": "SCAN:725368977d3a680e579707504e59428a7e3acc9d", "timestamp": 1709170932.947014, "source": "DNS_NAME:f57ba0828becd7bf94faa616db081ed06f31bd3d", "tags": ["distance-1"], "module": "azure_tenant", "module_sequence": "azure_tenant"}


Final new line.

efd6 · 2024-11-17T22:25:41Z

packages/bbot/data_stream/asm_intel/fields/ecs.yml

Are there specific fields that need to be included here to prevent mismapping? If not, this does not need to be present.

efd6 · 2024-11-17T22:26:22Z

packages/bbot/data_stream/asm_intel/elasticsearch/ingest_pipeline/ecs.yml

+
+# EOF


Suggested change

# EOF

efd6 · 2024-11-17T22:29:48Z

packages/bbot/data_stream/asm_intel/elasticsearch/ingest_pipeline/default.yml

+
+- set:
+    description: "Set bbot.data.SCAN based on bbot.data.SCAN_CONFIG to emulate BBOT v1.x field type"
+    if: ctx.bbot?.data?.SCAN == null && ctx.bbot?.data?.SCAN_CONFIG?.name != null && ctx.bbot?.data?.SCAN_CONFIG?.id != null


Suggested change

if: ctx.bbot?.data?.SCAN == null && ctx.bbot?.data?.SCAN_CONFIG?.name != null && ctx.bbot?.data?.SCAN_CONFIG?.id != null

if: ctx.bbot?.data?.SCAN == null && ctx.bbot?.data?.SCAN_CONFIG?.name != null && ctx.bbot.data.SCAN_CONFIG?.id != null

Already known to be non-null due to the second expression.

efd6 · 2024-11-17T22:30:22Z

packages/bbot/data_stream/asm_intel/elasticsearch/ingest_pipeline/default.yml

+        - bbot.data.vulnerability.severity
+        - bbot.data.dns_name
+        - bbot.resolved_hosts
+    if: "ctx.tags == null || !(ctx.tags.contains('preserve_duplicate_custom_fields'))"


Suggested change

if: "ctx.tags == null || !(ctx.tags.contains('preserve_duplicate_custom_fields'))"

if: ctx.tags == null || !(ctx.tags.contains('preserve_duplicate_custom_fields'))

efd6 · 2024-11-17T22:30:32Z

packages/bbot/data_stream/asm_intel/elasticsearch/ingest_pipeline/default.yml

+- remove:
+    description: "Remove event.original unless tags indicate not to"
+    field: event.original
+    if: "ctx.tags == null || !(ctx.tags.contains('preserve_original_event'))"


Suggested change

if: "ctx.tags == null || !(ctx.tags.contains('preserve_original_event'))"

if: ctx.tags == null || !(ctx.tags.contains('preserve_original_event'))

elasticmachine · 2024-11-18T16:17:24Z

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

Update based on testing using BBOT v2.1.2

95bcd59

colin-stubbs requested a review from a team as a code owner November 14, 2024 13:36

colin-stubbs mentioned this pull request Nov 14, 2024

BBOT v1.x vs v2.x SIEM Friendly JSON blacklanternsecurity/bbot#1672

Open

colin-stubbs added 2 commits November 15, 2024 07:23

update PR number now that we have one

34bf979

add integration screenshots, increment version

e1a17f3

efd6 reviewed Nov 17, 2024

View reviewed changes

andrewkroh added enhancement New feature or request Integration:bbot BBOT (Bighuge BLS OSINT Tool) Team:Security-Service Integrations Security Service Integrations Team [elastic/security-service-integrations] labels Nov 18, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Resolve #11741, bbot update based on testing using BBOT v2.1.2 #11742

Resolve #11741, bbot update based on testing using BBOT v2.1.2 #11742

colin-stubbs commented Nov 14, 2024

efd6 Nov 17, 2024

efd6 Nov 17, 2024

efd6 Nov 17, 2024

efd6 Nov 17, 2024

efd6 Nov 17, 2024

efd6 Nov 17, 2024

efd6 Nov 17, 2024

efd6 Nov 17, 2024

efd6 Nov 17, 2024

efd6 Nov 17, 2024

elasticmachine commented Nov 18, 2024

	1. The `bbot.data.scan` field, if received as a Map aka. object, is renamed to `bbot.data.scan_config`
	1. The `bbot.data.scan` field, if received as an object, is renamed to `bbot.data.scan_config`

	if: ctx.bbot?.data?.SCAN == null && ctx.bbot?.data?.SCAN_CONFIG?.name != null && ctx.bbot?.data?.SCAN_CONFIG?.id != null
	if: ctx.bbot?.data?.SCAN == null && ctx.bbot?.data?.SCAN_CONFIG?.name != null && ctx.bbot.data.SCAN_CONFIG?.id != null

	if: "ctx.tags == null \|\| !(ctx.tags.contains('preserve_duplicate_custom_fields'))"
	if: ctx.tags == null \|\| !(ctx.tags.contains('preserve_duplicate_custom_fields'))

Resolve #11741, bbot update based on testing using BBOT v2.1.2 #11742

Are you sure you want to change the base?

Resolve #11741, bbot update based on testing using BBOT v2.1.2 #11742

Conversation

colin-stubbs commented Nov 14, 2024

Proposed commit message

Checklist

Author's Checklist

How to test this PR locally

Related issues

Screenshots

See Also

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

elasticmachine commented Nov 18, 2024