diff --git a/gdi/opentelemetry/otel-other/otel-other-landing.rst b/gdi/opentelemetry/otel-other/otel-other-landing.rst index 0137a1661..f6752d0d4 100644 --- a/gdi/opentelemetry/otel-other/otel-other-landing.rst +++ b/gdi/opentelemetry/otel-other/otel-other-landing.rst @@ -14,8 +14,10 @@ Other OpenTelemetry ingestion methods prometheus-generic telegraf + other-ingestion-collectd On top of the available native :ref:`OpenTelemetry receivers `, you can also send data to Splunk Observability Cloud with OpenTelemetry with the following options: * :ref:`prometheus-generic` -* :ref:`telegraf-generic` \ No newline at end of file +* :ref:`telegraf-generic` +* :ref:`other-ingestion-collectd` \ No newline at end of file diff --git a/gdi/opentelemetry/otel-other/other-ingestion-collectd.rst b/gdi/opentelemetry/otel-other/other-ingestion-collectd.rst new file mode 100644 index 000000000..c71039277 --- /dev/null +++ b/gdi/opentelemetry/otel-other/other-ingestion-collectd.rst @@ -0,0 +1,171 @@ +.. _other-ingestion-collectd: + +Monitor hosts with collectd and OpenTelemetry +===================================================================== + +.. meta:: + :description: Use collectd and native OpenTelemetry to monitor services in Splunk Observability Cloud. See benefits, install, configuration, and metrics. + +To monitor your infrastructure with collectd using native OpenTelemetry in Splunk Observability Cloud, install a collectd daemon in your host and connect it to your Collector instance as described in this document. + +Benefits +-------- + +.. raw:: html + +
+ +.. include:: /_includes/benefits.rst + +.. raw:: html + +
+ + +Configuration +---------------------------------- + +Install a collectd daemon in your host and connect it to an OpenTelemetry Collector with the following steps: + +1. Install and configure collectd +2. Configure the OpenTelemetry Collector +3. Build and run + +1. Install and configure collectd +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Follow these steps to install and configure the collectd daemon: + +#. Install collectd as a Debian or Yum package in your host +#. Configure the daemon to ingest free disk related metrics through `collectd/metrics.conf` +#. Configure the daemon to send data over HTTP using `collectd/http.conf` + +In this example, the host is represented by an Ubuntu 24.04 docker image. + +.. code:: + + services: + collectd: + build: collectd + container_name: collectd + depends_on: + - otelcollector + volumes: + - ./collectd/http.conf:/etc/collectd/collectd.conf.d/http.conf + - ./collectd/metrics.conf:/etc/collectd/collectd.conf.d/metrics.conf + + # OpenTelemetry Collector + otelcollector: + image: quay.io/signalfx/splunk-otel-collector:latest + container_name: otelcollector + command: ["--config=/etc/otel-collector-config.yml", "--set=service.telemetry.logs.level=debug"] + volumes: + - ./otel-collector-config.yml:/etc/otel-collector-config.yml + +The http and metrics configuration files look like this: + +.. code:: yaml + + # http.conf + # The minimal configuration required to have collectd send data to an OpenTelemetry Collector + # with a collectdreceiver deployed on port 8081. + + LoadPlugin write_http + + + URL "http://otelcollector:8081" + Format JSON + VerifyPeer false + VerifyHost false + + + +.. code:: yaml + + # metrics.conf + # An example of collectd plugin configuration reporting free disk space on the host. + + + Interval 3600 + + + ValuesPercentage true + + +1. Configure the OpenTelemetry Collector +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Set up your Collector instance to listen for traffic from the collectd daemon over HTTP with the :ref:`collectd-receiver`: + +.. code:: yaml + + receivers: + collectd: + endpoint: "0.0.0.0:8081" + + exporters: + debug: + verbosity: detailed + + service: + pipelines: + metrics: + receivers: [collectd] + exporters: [debug] + +.. caution:: Make sure to use ``0.0.0.0`` to expose port 8081 over the Docker network interface so that both Docker containers can interact. + +3. Build and run +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Run the example with the instruction to start the docker-compose setup and build the collectd container: + +.. code:: bash + + $> docker compose up --build + +Check that the Collector is receiving metrics and logging them to ``stdout`` via the debug exporter: + +.. code:: bash + + $> docker logs otelcollector + +A typical output is: + +.. code:: + + StartTimestamp: 1970-01-01 00:00:00 +0000 UTC + Timestamp: 2024-12-20 19:55:44.006000128 +0000 UTC + Value: 38.976566 + Metric #17 + Descriptor: + -> Name: percent_bytes.reserved + -> Description: + -> Unit: + -> DataType: Gauge + NumberDataPoints #0 + Data point attributes: + -> plugin: Str(df) + -> plugin_instance: Str(etc-hosts) + -> host: Str(ea1d62c7a229) + -> dsname: Str(value) + StartTimestamp: 1970-01-01 00:00:00 +0000 UTC + Timestamp: 2024-12-20 19:55:44.006000128 +0000 UTC + Value: 5.102245 + {"kind": "exporter", "data_type": "metrics", "name": "debug"} + +Troubleshooting +--------------- + +.. raw:: html + +
+ +.. include:: /_includes/troubleshooting-components.rst + +.. raw:: html + +
+ + + diff --git a/gdi/opentelemetry/otel-other/telegraf.rst b/gdi/opentelemetry/otel-other/telegraf.rst index 3f466b0a0..c76a523f7 100644 --- a/gdi/opentelemetry/otel-other/telegraf.rst +++ b/gdi/opentelemetry/otel-other/telegraf.rst @@ -5,7 +5,7 @@ Monitor services with Telegraf Input plugins and OpenTelemetry ===================================================================== .. meta:: - :description: Use this Splunk Observability Cloud integration for the Telegraf monitor. See benefits, install, configuration, and metrics. + :description: Use Telegraf and OpenTelemetry to monitor your services in Splunk Observability Cloud. See benefits, install, configuration, and metrics. To monitor your service with Telegraf using native OpenTelemetry in Splunk Observability Cloud, install the service's Telegraf Input plugin then push metrics to the Splunk Opentelemetry Collector via OTLP. @@ -39,7 +39,7 @@ Follow these steps to scrape Telegraf metrics with the OTel Collector: 3. Set up the Telegraf OpenTelemetry Output plugin 4. Configure the OpenTelemetry Collector -1. Install Telegraf +5. Install Telegraf ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Run the following commands to install Telegraf from the InfluxData repository: diff --git a/sp-oncall/spoc-integrations/sensu-integration.rst b/sp-oncall/spoc-integrations/sensu-integration.rst index c08f0a2c9..576bbe3f2 100644 --- a/sp-oncall/spoc-integrations/sensu-integration.rst +++ b/sp-oncall/spoc-integrations/sensu-integration.rst @@ -138,5 +138,5 @@ Splunk On-Call Handler end end -For more information, see :new-page:`Sensu documentation on Handlers `. +For more information, see search "handlers" on the Sensu documentation website. diff --git a/synthetics/key-concepts.rst b/synthetics/key-concepts.rst index 8f2cbdb96..8b6a0f5fa 100644 --- a/synthetics/key-concepts.rst +++ b/synthetics/key-concepts.rst @@ -22,43 +22,45 @@ These key concepts help you get the most out of your experience with Splunk Synt * - Synthetics - A common abbreviation for Synthetic Monitoring. - * - Tests - - Tests are the primary mechanism of application monitoring in Splunk Synthetic Monitoring. You can set up Browser, Uptime, and API tests to run at your preferred frequency from the devices and public locations of your choosing. + * - Test + - An ordered sequence of actions or operations to perform on a target URL or endpoint, similar to a script. A test is the primary mechanism for monitoring applications in Splunk Synthetic Monitoring. You can set up browser, uptime, and API tests to run at your preferred frequency from the devices and locations of your choosing. - * - Runs - - Each occurrence of a test from a particular device and location at a specific time is called a run. Each run of a test captures a set of metrics and diagnostics that provide insight into your application's performance. + * - Run + - An instance of a test invocation from a specific device and location and at a specific time. - * - Metrics - - | Numeric indicators of site performance that synthetic tests capture in each run of a test. See the following links for the lists of metrics available for each test type: + * - Metric + - A numeric indicator of a performance factor. Each run of a test captures metrics and diagnostics that provide insight into your application's performance. See the following links for details on the metrics available captured by each test type: | | - :ref:`browser-metrics` | - :ref:`uptime-metrics` | - :ref:`api-test-metrics` | - * - Browser tests - - Synthetic tests that simulate and analyze the user experience of loading a page or performing a workflow on a site or application. + * - Browser test + - A synthetic test that simulates and analyzes the user experience of loading a page or performing a workflow on a site or application. - * - Uptime tests - - Synthetic tests that monitor the response time and response code of HTTP requests and server ports. + * - Uptime test + - A synthetic test that monitors the response times and response codes of HTTP requests and server ports. - * - API tests - - Synthetic tests that monitor the functionality and performance of API endpoints. + * - API test + - A synthetic test that monitors the functionality and performance of API endpoints. - * - Waterfall charts - - A visualization that represents the performance of resources on a webpage in a Browser test. + * - Waterfall chart + - A visualization that represents the performance of resources on a webpage in a browser test. - * - Devices - - Options to configure the viewport and network connection type of your tests. For example, you can test a webpage from a laptop screen or a smartphone. + * - Device + - A predefined combination of a specific viewport size and network connection. For more information about devices, viewports, and network connections, see :ref:`devices`. - * - Viewport - - Browser tests in Splunk Synthetic Monitoring capture the visual experience of a user interacting with your application. The viewport is the framed area on a device's screen for viewing information, such as the browser window on a desktop. When you set up a test, you can choose the viewport size from a list of common devices, or set a custom viewport by height and width. + * - Public location + - A public IP address from which you can run synthetic tests to simulate performance for users in that location. See :ref:`public-locations`. + + * - Private location + - A name you create in Splunk Synthetic Monitoring to represent a custom location from which you can run synthetic tests. The name you give to a private location allows you to specify that name in a synthetic test's :guilabel:`Locations` field. You must also set up one or more private runners within every private location to do the actual communication with your targets and with Splunk Synthetic Monitoring. You can use a private location to test an internal endpoint or to test a public endpoint from a location that isn't included in :ref:`the list of Splunk Synthetic Monitoring public locations `. See :ref:`private-locations`. + + * - Private runner + - A private runner queries Splunk Synthetic Monitoring for tests configured to run in its inherent private location, performs the test's steps on your private target, and reports the results back to Splunk Synthetic Monitoring. Because a private runner must have access to your private target, it is a Docker image which you deploy on your own infrastructure, within your own internal network. See :ref:`private-locations`. - * - Network connection - - You can run Browser, Uptime or API tests to simulate network connections of various latencies, including Mobile LTE, Mobile 3G, DSL, and cable internet. Testing your site from a variety of connection types lets you monitor the experience of users in a variety of settings. - * - Public Locations - - Global checkpoints from which you can synthetically run tests to simulate performance for users in that location. Learn more =========== diff --git a/synthetics/test-config/private-locations.rst b/synthetics/test-config/private-locations.rst index 60918a1af..9a8479995 100644 --- a/synthetics/test-config/private-locations.rst +++ b/synthetics/test-config/private-locations.rst @@ -7,31 +7,33 @@ Private locations .. meta:: :description: Run synthetic tests from private locations such as internal sites, private web applications, or private networks. -A private location is a software package that offers a quick and easy deployment of Splunk Synthetic Monitoring solutions beyond the public network so that you can find, fix, and prevent web performance defects on any internal web application, in any environment - whether inside or outside of your firewalls. Private locations allow Splunk Synthetics Monitoring users to test sooner in the development cycle and against internal sites or applications that aren't available to the public. +A private location is a name you create in Splunk Synthetic Monitoring to represent a custom location from which you can run synthetic tests. The name you give to a private location allows you to specify that name in a synthetic test's :guilabel:`Locations` field. To run synthetic tests from private locations, you must also set up one or more private runners within the private location to do the actual communication with your test targets and with Splunk Synthetic Monitoring. -Customers can, through the Splunk Synthetics Monitoring web interface, create new private locations and open a runner to run any checks assigned to them. -What is a runner? -=================== - -A runner is a Docker container set up to run tests from a particular private location. A single private location can have one or more runners. - -A location consists of a queue of tests assigned to a particular private location. Runners pick up runs from the queue, so the more active runners you have, the faster the queue of tests is processed. +Use cases for private locations +==================================== -Splunk Synthetic Monitoring doesn't track how many runners there are for a given location. It is up to you to manage your own fleet of runners. +Private locations provide a way for you to find, fix, and prevent performance problems in internal applications in any environment, inside or outside of your firewalls. You can use private locations to run tests earlier in your development cycle against internal sites or applications that aren't available to the public. You can also use private locations to test public endpoints from locations that aren't included in the :ref:`list of Splunk Synthetic Monitoring public locations `. - -Use cases for private locations -================================= +To summarize, here are sample use cases for private locations: * Test private applications that aren't exposed to the public. * Test pre-production applications which don't have public staging sites. -* Gain a higher level of flexibility in giving Splunk Synthetic Monitoring access to applications. +* Gain a higher level of flexibility by giving Splunk Synthetic Monitoring access to applications. * Test from locations not currently supported by Splunk Synthetic Monitoring's public locations. -Requirements -============= +What is a private runner? +================================ + +A private runner is a Docker image which you deploy on your own infrastructure, within your own internal network. It picks up test runs from the queue of tests assigned to its associated private location, performs the actions in the test run on the test target, and reports the results back to Splunk Synthetic Monitoring. + + +If you create multiple private runners for a private location, they can process that location's test queue faster. Splunk Synthetic Monitoring doesn't track how many private runners there are for a given private location. It's up to you to manage your own fleet of private runners. + + +Requirements for private runners +========================================== .. list-table:: @@ -55,7 +57,7 @@ Requirements - Linux, Windows, or macOS -For optimal performance when running Browser tests: +For optimal performance when running browser tests: * Linux * 2.3 GHz Dual-Core Intel Xeon (or equivalent) processor @@ -63,13 +65,13 @@ For optimal performance when running Browser tests: Set up a new private location -=============================== +===================================== Follow these steps to set up a new private location: 1. In Splunk Synthetic Monitoring, select the settings gear icon, then :guilabel:`Private locations`. 2. Select :guilabel:`+ Add` and add a name. -3. Follow the steps in the guided setup to set up your runner. +3. Follow the steps in the guided setup to set up one or more private runners for that private location. 4. Save your private location. @@ -139,62 +141,60 @@ For example, here is what a command might look like after you modify it to fit y -Configuring Proxy Settings for Private Locations +Configure proxy settings for a private location =================================================== In environments where direct internet access is restricted, you can route synthetic test traffic through a proxy server by configuring the following environment variables: -* ``HTTP_PROXY``: Specifies the proxy server for HTTP traffic. +* ``http_proxy``: Specifies the proxy server for HTTP traffic. - * Example: ``export HTTP_PROXY="\http://proxy.example.com:8080"`` + * Example: ``export http_proxy="http://proxy.example.com:8443"`` -* ``HTTPS_PROXY``: Specifies the proxy server for HTTPS traffic. +* ``https_proxy``: Specifies the proxy server for HTTPS traffic. - * Example: ``export HTTPS_PROXY="\https://proxy.example.com:8443"`` + * Example: ``export https_proxy="http://proxy.example.com:8443"`` -* ``NO_PROXY``: Specifies a comma-separated list of domains or IP addresses that should bypass the proxy. +* ``no_proxy``: Specifies a comma-separated list of domains or IP addresses that should bypass the proxy. - * Example: ``export NO_PROXY="localhost,127.0.0.1,.internal-domain.com"`` + * Example: ``export no_proxy="localhost,127.0.0.1,.internal-domain.com"`` For example, here is what a command might look like after you modify it to fit your environment: .. code:: yaml - docker run --cap-add NET_ADMIN -e "RUNNER_TOKEN=*****" quay.io/signalfx/splunk-synthetics-runner:latest -e NO_PROXY=".signalfx.com,.amazonaws.com" -e HTTPS_PROXY="https://172.17.0.1:1234" -e HTTP_PROXY="http://172.17.0.1:1234" + docker run --cap-add NET_ADMIN -e "RUNNER_TOKEN=*****" -e "no_proxy=.signalfx.com,.amazonaws.com,127.0.0.1,localhost" -e "https_proxy=http://172.17.0.1:1234" -e "http_proxy=http://172.17.0.1:1234" quay.io/signalfx/splunk-synthetics-runner:latest + In this example: -``HTTP_PROXY`` and ``HTTPS_PROXY`` are set to route traffic through a proxy at ``http://172.17.0.1:1234``. +``http_proxy`` and ``https_proxy`` are set to route traffic through a proxy at ``http://172.17.0.1:1234``. -``NO_PROXY`` is configured to bypass the proxy for local addresses and specific domains like .signalfx.com and .amazonaws.com. +``no_proxy`` is configured to bypass the proxy for local addresses and specific domains like .signalfx.com and .amazonaws.com. Ensure that these variables are correctly configured to comply with your network policies. This setup allows the synthetic tests to communicate securely and efficiently in a controlled network environment. -When using runner, it's important to correctly configure the proxy settings to avoid issues with browser-based tests. The following steps should be followed when setting up their environment: +When using a private runner, it's important to correctly configure the proxy settings to avoid issues with browser-based tests. The following steps should be followed when setting up their environment: -1. :strong:`Ensure proper NO_PROXY setup`: +1. :strong:`Ensure proper no_proxy setup`: - - When configuring ``NO_PROXY`` always include the following addresses: + - When configuring ``no_proxy`` always include the following addresses: - ``127.0.0.1`` (for localhost communication) - ``localhost`` (for resolving local tests) These addresses ensure that internal services and tests run correctly without routing through a proxy, preventing potential failures. -2. :strong:`Merging HTTP_PROXY and http_proxy`: - - - The system automatically handles both ``HTTP_PROXY`` and ``http_proxy`` environment variables. If you define one of these, ensure the other is also set, or they will be automatically merged at start-up. - 3. :strong:`Dockerfile defaults`: - - By default, the runner will set the ``NO_PROXY`` variable in the Dockerfile to include ``127.0.0.1``. If you override ``NO_PROXY``, you must ensure that ``127.0.0.1`` and ``localhost`` are still present, or browser tests may fail. - + - By default, the private runner sets the ``no_proxy`` variable in the Dockerfile to include ``127.0.0.1``. If you override ``no_proxy``, you must ensure that ``127.0.0.1`` and ``localhost`` are still present, or browser tests may fail. +.. note:: + Lower case variable names take precedence and are best practice. -Assess the health of your private location +Assess the health of a private location ============================================== A private location's health depends on three factors: