Merge pull request #494 from splunk/develop

Release/8.0.4.1 and 7.3.6

CODEOWNERS

@@ -11,3 +11,6 @@
 
 # Docs-only pull requests:
 /docs/  @alishamayor @nwang92 @bb03
+
+# Release changelog
+docs/CHANGELOG.md  @nwang92 @alishamayor @arctan5x @lephino @jrigassio-splunk @jmeixensperger @hendolim @jonathan-vega-splunk @bb03

docs/ADVANCED.md

@@ -10,6 +10,7 @@
 * [Apps](#apps)
 * [SmartStore](#smartstore)
 * [Custom splunk-launch.conf](#custom-splunk-launchconf)
+* [Multi-cluster Search](#multi-cluster-search)
 
 ---
 
@@ -110,6 +111,7 @@ Splunk-Ansible ships with an inventory script in `inventory/environ.py`. The scr
 | SPLUNK_ANSIBLE_PRE_TASKS | Pass in a comma-separated list of local paths or remote URLs to Ansible playbooks that will be executed before `site.yml` | no | no | no |
 | SPLUNK_ANSIBLE_POST_TASKS | Pass in a comma-separated list of local paths or remote URLs to Ansible playbooks that will be executed after `site.yml` | no | no | no |
 | SPLUNK_ANSIBLE_ENV | Pass in a comma-separated list of "key=value" pairs that will be mapped to environment variables used during `site.yml` execution | no | no | no |
+| SPLUNK_CONNECTION_TIMEOUT | Configures splunkdConnectionTimeout in `web.conf` with passed integer value (in seconds) | no | no | no |
 
 \* Password must be set either in `default.yml` or as the environment variable `SPLUNK_PASSWORD`
 
@@ -245,3 +247,9 @@ splunk:
     OPTIMISTIC_ABOUT_FILE_LOCKING: 1
   ...
 ```
+
+---
+
+## Multi-cluster Search
+
+See the [documentation on how multi-cluster search](advanced/MULTICLUSTERSEARCH.md) can be configured.

docs/CHANGELOG.md

@@ -2,12 +2,14 @@
 
 ## Navigation
 
+* [8.0.4.1](#8041)
 * [8.0.4](#804)
 * [8.0.3](#803)
 * [8.0.2.1](#8021)
 * [8.0.2](#802)
 * [8.0.1](#801)
 * [8.0.0](#800)
+* [7.3.6](#736)
 * [7.3.5](#735)
 * [7.3.4.2](#7342)
 * [7.3.4](#734)
@@ -31,19 +33,36 @@
 
 ---
 
+## 8.0.4.1
+
+#### What's New?
+* Support for setting the [deployer push mode](https://docs.splunk.com/Documentation/Splunk/latest/DistSearch/PropagateSHCconfigurationchanges#Choose_a_deployer_push_mode) to control how apps are bundled and distributed to cluster members:
+    * `shc.deployer_push_mode` in `default.yml`
+* Added the config variable `auxiliary_cluster_masters` to support enabling a search head to search across multiple indexer clusters. See [Multi-Cluster Search](advanced/MULTICLUSTERSEARCH.md) for details on configuration.
+* Documentation on executing `splunk-ansible` remotely, through a controller node such as Ansible Tower/AWX
+
+
+#### Changes
+* Apps copied from `etc/apps` now include the `local` directory, ignoring `local/app.conf`
+* Set custom Splunkd connection timeout using either:
+    * `splunk.connection_timeout` in `default.yml`
+    * `SPLUNK_CONNECTION_TIMEOUT` environment variable
+
+---
+
 ## 8.0.4
 
 #### What's New?
 * Support for custom SSL certificates for the Splunkd management endpoint
-* Support for custom ports for [Splunk Application Server](https://docs.splunk.com/Documentation/ITSI/latest/IModules/AboutApplicationServerModule) and [App KV Store](https://docs.splunk.com/Documentation/Splunk/8.0.0/Admin/AboutKVstore) using:
+* Support for custom ports for [Splunk Application Server](https://docs.splunk.com/Documentation/ITSI/latest/IModules/AboutApplicationServerModule) and [App KV Store](https://docs.splunk.com/Documentation/Splunk/8.0.0/Admin/AboutKVstore) using either:
     * `splunk.appserver.port`, `splunk.kvstore.port` in `default.yml`
     * `SPLUNK_APPSERVER_PORT`, `SPLUNK_KVSTORE_PORT` environment variables
 * Java installation through `default.yml` with `java_download_url`, `java_update_version`, and `java_version`
 * Support for Windows+AWS deployments for Splunk v7.2 and v7.3
 
 
 #### Changes
-* Set pass4SymmKey for indexer discovery separately from pass4SymmKey for indexer clustering with:
+* Set pass4SymmKey for indexer discovery separately from pass4SymmKey for indexer clustering with either:
     * `splunk.idxc.discoveryPass4SymmKey` in `default.yml`
     * `SPLUNK_IDXC_DISCOVERYPASS4SYMMKEY` environment variable
 * `outputs.conf` is configured without REST calls to ensure forwarding is enabled before Splunk starts
@@ -75,7 +94,7 @@
 * Added support for auto-detecting the `service_name` for SplunkForwarder and allowing manual configuration with `splunk.service_name`
 
 #### Changes
-* All HEC related variables were revised to follow a nested dict format in `default.yml`, i.e. `splunk.hec_enableSSL` is now `splunk.hec.ssl`. See the [Provision HEC](https://github.com/splunk/splunk-ansible/blob/develop/docs/EXAMPLES.md#provision-hec) example in the docs.
+* All HEC related variables were revised to follow a nested dict format in `default.yml`, i.e. `splunk.hec_enableSSL` is now `splunk.hec.ssl`. See the [Provision HEC](EXAMPLES.md#provision-hec) example in the docs.
 * Fixed HEC-related API calls to be idempotent. This supports changing anything in `splunk.hec.*` and having the change be reflected upon next container restart.
 
 ---
@@ -131,6 +150,16 @@
 
 ---
 
+## 7.3.6
+
+#### What's New?
+Syncing with latest codebase - currently up to sync with 8.0.4.1.
+
+#### Changes
+* See [8.0.4.1](#8041) changes.
+
+---
+
 ## 7.3.5
 
 #### What's New?
@@ -170,7 +199,7 @@ Syncing with latest codebase - currently up to sync with 8.0.2.1.
 
 #### Changes
 * Removing unnecessary apps in distributed ITSI installations
-* Partioning apps in `serverclass.conf` when using the deployment server
+* Partitioning apps in `serverclass.conf` when using the deployment server
 * Adding support for activating Splunk Free license on boot
 * Support for cluster labels via environment variables
 * Bugfixes around app installation (through `default.yml` and pathing)

docs/advanced/MULTICLUSTERSEARCH.md

@@ -0,0 +1,33 @@
+## Multi-cluster Search
+
+When configuring a search head, it's possible that enable multi-cluster search. This enables the ability to search for data across a series of indexer clusters, whether they be located in different datacenters or different geographical regions.
+
+For more information, see [Splunk docs on multi-cluster search](https://docs.splunk.com/Documentation/Splunk/latest/Indexer/Configuremulti-clustersearch).
+
+The Ansible playbooks provided in this repository offer this feature through the `auxiliary_cluster_masters` option in the `default.yml` variables. To enable this, modify this section of the `default.yml` to include a list of cluster masters responsible for brokering the indexer clusters:
+```
+splunk:
+  ...
+  cluster_master_url: master-primary.regionA.corp.net
+  auxiliary_cluster_masters:
+    - url: https://master-secondary.regionA.corp.net:8089
+      pass4SymmKey: secretidxckey
+    - url: https://master-tertiary.regionB.corp.net:8089
+      pass4SymmKey: newsecretidxckey
+  ...
+```
+
+Note that in the above, the search head being created must also set `cluster_master_url`. It is only possible to peer multiple indexer clusters when the search head has a primary indexer cluster to send its own internal logs and data to. 
+
+Each additional cluster master must also be given their own `pass4SymmKey` to enable authorization for this Splunk search head to connect and search over the various other clusters. 
+
+To confirm that the multi-cluster search works after Ansible has been completed, visit SplunkWeb on this search head and run the following query:
+```
+search index=_internal
+```
+
+If successful, you should see:
+* The data from `host=master-primary.regionA.corp.net`, plus any downstream indexers that connect to this cluster
+* The data from `host=master-secondary.regionA.corp.net`, plus any downstream indexers that connect to this cluster
+* The data from `host=master-tertiary.regionB.corp.net`, plus any downstream indexers that connect to this cluster
+* The data from the node just provisioned, which should be forwarded to `master-primary.regionA.corp.net`

docs/advanced/default.yml.spec.md

@@ -18,6 +18,10 @@ ansible_pre_tasks: <str>
 * Comma-separated list of paths or URLs to custom Ansible playbooks to run BEFORE Splunk sets up using the provided site.yml
 * Default: null
 
+ansible_environment: <dict>
+* Map of environment variables used only during the execution context of all the Ansible tasks. For more information, see https://docs.ansible.com/ansible/latest/user_guide/playbooks_environment.html
+* Default: {}
+
 hide_password: <bool>
 * Boolean that determines whether or not to output Splunk admin passwords through Ansible
 * Default: false
@@ -34,6 +38,10 @@ shc_sync_retry_num: <int>
 * Number of retries to make when waiting for sync up with a search head cluster
 * Default: 60
 
+retry_delay: <int>
+* Duration of waits between each of the aforementioned retries (in seconds)
+* Default: 6
+
 splunk_home_ownership_enforcement: true
 * Boolean that to control and enable UAC on $SPLUNK_HOME (recommended to be enabled)
 * Default: true
@@ -94,18 +102,54 @@ splunkbase_password: <str>
 * NOTE: Use this in combination with splunkbase_username. You will also need to run Ansible using the dynamic inventory script (environ.py) for this to register and work properly.
 * Default: null
 
+splunkbase_token: <str>
+* Used for authentication when downloading apps from https://splunkbase.splunk.com/ (this is NOT required to even be specified, unless you have SplunkBase apps defined in your splunk.apps_location)
+* NOTE: This is ordinarily generated using the dynamic inventory script (environ.py) using the aforementioned `splunkbase_username` and `splunkbase_password` variables above, and every token has an expiry.
+* Default: null
+
+cert_prefix: <str>
+* Specify the scheme used for the SplunkD management endpoint (typically port 8089). If you plan on running SplunkD over HTTP, you should set this to "http" so the Ansible plays are aware of the intended scheme.
+* Default: https
+
+java_download_url: <str>
+* Java JDK URL that is dynamically fetched and installed at container run-time. For example: "https://download.java.net/java/GA/jdk11/9/GPL/openjdk-11.0.2_linux-x64_bin.tar.gz"
+* Default: null
+
+java_update_version: <str>
+* Name of the Java JDK file used for installation. For example: "openjdk-11.0.2_linux-x64_bin.tar.gz"
+* Default: null
+
+java_version: <str>
+* String notifying the Ansible plays which version of Java is being installed so variables can be parsed properly. For example: "openjdk:11"
+* Default: null
+
+dmc_forwarder_monitoring: <bool>
+* Feature-flag to enable forwarder asset monitoring through the Distributed Management Console (DMC). This is disabled by default.
+* Default: false
+
+dmc_asset_interval: <str>
+* Cron-formatted string of the frequency and recurrence of the query that builds the forwarding assets table
+* Default: "3,18,33,48 * * * *"
+
+docker: <bool>
+* DEPRECATED - this was used to signal whether or not the instance being provisioned was running in Docker. This does not affect playbook execution at all, but the dynamic inventory script environ.py will set this to setup host::vars mapping as needed.
+
 splunk:
   role: <str>
   * Role to assume when setting up Splunk
   * Default: splunk_standalone
 
-  upgrade: <bool>
+  allow_upgrade:
   * Determines whether or not to perform an upgrade (to the splunk.build_location)
-  * Default: false
+  * Default: true
 
   build_location: <str>
   * Splunk build location, either on the filesystem or a remote URL
-  * Default: /tmp/splunk.tgz
+  * Default: null
+
+  build_url_bearer_token: <str>
+  * Bearer token used to provide authorization when fetching a Splunk build from a remote URL. 
+  * Default: null
 
   license_master_url: <str>
   * Hostname of Splunk Enterprise license master instance. May be overridden using SPLUNK_LICENSE_MASTER_URL environment variable.
@@ -114,6 +158,16 @@ splunk:
   cluster_master_url: <str>
   * Hostname of Splunk Enterprise cluster master instance. May be overridden using SPLUNK_CLUSTER_MASTER_URL environment variable.
   * Default: null
+
+  auxiliary_cluster_masters: <list>
+  * Array of other cluster masters to support multi-cluster distributed search. The node must be a search head configured to peer an initial cluster master before the masters listed here are added. For more information, see https://docs.splunk.com/Documentation/Splunk/latest/Indexer/Configuremulti-clustersearch. 
+  * Default: []
+  * Example:
+  *   auxiliary_cluster_masters:
+  *   - url: https://master.us-west.corp.net:8089
+  *     pass4SymmKey: thisisasecret
+  *   - url: https://master.us-east.corp.net:8089
+  *     pass4SymmKey: thisisanothersecret
   
   deployer_url: null
   * Hostname of Splunk Enterprise deployer instance. May be overridden using SPLUNK_DEPLOYER_URL environment variable.
@@ -153,11 +207,11 @@ splunk:
   * Default: false
 
   admin_user: <str>
-  * Default admin-level user to run provisioning commands under
+  * Default admin-level user to run provisioning commands under. It is only possible to change the admin user name at the first-time execution of Splunk Enterprise.
   * Default: admin
 
   password: <str>
-  * Default Splunk admin user password. This is REQUIRED when starting Splunk
+  * Default Splunk admin user password. This is REQUIRED when starting Splunk, and can only be set during the first-time run of the playbooks. If changes are required to the admin password, they should be done through SplunkWeb/CLI and the new value should be re-entered here.
   * Default: null
 
   user: <str>
@@ -172,6 +226,10 @@ splunk:
   * Determine whether or not to enable Splunk for boot-start (start via sysinitv or systemd, etc.)
   * Default: false
 
+  service_name: <str>
+  * Specify the service name of splunkd when running through sysinitv, systemd, etc.
+  * Default: null
+
   opt: <str - filepath>
   * Path in filesystem where Splunk will be installed
   * Default: /opt
@@ -254,6 +312,10 @@ splunk:
   * Determine the port used for SplunkWeb
   * Default: 8000
 
+  root_endpoint: <str>
+  * Root endpoint used when serving SplunkWeb over a different path
+  * Default: null
+
   s2s:
     enable: <bool>
     * Determine whether or not to enable Splunk-to-Splunk communication. This is REQUIRED for any distributed topologies.
@@ -297,6 +359,14 @@ splunk:
   * key::value pairs for environment variables that get written to ${SPLUNK_HOME}/etc/splunk-launch.conf
   * Default: null
 
+  asan: <bool>
+  * Feature-flag to enable special configurations when using debug, address-sanitized builds. This is not used externally and not recommended to change.
+  * Default: false
+
+  connection_timeout: <int>
+  * Change timeout value (in seconds) for the setting `splunkdConnectionTimeout` in web.conf. This triggers a change only when the value is non-zero.
+  * Default: 0
+
   secret: <str>
   * Secret passcode used to encrypt all of Splunk's sensitive information on disk. When not set, Splunk will autogenerate a unique secret local to each installation. This is NOT required for any standalone or distributed Splunk topology
   * NOTE: This may be set once at the start of provisioning any deployment. Any changes made to this splunk.secret after the deployment has been created must be resolved manually, otherwise there is a severe risk of bricking the capabilities of your Splunk environment.
@@ -355,6 +425,30 @@ splunk:
     * Determine the secret used to enable indexer discovery (for any forwarding clients connecting to the cluster master). This is pass4SymmKey in the `[indexer_discovery]` stanza of server.conf.
     * Default: null
 
+  multisite_master_port:
+  * Specify the management port of the multisite cluster master
+  * Default: 8089
+
+  multisite_replication_factor_origin:
+  * Determine origin-level knowledge object replication factor when in a multisite environment
+  * Default: 2
+
+  multisite_replication_factor_total:
+  * Determine site-level knowledge object replication factor when in a multisite environment
+  * Default: 3
+
+  multisite_search_factor_origin:
+  * Determine origin-level search replication factor when in a multisite environment
+  * Default: 1
+
+  multisite_search_factor_total:
+  * Determine site-level search replication factor when in a multisite environment
+  * Default: 3
+
+  set_search_peers: <bool>
+  * Feature-flag to disable the automatic peering from the search tier to the indexer tier (cluster master or indexers directly). It is discouraged to change this to false, but it is exposed for the purposes of testing and isolating the groups.
+  * Default: true
+
   shc:
     label: <str>
     * Provide a label for search head clustering configuration
@@ -377,6 +471,11 @@ splunk:
     * Determine the secret used to configure search head clustering. This is REQUIRED when setting up search head clustering. This is pass4SymmKey in the `[shclustering]` stanza of server.conf.
     * Default: null
 
+    deployer_push_mode: <str>
+    * Change the strategy used by the deployer when bundling apps and distributing them across the search head cluster. The acceptable modes are: full, local_only, default_only, and merge_to_default (merge_to_default is the default unless otherwise specified).
+    * For more information, please see: https://docs.splunk.com/Documentation/Splunk/latest/DistSearch/PropagateSHCconfigurationchanges#Set_the_deployer_push_mode
+    * Default: null
+
   dfs:
     enable: <bool>
     * Enable Data Fabric Search (DFS)

docs/execution_patterns/EXECUTION.md

@@ -0,0 +1,39 @@
+# Execution
+
+The Ansible plays in the `splunk-ansible` project can be run in two ways: separately on each instance/host of the Splunk Enterprise deployment, or through more traditional separation of control nodes and managed nodes. In the first method each host asynchronously sets itself up using Ansible roles to form the final desired topology which is most clearly displayed through the [docker-splunk](https://github.com/splunk/docker-splunk) project. All execution methods are listed below.
+
+--- 
+
+## Navigation
+
+* [Local](#local)
+* [Embedded](#embedded)
+* [Remote](#remote)
+
+---
+
+## Local
+Local connection is the intended mode of using `splunk-ansible`. The dynamic inventory script `environ.py` reads environment variables and maps them into Ansible run-time variables that determine how Splunk Enterprise is setup. 
+
+In order to bring up the most basic Splunk standalone instance on a local host, you can run the following:
+
+```bash
+export SPLUNK_PASSWORD=helloworld
+export SPLUNK_BUILD_URL=https://download.splunk.com/products/splunk/releases/8.0.3/linux/splunk-8.0.3-a6754d8441bf-Linux-x86_64.tgz
+export SPLUNK_USER=$(whoami)
+export SPLUNK_GROUP=$(id -gn)
+
+ansible-playbook --inventory inventory/environ.py --limit localhost site.yml
+```
+
+---
+
+## Embedded
+The embedded, or wrapper, mode of using `splunk-ansible` involves treating this entire project as a package. See [these instructions](wrapper-example/README.md) on how to install `splunk-ansible` on multiple target machines to bring up an indexer cluster.
+
+---
+
+## Remote
+The more traditional and familiar approach to running Ansible can also be used with `splunk-ansible`. This fits the use-case where `splunk-ansible` is installed on some controller node (ex. your personal workstation, Ansible Tower, or Ansible AWX) and this controller uses the ssh connection to setup Splunk on a series of target hosts. 
+
+See [these instructions](remote/README.md) on how to install `splunk-ansible` on multiple target machines to bring up an indexer cluster.