Merge pull request #564 from splunk/develop

Release/8.1.0

docs/ADVANCED.md

@@ -80,6 +80,10 @@ Splunk-Ansible ships with an inventory script in `inventory/environ.py`. The scr
 | SPLUNK_HTTP_ENABLESSL_CERT | Path to SSL certificate used for SplunkWeb, if HTTPS is enabled | no | no | no |
 | SPLUNK_HTTP_ENABLESSL_PRIVKEY | Path to SSL private key used for SplunkWeb, if HTTPS is enabled | no | no | no |
 | SPLUNK_HTTP_ENABLESSL_PRIVKEY_PASSWORD | SSL certificate private key password used with SplunkWeb, if HTTPS is enabled | no | no | no |
+| SPLUNKD_SSL_ENABLE | Enable HTTPS on Splunkd. By default, this is enabled out-of-the-box. To disable, set this to "false" | no | no | no |
+| SPLUNKD_SSL_CERT | Path to custom SSL certificate used for Splunkd when HTTPS is enabled | no | no | no |
+| SPLUNKD_SSL_CA | Path to custom CA certificate used for Splunkd when HTTPS is enabled | no | no | no |
+| SPLUNKD_SSL_PASSWORD | Custom SSL password used with Splunkd when HTTPS is enabled | no | no | no |
 | SPLUNK_KVSTORE_PORT | Port to run Splunk KVStore. Default: `8191` | no | no | no |
 | SPLUNK_APPSERVER_PORT | Port to run Splunk appserver. Default: `8065` | no | no | no |
 | SPLUNK_SET_SEARCH_PEERS | Boolean to configure whether search heads should connect to search peers. Default: `True`. Not recommended to change | no | no | no |

docs/CHANGELOG.md

@@ -2,6 +2,7 @@
 
 ## Navigation
 
+* [8.1.0](#810)
 * [8.0.6](#806)
 * [8.0.5.1](#8051)
 * [8.0.5](#805)
@@ -37,12 +38,26 @@
 
 ---
 
+## 8.1.0
+
+#### What's New?
+* Added environment variables to configure HTTPS on Splunkd. See [Supported environment variables](ADVANCED.md#supported-environment-variables) for details.
+    * `SPLUNKD_SSL_` prefixed environment variables
+    * `splunk.ssl` section in `default.yml`
+
+#### Changes
+* Enabled multisite for the `splunk_monitor` role
+* Enabled local indexing on the license master
+* Bugfixes and cleanup
+
+---
+
 ## 8.0.6
 
 #### What's New?
 * Support for declarative admin password, enabling password updates and rotations. `splunk.password` will always be the password for the admin user and changes to `splunk.password` will drive password reconciliation.
     * `splunk.declarative_admin_password` in `default.yml`
-    * `SPLUNK_DECLARATVE_ADMIN_PASSWORD` environment variable
+    * `SPLUNK_DECLARATIVE_ADMIN_PASSWORD` environment variable
 * Added flag to disable pop-ups and new user tour
     * `splunk.disable_popups` in `default.yml`
     * `SPLUNK_DISABLE_POPUPS` environment variable

docs/SETUP.md

@@ -52,9 +52,8 @@ $ docker run -it splunk/splunk:latest create-defaults > default.yml
 Alternatively, you can download the example `default.yml` supplied [here](advanced/default.yml.spec.md#sample).
 
 3. Define a few key variables in your `default.yml`:
-* `splunk.role`: the role this instance will play in the Splunk Enterprise deployment
+* `splunk.role`: the role this instance will play in the Splunk Enterprise deployment. (e.g. `splunk_standalone`)
 * `splunk.build_location`: URL to dynamically fetch the Splunk Enterprise build and install it at run time
-* `splunk.build_remote_src`: this wll be `true` when `splunk.build_location` above is a URL
 * `splunk.password`: default `admin` user password that Splunk will be provisioned with on first-time run
 
 4. Inspect your newly-created `default.yml` and tweak options as you see fit. For a full list of parameters, please see the [`default.yml.spec`](advanced/default.yml.spec.md#spec).

docs/execution_patterns/remote/README.md

@@ -83,7 +83,6 @@ $ docker run -it splunk/splunk:latest create-defaults > default.yml
 If you plan on running Ansible remotely, there are a few key variables you must make sure you define:
 * `splunk.role`: the role this instance will play in the Splunk Enterprise deployment
 * `splunk.build_location`: URL to dynamically fetch the Splunk Enterprise build and install it at run time
-* `splunk.build_remote_src`: this wll be `true` when `splunk.build_location` above is a URL
 * `splunk.password`: default `admin` user password that Splunk will be provisioned with on first-time run
 
 ## Deploy Splunk

docs/execution_patterns/remote/default.yml

@@ -5,7 +5,6 @@ shc_sync_retry_num: 60
 
 splunk:
   build_location: https://download.splunk.com/products/splunk/releases/8.0.2.1/linux/splunk-8.0.2.1-f002026bad55-Linux-x86_64.tgz
-  build_remote_src: true
   admin_user: admin
   allow_upgrade: true
   app_paths:

inventory/environ.py

@@ -259,6 +259,21 @@ def getSplunkWebSSL(vars_scope):
     splunk_vars["http_enableSSL_privKey_password"] = os.environ.get('SPLUNK_HTTP_ENABLESSL_PRIVKEY_PASSWORD', splunk_vars.get("http_enableSSL_privKey_password"))
     splunk_vars["http_port"] = int(os.environ.get('SPLUNK_HTTP_PORT', splunk_vars.get("http_port")))
 
+def getSplunkdSSL(vars_scope):
+    """
+    Parse and set parameters to define Splunkd
+    """
+    if "ssl" not in vars_scope["splunk"]:
+        vars_scope["splunk"]["ssl"] = {}
+    ssl_vars = vars_scope["splunk"]["ssl"]
+    ssl_vars["cert"] = os.environ.get("SPLUNKD_SSL_CERT", ssl_vars.get("cert"))
+    ssl_vars["ca"] = os.environ.get("SPLUNKD_SSL_CA", ssl_vars.get("ca"))
+    ssl_vars["password"] = os.environ.get("SPLUNKD_SSL_PASSWORD", ssl_vars.get("password"))
+    ssl_vars["enable"] = ssl_vars.get("enable", True)
+    enable = os.environ.get("SPLUNKD_SSL_ENABLE", "")
+    if enable.lower() == "false":
+        ssl_vars["enable"] = False
+
 def getDistributedTopology(vars_scope):
     """
     Parse and set parameters to define topology if this is a distributed environment
@@ -340,19 +355,22 @@ def getSplunkApps(vars_scope):
     """
     Determine the set of Splunk apps to install as union of defaults.yml and environment variables
     """
-    appSet = set()
+    appList = []
     if not "apps_location" in vars_scope["splunk"]:
         vars_scope["splunk"]["apps_location"] = []
     # From default.yml
     elif type(vars_scope["splunk"]["apps_location"]) == str:
-        appSet.update(vars_scope["splunk"]["apps_location"].split(","))
+        appList = vars_scope["splunk"]["apps_location"].split(",")
     elif type(vars_scope["splunk"]["apps_location"]) == list:
-        appSet.update(vars_scope["splunk"]["apps_location"])
+        appList = vars_scope["splunk"]["apps_location"]
     # From environment variables
     apps = os.environ.get("SPLUNK_APPS_URL")
     if apps:
-        appSet.update(apps.split(","))
-    vars_scope["splunk"]["apps_location"] = list(appSet)
+        apps = apps.split(",")
+        for app in apps:
+            if app not in appList:
+                appList.append(app)
+    vars_scope["splunk"]["apps_location"] = appList
 
 def getSecrets(vars_scope):
     """
@@ -559,6 +577,8 @@ def merge_dict(dict1, dict2, path=None):
         if key in dict1:
             if isinstance(dict1[key], dict) and isinstance(dict2[key], dict):
                 merge_dict(dict1[key], dict2[key], path + [str(key)])
+            elif isinstance(dict1[key], list) and isinstance(dict2[key], list):
+                dict1[key] += dict2[key]
             else:
                 dict1[key] = dict2[key]
         else:

roles/splunk_cluster_master/tasks/generate_ess_bundle.yml

@@ -20,6 +20,7 @@
     validate_certs: false
     status_code: 200
     timeout: 10
+    use_proxy: no
   register: ess_info
   no_log: "{{ hide_password }}"
 

roles/splunk_cluster_master/tasks/initialize_cluster_master.yml

@@ -12,6 +12,7 @@
     body_format: "form-urlencoded"
     status_code: 201,409
     timeout: 10
+    use_proxy: no
   register: set_indexer_discovery
   changed_when: set_indexer_discovery.status == 201
   no_log: "{{ hide_password }}"

roles/splunk_common/tasks/apply_licenses.yml

@@ -2,6 +2,9 @@
 - include_tasks: licenses/enable_free_license.yml
   when: license | lower == "free"
 
+- include_tasks: licenses/enable_forwarder_license.yml
+  when: license | lower == "forwarder"
+
 - include_tasks: licenses/add_wildcard_license.yml
   vars:
     licenses: "{{ license }}"

roles/splunk_common/tasks/check_for_required_restarts.yml

@@ -8,6 +8,7 @@
     validate_certs: false
     status_code: 200,404
     timeout: 10
+    use_proxy: no
   register: restart_required
   changed_when: restart_required.status == 200
   no_log: "{{ hide_password }}"

roles/splunk_common/tasks/disable_popups.yml

@@ -9,6 +9,7 @@
     status_code: 200
     timeout: 10
     return_content: yes
+    use_proxy: no
   register: telemetry
   no_log: "{{ hide_password }}"
 
@@ -21,7 +22,8 @@
     body: "{{ item.value }}"
     validate_certs: false
     status_code: 200,201,409
+    use_proxy: no
   with_items:
     - { key: "servicesNS/admin/user-prefs/data/user-prefs/general", value: "hideInstrumentationOptInModal=1&notification_python_3_impact=false&showWhatsNew=0" }
     - { key: "servicesNS/nobody/splunk_instrumentation/admin/telemetry/general", value: "showOptInModal=0&optInVersionAcknowledged={{ telemetry['json']['entry'][0]['content']['optInVersion'] }}" }
-    - { key: "servicesNS/admin/search/data/ui/ui-tour/search-tour", value: "tourPage=search&viewed=1" }
+    - { key: "servicesNS/admin/search/data/ui/ui-tour/search-tour", value: "tourPage=search&viewed=1" }

roles/splunk_common/tasks/enable_dfs.yml

@@ -14,6 +14,7 @@
     body_format: "form-urlencoded"
     status_code: 200
     timeout: 10
+    use_proxy: no
   register: dfs_enable_result
   changed_when: dfs_enable_result.status == 200
 
@@ -31,6 +32,7 @@
     body_format: "form-urlencoded"
     status_code: 200
     timeout: 10
+    use_proxy: no
   register: dfs_limits_result
   changed_when: dfs_limits_result.status == 200
 
@@ -47,6 +49,7 @@
     body_format: "form-urlencoded"
     status_code: 200
     timeout: 10
+    use_proxy: no
   register: search_limits_result
   changed_when: search_limits_result.status == 200
 
@@ -57,6 +60,7 @@
     user: "{{ splunk.admin_user }}"
     password: "{{ splunk.password }}"
     validate_certs: false
+    use_proxy: no
   register: check_dfs_job_extractor_result
   failed_when: False
   changed_when: False
@@ -75,6 +79,7 @@
     body_format: "form-urlencoded"
     status_code: 201
     timeout: 10
+    use_proxy: no
   register: create_dfs_job_extractor_result
   changed_when: create_dfs_job_extractor_result.status == 201
   when: check_dfs_job_extractor_result.status != 200
@@ -92,6 +97,7 @@
     body_format: "form-urlencoded"
     status_code: 200
     timeout: 10
+    use_proxy: no
   register: update_dfs_job_extractor_result
   changed_when: update_dfs_job_extractor_result.status == 200
   when: check_dfs_job_extractor_result.status == 200

roles/splunk_common/tasks/enable_forwarder_monitoring.yml

@@ -12,6 +12,7 @@
     validate_certs: false
     status_code: 200,201
     timeout: 10
+    use_proxy: no
   register: dmc_forwarder_build_assets
   no_log: "{{ hide_password }}"
 
@@ -31,6 +32,7 @@
     validate_certs: false
     status_code: 200,201
     timeout: 10
+    use_proxy: no
   no_log: "{{ hide_password }}"
 
 - name: Build forwarder assets
@@ -41,4 +43,5 @@
     password: "{{ splunk.password }}"
     validate_certs: false
     status_code: 200,201
+    use_proxy: no
   no_log: "{{ hide_password }}"

roles/splunk_common/tasks/enable_forwarding.yml

@@ -51,22 +51,19 @@
   when:
     - splunk_indexer_cluster | bool
     - item.value | length > 0
-  no_log: "{{ hide_password }}"
   register: default_tcpout_group
 
 # NOTE: If this task is called or used, it will disable all local indexing!
 - name: Disable indexing on the current node
   ini_file:
     path: "{{ splunk.home }}/etc/system/local/outputs.conf"
     section: "indexAndForward"
-    option: "{{ item.key }}"
-    value: "{{ item.value }}"
+    option: "index"
+    value: "false"
     owner: "{{ splunk.user }}"
     group: "{{ splunk.group }}"
-  with_items:
-    - {key: "index", value: "false"}
-  no_log: "{{ hide_password }}"
   register: index_disabling
+  when: splunk.role != "splunk_license_master"
 
 # set up forward servers set by get_facts
 - name: Add forward_servers

roles/splunk_common/tasks/enable_service.yml

@@ -40,41 +40,42 @@
     - splunk_systemd
     - installed_splunk_version[0] is version("7.2.2", "<")
 
-- name: "Update cpu settings"
-  replace:
-    path: "/etc/systemd/system/{{ splunk_service_name }}"
-    regexp: '^ExecStartPost=.*/sys/fs/cgroup/cpu.*$'
-    replace: 'ExecStartPost=/bin/bash -c "chown -R {{ splunk.user }}:{{ splunk.group }} /sys/fs/cgroup/cpu/system.slice/%n"'
-  become: yes
-  become_user: "{{ privileged_user }}"
+- name: Update systemd settings
   when:
     - ansible_system is match("Linux")
     - splunk_systemd
     - installed_splunk_version[0] is version("7.2.2", ">=")
-
-- name: "Update memory settings"
-  replace:
-    path: "/etc/systemd/system/{{ splunk_service_name }}"
-    regexp: '^ExecStartPost=.*/sys/fs/cgroup/memory.*$'
-    replace: 'ExecStartPost=/bin/bash -c "chown -R {{ splunk.user }}:{{ splunk.group }} /sys/fs/cgroup/memory/system.slice/%n"'
   become: yes
   become_user: "{{ privileged_user }}"
-  when:
-    - ansible_system is match("Linux")
-    - splunk_systemd
-    - installed_splunk_version[0] is version("7.2.2", ">=")
+  block:
+  - name: Update cpu cgroup
+    replace:
+      path: "/etc/systemd/system/{{ splunk_service_name }}"
+      regexp: '^ExecStartPost=.*/sys/fs/cgroup/cpu.*$'
+      replace: 'ExecStartPost=/bin/bash -c "chown -R {{ splunk.user }}:{{ splunk.group }} /sys/fs/cgroup/cpu/system.slice/%n"'
+    when: installed_splunk_version[0] is version("8.0.0", "<")
 
-- name: "Disable cgroup systemd settings in Docker"
-  replace:
-    path: "/etc/systemd/system/{{ splunk_service_name }}"
-    regexp: '^ExecStartPost=(.*)$'
-    replace: '#ExecStartPost=\1'
-  become: yes
-  become_user: "{{ privileged_user }}"
-  when:
-    - ansible_system is match("Linux")
-    - splunk_systemd
-    - installed_splunk_version[0] is version("8.0.0", ">=")
+  - name: Update memory cgroup
+    replace:
+      path: "/etc/systemd/system/{{ splunk_service_name }}"
+      regexp: '^ExecStartPost=.*/sys/fs/cgroup/memory.*$'
+      replace: 'ExecStartPost=/bin/bash -c "chown -R {{ splunk.user }}:{{ splunk.group }} /sys/fs/cgroup/memory/system.slice/%n"'
+    when: installed_splunk_version[0] is version("8.0.0", "<")
+
+  - name: Disable cgroup per Splunk version
+    replace:
+      path: "/etc/systemd/system/{{ splunk_service_name }}"
+      regexp: '^ExecStartPost=(.*)$'
+      replace: '#ExecStartPost=\1'
+    when:
+      - installed_splunk_version[0] is version("8.0.0", ">=")
+      - installed_splunk_version[0] is version("8.1.0", "<")
+
+  - name: Remove init.scope pathing
+    replace:
+      path: "/etc/systemd/system/{{ splunk_service_name }}"
+      regexp: '^(ExecStartPost=.*?)\/init.scope\/(.*)'
+      replace: '\1/\2'
 
 - name: "Reload daemons via systemctl - Linux (systemd)"
   become: yes

roles/splunk_common/tasks/get_facts.yml

@@ -70,13 +70,16 @@
 - name: "Set current version fact"
   set_fact:
     splunk_current_version: "{{ manifests.files[0].path | regex_search(regexp, '\\1') if (manifests.matched == 1) else '0' }}"
+    splunk_current_build_hash: "{{ manifests.files[0].path | regex_search(regexp, '\\3') if (manifests.matched == 1) else '0' }}"
+    splunk_target_build_hash: "{{ splunk.build_location | regex_search(regexp, '\\3') | default('0') }}"
   vars:
-    regexp: 'splunk\D*?-((\d+)\.(\d+)\.(\d+))'
+    regexp: 'splunk\D*?-(\d+\.\d+\.\d+(\.\d+)?)-(.*?)-.*?'
 
-# We are upgrading if it is not a fresh installation and the current version is different from the target version
+# We are upgrading if it is not a fresh installation and the current version is different from the target version,
+# and allowing upgrades between new and old hashes of the same version.
 - name: "Setting upgrade fact"
   set_fact:
-    splunk_upgrade: "{{ 'build_location' in splunk and splunk.build_location and not splunk_install and splunk_target_version and splunk_target_version != splunk_current_version | default(False) }}"
+    splunk_upgrade: "{{ 'build_location' in splunk and splunk.build_location and not splunk_install and splunk_target_version and ((splunk_target_version != splunk_current_version) or (splunk_current_build_hash != splunk_target_build_hash))| default(False) }}"
 
 # determine if we need to set up indexer clusters
 - name: "Setting indexer cluster fact from config"

roles/splunk_common/tasks/get_facts_target_version.yml

@@ -9,7 +9,7 @@
   set_fact:
     splunk_target_version: "{{ splunk.build_location | regex_search(regexp, '\\1') | default('0') }}"
   vars:
-    regexp: 'splunk\D*?-((\d+)\.(\d+)\.(\d+))'
+    regexp: 'splunk\D*?-(\d+\.\d+\.\d+(\.\d+)?)'
   when: "splunk_build_type is defined and splunk_build_type is match('(tgz|msi|rpm|deb)')"
 
 # if using yum to install a package, we can use the list option to see what versions are available