Improve argument streaming for remote job execution (part 1)

docs/ansible_runner.config.rst

@@ -7,4 +7,5 @@ Submodules
 ansible_runner.config.runner module
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
+.. autoclass:: ansible_runner.config.runner.BaseConfig
 .. autoclass:: ansible_runner.config.runner.RunnerConfig

docs/ansible_runner.rst

@@ -25,6 +25,7 @@ ansible_runner.interface module
 
 .. automodule:: ansible_runner.interface
     :members:
+    :exclude-members: init_runner
     :undoc-members:
     :show-inheritance:
 
@@ -44,14 +45,6 @@ ansible_runner.runner module
     :undoc-members:
     :show-inheritance:
 
-ansible_runner.runner\_config module
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-.. automodule:: ansible_runner.runner_config
-    :members:
-    :undoc-members:
-    :show-inheritance:
-
 ansible_runner.utils module
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^
 

docs/conf.py

@@ -26,6 +26,10 @@ def _get_version():
 
 
 nitpicky = True
+nitpick_ignore = {
+    ('py:class', '_io.FileIO')
+}
+
 default_role = 'any'  # This catches single backticks (incorrectly) used for inline code formatting
 project = 'ansible-runner'
 copyright = f'2018-{datetime.datetime.today().year}, Red Hat, Inc'

docs/intro.rst → docs/configuration.rst

@@ -1,7 +1,7 @@
-.. _intro:
+.. _config:
 
-Introduction to Ansible Runner
-==============================
+Configuring Ansible Runner
+==========================
 
 **Runner** is intended to be most useful as part of automation and tooling that needs to invoke Ansible and consume its results.
 Most of the parameterization of the **Ansible** command line is also available on the **Runner** command line but **Runner** also

docs/index.rst

@@ -15,7 +15,7 @@ want to manage the complexities of the interface on their own (such as CI/CD pla
 for running ``ansible`` and ``ansible-playbook`` tasks and gathers the output from it. It does this by presenting a common interface that doesn't
 change, even as **Ansible** itself grows and evolves.
 
-Part of what makes this tooling useful is that it can gather its inputs in a flexible way (See :ref:`intro`:). It also has a system for storing the
+Part of what makes this tooling useful is that it can gather its inputs in a flexible way (See :ref:`config`). It also has a system for storing the
 output (stdout) and artifacts (host-level event data, fact data, etc) of the playbook run.
 
 There are 3 primary ways of interacting with **Runner**
@@ -30,21 +30,31 @@ Examples of this could include:
 * Sending status to Ansible AWX
 * Sending events to an external logging service
 
-
 .. toctree::
    :maxdepth: 1
-   :caption: Contents:
+   :caption: Installation, Upgrade & Configuration
 
-   intro
    install
-   community
+   configuration
+   porting_guides/porting_guide
+
+.. toctree::
+   :maxdepth: 1
+   :caption: Using Ansible Runner
+
    external_interface
    standalone
    python_interface
    execution_environments
    remote_jobs
    modules
 
+.. toctree::
+   :maxdepth: 1
+   :caption: Getting Help
+
+   community
+
 
 Indices and tables
 ==================

docs/porting_guides/porting_guide.rst

@@ -0,0 +1,12 @@
+*****************************
+Ansible Runner Porting Guides
+*****************************
+
+This section lists porting guides that can help you when upgrading to newer
+versions of ``ansible-runner``.
+
+
+.. toctree::
+   :maxdepth: 1
+
+   porting_guide_v2.5

docs/porting_guides/porting_guide_v2.5.rst

@@ -0,0 +1,46 @@
+********************************
+Ansible Runner 2.5 Porting Guide
+********************************
+
+This section discusses the behavioral changes between ``ansible-runner`` version 2.4 and version 2.5.
+
+.. contents:: Topics
+
+Changes to the Python Interface
+===============================
+
+:ref:`Remote job execution <remote_jobs>` could experience problems when transmitting job arguments
+to a worker node with an older version of ``ansible-runner``. If the older worker received a job
+argument that it did not understand, like a new API value, that worker would terminate abnormally.
+To address this, a two-stage plan was devised:
+
+#. Modify the streaming job arguments so that only job arguments that have non-default values are
+   streamed to the worker node. A new API job argument will not be problematic for older workers
+   unless that argument is actually used.
+#. Have the worker node fail gracefully on unrecognized job arguments.
+
+The first step of this process is implemented in this version of ``ansible-runner``. The second
+step will be completed in a future version.
+
+For this change, it was necessary to modify how the ``run()`` and ``run_async()`` functions
+of the :ref:`Python API <python_interface>` are implemented. The API function arguments are now completely
+defined in the ``RunnerConfig`` object where we can have better control of the job arguments, and both
+functions now take an optional ``config`` parameter.
+
+For backward compatibility, keyword arguments to the ``run()/run_async()`` API functions are passed
+along to the ``RunnerConfig`` object initialization for you. Alternatively, you may choose to use
+the more up-to-date signature of those API functions where you pass in a manually created ``RunnerConfig``
+object. For example:
+
+.. code-block:: python
+
+    import ansible_runner
+    config = ansible_runner.RunnerConfig(private_data_dir='/tmp/demo', playbook='test.yml')
+    r = ansible_runner.interface.run(config=config)
+
+The above is identical to the more familiar usage of the API:
+
+.. code-block:: python
+
+    import ansible_runner
+    r = ansible_runner.interface.run(private_data_dir='/tmp/demo', playbook='test.yml')

pyproject.toml

@@ -21,6 +21,7 @@ disable = [
   "C0115",  # missing-class-docstring
   "C0116",  # missing-function-docstring
   "C0301",  # line-too-long
+  "C0302",  # too-many-lines
   "R0401",  # cyclic-import
   "R0801",  # duplicate-code
   "R0902",  # too-many-instance-attributes

src/ansible_runner/__main__.py

@@ -44,7 +44,9 @@
 from ansible_runner import run
 from ansible_runner import output
 from ansible_runner import cleanup
-from ansible_runner.utils import dump_artifact, Bunch, register_for_cleanup
+from ansible_runner._internal._dump_artifacts import dump_artifact
+from ansible_runner.defaults import default_process_isolation_executable
+from ansible_runner.utils import Bunch, register_for_cleanup
 from ansible_runner.utils.capacity import get_cpu_count, get_mem_in_bytes, ensure_uuid
 from ansible_runner.utils.importlib_compat import importlib_metadata
 from ansible_runner.runner import Runner
@@ -822,14 +824,11 @@ def main(sys_args=None):
             else:
                 vargs['inventory'] = abs_inv
 
-    output.configure()
-
-    # enable or disable debug mode
-    output.set_debug('enable' if vargs.get('debug') else 'disable')
-
-    # set the output logfile
+    debug = bool(vargs.get('debug'))
+    logfile = ''
     if ('logfile' in args) and vargs.get('logfile'):
-        output.set_logfile(vargs.get('logfile'))
+        logfile = vargs.get('logfile')
+    output.configure(debug, logfile)
 
     output.debug('starting debug logging')
 
@@ -886,8 +885,8 @@ def main(sys_args=None):
                     "project_dir": vargs.get('project_dir'),
                     "artifact_dir": vargs.get('artifact_dir'),
                     "roles_path": [vargs.get('roles_path')] if vargs.get('roles_path') else None,
-                    "process_isolation": vargs.get('process_isolation'),
-                    "process_isolation_executable": vargs.get('process_isolation_executable'),
+                    "process_isolation": bool(vargs.get('process_isolation')),
+                    "process_isolation_executable": vargs.get('process_isolation_executable') or default_process_isolation_executable,
                     "process_isolation_path": vargs.get('process_isolation_path'),
                     "process_isolation_hide_paths": vargs.get('process_isolation_hide_paths'),
                     "process_isolation_show_paths": vargs.get('process_isolation_show_paths'),

src/ansible_runner/_internal/_dump_artifacts.py

@@ -0,0 +1,127 @@
+from __future__ import annotations
+
+import fcntl
+import hashlib
+import json
+import os
+import stat
+import tempfile
+
+from collections.abc import MutableMapping
+from typing import Any
+
+from ansible_runner.config.runner import RunnerConfig
+from ansible_runner.utils import isinventory, isplaybook
+
+
+def dump_artifacts(config: RunnerConfig) -> None:
+    """Introspect the arguments and dump objects to disk"""
+    private_data_dir = config.private_data_dir or ""
+
+    if config.role:
+        role: dict[str, Any] = {'name': config.role}
+        if config.role_vars:
+            role['vars'] = config.role_vars
+
+        hosts = config.host_pattern or 'all'
+        play: list[dict[str, Any]] = [{'hosts': hosts, 'roles': [role]}]
+
+        if config.role_skip_facts:
+            play[0]['gather_facts'] = False
+
+        config.playbook = play
+
+        if config.envvars is None:
+            config.envvars = {}
+
+        roles_path = config.roles_path
+        if not roles_path:
+            roles_path = os.path.join(private_data_dir, 'roles')
+        else:
+            roles_path += f":{os.path.join(private_data_dir, 'roles')}"
+
+        config.envvars['ANSIBLE_ROLES_PATH'] = roles_path
+
+    playbook = config.playbook
+    if playbook:
+        # Ensure the play is a list of dictionaries
+        if isinstance(playbook, MutableMapping):
+            playbook = [playbook]
+
+        if isplaybook(playbook):
+            path = os.path.join(private_data_dir, 'project')
+            config.playbook = dump_artifact(json.dumps(playbook), path, 'main.json')
+
+    obj = config.inventory
+    if obj and isinventory(obj):
+        path = os.path.join(private_data_dir, 'inventory')
+        if isinstance(obj, MutableMapping):
+            config.inventory = dump_artifact(json.dumps(obj), path, 'hosts.json')
+        elif isinstance(obj, str):
+            if not os.path.exists(os.path.join(path, obj)):
+                config.inventory = dump_artifact(obj, path, 'hosts')
+            elif os.path.isabs(obj):
+                config.inventory = obj
+            else:
+                config.inventory = os.path.join(path, obj)
+
+    if not config.suppress_env_files:
+        for key in ('envvars', 'extravars', 'passwords', 'settings'):
+            obj = getattr(config, key, None)
+            if obj and not os.path.exists(os.path.join(private_data_dir, 'env', key)):
+                path = os.path.join(private_data_dir, 'env')
+                dump_artifact(json.dumps(obj), path, key)
+
+        for key in ('ssh_key', 'cmdline'):
+            obj = getattr(config, key, None)
+            if obj and not os.path.exists(os.path.join(private_data_dir, 'env', key)):
+                path = os.path.join(private_data_dir, 'env')
+                dump_artifact(obj, path, key)
+
+
+def dump_artifact(obj: str,
+                  path: str,
+                  filename: str | None = None
+                  ) -> str:
+    """Write the artifact to disk at the specified path
+
+    :param str obj: The string object to be dumped to disk in the specified
+        path. The artifact filename will be automatically created.
+    :param str path: The full path to the artifacts data directory.
+    :param str filename: The name of file to write the artifact to.
+        If the filename is not provided, then one will be generated.
+
+    :return: The full path filename for the artifact that was generated.
+    """
+    if not os.path.exists(path):
+        os.makedirs(path, mode=0o700)
+
+    p_sha1 = hashlib.sha1()
+    p_sha1.update(obj.encode(encoding='UTF-8'))
+
+    if filename is None:
+        _, fn = tempfile.mkstemp(dir=path)
+    else:
+        fn = os.path.join(path, filename)
+
+    if os.path.exists(fn):
+        c_sha1 = hashlib.sha1()
+        with open(fn) as f:
+            contents = f.read()
+        c_sha1.update(contents.encode(encoding='UTF-8'))
+
+    if not os.path.exists(fn) or p_sha1.hexdigest() != c_sha1.hexdigest():
+        lock_fp = os.path.join(path, '.artifact_write_lock')
+        lock_fd = os.open(lock_fp, os.O_RDWR | os.O_CREAT, stat.S_IRUSR | stat.S_IWUSR)
+        fcntl.lockf(lock_fd, fcntl.LOCK_EX)
+
+        try:
+            with open(fn, 'w') as f:
+                os.chmod(fn, stat.S_IRUSR | stat.S_IWUSR)
+                f.write(str(obj))
+        finally:
+            fcntl.lockf(lock_fd, fcntl.LOCK_UN)
+            os.close(lock_fd)
+            os.remove(lock_fp)
+
+    return fn