Implement custom configuration for daemons and bgp.session password f…

…or bird

Implementing custom configuration templates for daemons is a bit harder than
expected:

* As we're assuming we cannot deploy extra stuff to daemons with Ansible
  playbooks, we have to add custom configuration templates into daemon_config
  dictionary so they get rendered as files and deployed to the container (and
  later VM)
* The custom configuration templates can be in the usual places, but have to be
  registered in the daemon.daemon_config early in the process (for example,
  during the plugin initialization)
* As Box aggresively creates hierarchies out of dotted keys, we have to use '@'
  instead of '.' in the daemon_config dictionary (example: 'bgp@session' instead
  of 'bgp.session')

With this in mind, the following changes were made to the code:

* The daemon_config dictionary can be copied from devices to nodes only after
  plugins have been initialized. The copy operation is done in the
  augment_node_device_data
* The node _daemon_config dictionary is further cleaned in a new
  augment.nodes.cleanup function -- all entries referring to inactive modules or
  extra config templates are removed. This cleanup was previously done in the
  modules.augment_node_module, which is executed too early (before the plugins
  did their job)
* The mapping of clab binds has to be done as late as possible (when the
  _daemon_config dictionary has been cleaned up) and has been moved to a new
  node_post_transform clab hook.
* As there's no call to a node-specific provider post_transform hook, the main
  post_transform hook calls node_post_transform hooks (please note that every
  node could use a different provider)
* The 'find_extra_template' function uses different paths, path suffixes and
  file names when searching for templates that are listed in the node.config
  list. To make that work, we have to pass lab topology as an extra argument
  into that function.
* 'initial-config.ansible' playbook does not start tasks to deploy custom
  configuration templates if the same template (potentially using @) is listed
  in node._daemon_config
* 'create-config.yml' task list has to deal with '@'-means-'.' stupidity

Sample implementation (bird):

* Adds handling of MD5 password into the main bgp.j2 template because bird
  cannot have the same protocol defined (and merged) in two places
* To simplify future additions, the main Bird config file includes everything
  from the _daemon_config as an include file
* 'bird.j2' template in the extra/bgp.session directory is just a placeholder
* Bird device data were added to the bgp.session/defaults.yml file, including a
  mapping of the related configuration template
* There's a new integration test for the bgp.session MD5 password functionality.
  Similar tests will be added when we need them for further bird bgp.session
  features.

Other minor fixes:

* Add a few task/play headers in create-config.ansible, create-custom-config.yml

docs/dev/transform.md

@@ -104,7 +104,8 @@ The data transformation has three major steps:
 
 * Execute **post_transform** plugin hooks
 * Merge group-level and node-level [custom deployment templates](custom-config) (`netsim.augment.groups.node_config_template`)
-* Execute **post_transform** provider hooks
+* Cleanup node data in `augment.nodes.cleanup` function -- at the moment, the function prunes the `_daemon_config` dictionary.
+* Execute **post_transform** primary provider hook and node-specific **node_post_transform** provider hooks.
 * Process device quirks
 * Cleanup links: remove empty **links** list and **_linkname** attribute from individual links
 * Cleanup groups: remove settings (keys starting with '\_') from **groups** dictionary

docs/plugins/bgp.session.md

@@ -86,6 +86,12 @@ BGP session security features are available on these platforms:
 | Nokia SR Linux      |     ✅   |  ❌  |  ❌  |
 | Nokia SR OS         |     ✅   |  ❌  | ✅  |
 
+BGP session security features are also available on these daemons:
+
+| Operating system    | password | GTSM | TCP-AO |
+| ------------------- | :------: | :-: | :-: |
+| BIRD                |    ✅    | ❌   | ❌   |
+
 (bgp-session-as-path)=
 The plugin implements AS-path-mangling nerd knobs for the following platforms:
 

netsim/ansible/create-config.ansible

@@ -52,7 +52,7 @@
         vars:
           paths: "{{ paths_custom.dirs }}"
 
-- name: Create custom deployment templates
+- name: Create daemon configuration files
   hosts: daemons
   serial: 1
   vars:
@@ -64,6 +64,7 @@
   tasks:
   - include_tasks: "tasks/create-config.yml"
     loop: "{{ extra_config }}"
+    when: "'@' not in item"
     args:
       apply:
         vars:

netsim/ansible/initial-config.ansible

@@ -106,7 +106,10 @@
     args:
       apply:
         tags: [ always ]
-    when: custom_config in config
+    when: >
+      custom_config in config and 
+      custom_config not in _daemon_config|default({}) and
+      custom_config.replace('.','@') not in _daemon_config|default({})
     loop: "{{ netlab_custom_config }}"
     loop_control:
       loop_var: custom_config

netsim/ansible/tasks/create-config.yml

@@ -1,8 +1,12 @@
 - delegate_to: localhost
-  vars:
-    config_module: "{{ item }}"
   block:
-  - set_fact:
+  - name: "Compute final module name for {{ item }}"
+    set_fact:
+      config_module: "{{ item.replace('@','.') }}"
+      item: "{{ item.replace('@','.') }}"
+
+  - name: "Find configuration template {{ config_module }}"
+    set_fact:
       config_template: "{{ lookup('first_found',params,errors='ignore') }}"
     vars:
       params:
@@ -16,6 +20,6 @@
 
   - fail:
       msg: >
-        Missing configuration template for {{ item }} on device
+        Missing configuration template for {{ config_module }} on device
         {{ netlab_device_type|default(ansible_network_os) }}/{{ ansible_network_os }}
     when: not config_template

netsim/ansible/tasks/create-custom-config.yml

@@ -1,6 +1,7 @@
 - delegate_to: localhost
   block:
-  - set_fact:
+  - name: Find custom configuration template
+    set_fact:
       config_template: "{{ lookup('first_found',params,errors='ignore') }}"
     vars:
       node_provider: "{{ provider|default(netlab_provider) }}"

netsim/augment/config.py

@@ -126,7 +126,6 @@ def make_paths_absolute(p_top: Box) -> None:
     if isinstance(v,str):
       v = [ v ]
     if isinstance(v,list):
-      print(f'transforming: {k} {v}')
       p_top[k] = _files.absolute_search_path(v)
     elif isinstance(v,Box):
       make_paths_absolute(v)

netsim/augment/main.py

@@ -89,6 +89,7 @@ def post_transform(topology: Box) -> None:
   modules.post_transform(topology)
   augment.plugin.execute('post_transform',topology)
   augment.groups.node_config_templates(topology)
+  augment.nodes.cleanup(topology)
   providers.execute("post_transform",topology)
   log.exit_on_error()
 

netsim/augment/nodes.py

@@ -176,18 +176,6 @@ def find_node_device(n: Box, topology: Box) -> bool:
   if not isinstance(dev_def,Box):
     log.fatal(f"Device data for device {devtype} must be a dictionary")
 
-  if dev_def.get('daemon',False):                 # Special handling of daemons
-    n._daemon = True                              # First, set the daemon flag so we don't have to look up the device data
-    n._daemon_parent = dev_def.daemon_parent      # Next, remember the parent device -- we need that in template search paths
-    if 'daemon_config' in dev_def:                # Does the daemon need special configuration files?
-      n._daemon_config = dev_def.daemon_config    # Yes, save it for later (clab binds or Ansible playbooks)
-
-  # Do a sanity check on _daemon_config dictionary. Remove faulty value to prevent downstream crashes
-  #
-  if '_daemon_config' in n and not isinstance(n._daemon_config,Box):
-    log.error(f"Daemon configuration files for node {n} must be a dictionary")
-    n.pop('_daemon_config',None)
-
   return True
 
 """
@@ -320,6 +308,18 @@ def augment_node_device_data(n: Box, defaults: Box) -> None:
       if not k in n:
         n[k] = defaults.devices[n.device].node[k]
 
+  if dev_data.get('daemon',False):                # Special handling of daemons
+    n._daemon = True                              # First, set the daemon flag so we don't have to look up the device data
+    n._daemon_parent = dev_data.daemon_parent     # Next, remember the parent device -- we need that in template search paths
+    if 'daemon_config' in dev_data:               # Does the daemon need special configuration files?
+      n._daemon_config = dev_data.daemon_config   # Yes, save it for later (clab binds or Ansible playbooks)
+
+  # Do a sanity check on _daemon_config dictionary. Remove faulty value to prevent downstream crashes
+  #
+  if '_daemon_config' in n and not isinstance(n._daemon_config,Box):
+    log.error(f"Daemon configuration files for node {n} must be a dictionary")
+    n.pop('_daemon_config',None)
+
 '''
 Main node transformation code
 
@@ -378,6 +378,26 @@ def transform(topology: Box, defaults: Box, pools: Box) -> None:
     augment_mgmt_if(n,defaults,topology.addressing.mgmt)
     providers.execute_node("augment_node_data",n,topology)
 
+'''
+Final cleanup of node data
+'''
+def cleanup_daemon_config(n: Box) -> None:
+  for k in list(n._daemon_config.keys()):
+    if k.startswith('_'):                                   # Skip internal mappings (will have to be redone later)
+      continue
+
+    kn = k.replace('@','.')                                 # A workaround for aggressive de-dotting
+    # Leave config mappings for device configuration, module configuration, or extra configs
+    if kn == n.device or kn in n.get('module',[]) or kn in n.get('config',[]):
+      continue
+
+    n._daemon_config.pop(k,None)
+
+def cleanup(topology: Box) -> None:
+  for name,n in topology.nodes.items():
+    if '_daemon_config' in n:
+      cleanup_daemon_config(n)
+
 '''
 Return a copy of the topology (leaving original topology unchanged) with unmanaged devices removed
 '''

netsim/daemons/bird/bgp.j2

@@ -35,6 +35,9 @@ protocol bgp bgp_{{ n.name }}_{{ af }} {
 {%     set local_ip = loopback[af]|default('') %}
   local {{ local_ip.split('/')[0] if n.type == 'ibgp' else '' }} as {{ n.local_as|default(bgp.as) }};
   neighbor {{ n[af] }} as {{ n['as'] }};
+{%     if n.password is defined %}
+  password "{{ n.password }}";
+{%     endif %}
 {%     if bgp.rr|default('') and ((not n.rr|default('') and n.type == 'ibgp') or n.type == 'localas_ibgp') %}
   rr client;
 {%       if bgp.rr|default(False) and bgp.rr_cluster_id|default(False) %}

netsim/daemons/bird/bird.j2

@@ -18,9 +18,6 @@ protocol kernel {
 }
 {% endfor %}
 
-{% if 'bgp' in module %}
-include "/etc/bird/bgp.mod.conf";
-{% endif %}
-{% if 'ospf' in module %}
-include "/etc/bird/ospf.mod.conf";
-{% endif %}
+{% for k,v in _daemon_config.items() if k != device|default(netlab_device_type) %}
+include "{{ v }}";
+{% endfor %}

netsim/extra/bgp.session/bird.j2

@@ -0,0 +1 @@
+# Empty file, all configuration is done in bgp.j2

netsim/extra/bgp.session/defaults.yml

@@ -2,6 +2,11 @@
 #
 ---
 devices:
+  bird:
+    daemon_config:
+      bgp@session: /etc/bird/bgp.session.conf
+    features.bgp:
+      password: True
   csr.features.bgp:
     allowas_in: True
     as_override: True

netsim/modules/__init__.py

@@ -145,12 +145,6 @@ def augment_node_module(topology: Box) -> None:
       if g_module and (not is_host or daemon):
         n.module = g_module
 
-    # Remove modules that are not used on the current daemon from the _daemon_config dictionary
-    if '_daemon_config' in n:
-      for m in list(n._daemon_config.keys()):
-        if m in mod_list and not m in n.module:                 # If an entry in _daemon_config is a known module
-          n._daemon_config.pop(m,None)                          # that is not active on the node, remove it
-
 # Check whether the modules defined on individual nodes are valid module names
 # and supported by the node device type
 #

netsim/providers/__init__.py

@@ -41,19 +41,29 @@ def get_template_path(self) -> str:
   def get_full_template_path(self) -> str:
     return str(_files.get_moddir()) + '/' + self.get_template_path()
 
-  def find_extra_template(self, node: Box, fname: str) -> typing.Optional[str]:
-    path_suffix = [ node.device ]
-    path_prefix = [ '.', self.get_full_template_path() ]
+  def find_extra_template(self, node: Box, fname: str, topology: Box) -> typing.Optional[str]:
+    if fname in node.get('config',[]):                    # Are we dealing with extra-config template?
+      path_prefix = topology.defaults.paths.custom.dirs
+      path_suffix = [ fname ]
+      fname = node.device
+    else:
+      path_suffix = [ node.device ]
+      path_prefix = topology.defaults.paths.templates.dirs + [ self.get_full_template_path() ]
 
-    if node.get('_daemon',False):
-      if '_daemon_parent' in node:
-        path_suffix.append(node._daemon_parent)
-      path_prefix.append(str(_files.get_moddir() / 'daemons'))
+      if node.get('_daemon',False):
+        if '_daemon_parent' in node:
+          path_suffix.append(node._daemon_parent)
+        path_prefix.append(str(_files.get_moddir() / 'daemons'))
 
     path = [ pf + "/" + sf for pf in path_prefix for sf in path_suffix ]
     if log.debug_active('clab'):
       print(f'Searching for {fname}.j2 in {path}')
-    return _files.find_file(fname+'.j2',path)
+
+    found_file = _files.find_file(fname+'.j2',path)
+    if log.debug_active('clab'):
+      print(f'Found file: {found_file}')
+
+    return found_file
 
   def get_output_name(self, fname: typing.Optional[str], topology: Box) -> str:
     if fname:
@@ -110,9 +120,10 @@ def create_extra_files_mappings(
     cur_binds = node.get(f'{self.provider}.{outkey}',[])
     bind_dict = filemaps.mapping_to_dict(cur_binds)
     for file,mapping in map_dict.items():
+      file = file.replace('@','.')
       if file in bind_dict:
         continue
-      if not self.find_extra_template(node,file):
+      if not self.find_extra_template(node,file,topology):
         log.error(
           f"Cannot find template {file}.j2 for extra file {self.provider}.{inkey}.{file} on node {node.name}",
           category=log.IncorrectValue,
@@ -143,7 +154,7 @@ def create_extra_files(
       if not out_folder in file:                  # Skip files that are not mapped into the temporary provider folder
         continue
       file_name = file.replace(out_folder+"/","")
-      template_name = self.find_extra_template(node,file_name)
+      template_name = self.find_extra_template(node,file_name,topology)
       if template_name:
         node_data = node + { 'hostvars': topology.nodes }
         if '/' in file_name:                      # Create subdirectory in out_folder if needed
@@ -248,6 +259,9 @@ def execute(hook: str, topology: Box) -> None:
   p_module = get_provider_module(topology,topology.provider)
   p_module.call(hook,topology)
 
+  for node in topology.nodes.values():
+    execute_node(f'node_{hook}',node,topology)
+
 """
 Execute a node-level provider hook
 """

netsim/providers/clab.py

@@ -103,15 +103,16 @@ class Containerlab(_Provider):
 
   def augment_node_data(self, node: Box, topology: Box) -> None:
     node.hostname = "clab-%s-%s" % (topology.name,node.name)
+    node_fp = get_forwarded_ports(node,topology)
+    if node_fp:
+      add_forwarded_ports(node,node_fp)
+
+  def node_post_transform(self, node: Box, topology: Box) -> None:
     add_daemon_filemaps(node,topology)
     normalize_clab_filemaps(node)
 
     self.create_extra_files_mappings(node,topology)
 
-    node_fp = get_forwarded_ports(node,topology)
-    if node_fp:
-      add_forwarded_ports(node,node_fp)
-
   def post_configuration_create(self, topology: Box) -> None:
     for n in topology.nodes.values():
       if n.get('clab.binds',None):

tests/integration/bgp/session/01-password.yml

@@ -0,0 +1,34 @@
+message:
+  This lab tests the BGP MD5 password functionality. The EBGP session
+  between the probe and the lab device should be established.
+
+plugin: [ bgp.session ]
+module: [ bgp ]
+defaults.paths.validate: topology:../validate
+
+groups:
+  probes:
+    device: frr
+    provider: clab
+    members: [ x1 ]
+
+defaults.bgp.as: 65000
+
+nodes:
+  dut:
+  x1:
+    bgp.as: 65100
+
+links:
+- dut:
+  x1:
+  bgp.password: Secret
+
+validate:
+  wait:
+    description: Wait for EBGP sessions to come up
+    wait: 3
+  session:
+    description: Check EBGP sessions with DUT
+    nodes: [ x1 ]
+    plugin: bgp_neighbor(node.bgp.neighbors,'dut')