Merge pull request #275 from nautobot/patch-fix_cfs

Fix CustomField Bug

docs/user/integrations/ipfabric.md

@@ -35,8 +35,8 @@ Now back to running the job. Let's click on **Sync Now**.
 There are several options available.
 
 - **Debug**: Enables more verbose logging that can be useful for troubleshooting synchronization issues.
-- **Safe Delete Mode**: Delete operations changes the object status to a predefined value (configurable via settings) and tags the object with `ssot-safe-delete` tag.
-- **Sync Tagged Only**: Only load Nautobot data into DiffSync adapters that's been tagged with `ssot-synced-from-ipfabric` tag.
+- **Safe Delete Mode**: Delete operations changes the object status to a predefined value (configurable via settings) and tags the object with `SSoT Safe Delete` Tag.
+- **Sync Tagged Only**: Only load Nautobot data into DiffSync adapters that has the `SSoT Synced from IPFabric` Tag.
 - **Dry run**: This will only report the difference between the source and destination without synchronization.
 - **Site Filter**: Filter the data loaded into DiffSync by a top level location of a specified Site.
 
@@ -104,21 +104,21 @@ Currently, this integration will provide the ability to sync the following IP Fa
 
 ## Safe Delete Mode
 
-By design, a Nautobot SSoT app using DiffSync will Create, Update or Delete when synchronizing two data sources. However, this may not always be what we want to happen with our Source of Truth (Nautobot). A job configuration option is available and enabled by default to prevent deleting objects from the database and instead, update the `Status` of said object alongside assigning a default tag, `ssot-safe-delete`. For example, if an additional snapshot is created from IPFabric, synchronized with Nautobot and, it just so happens that a device was unreachable, down for maintenance, etc., This doesn't `always` mean that our Source of Truth should delete this object, but we may need to bring attention to this matter. We let you decide what should happen. One thing to note is that some of the objects will auto recover from the changed status if a new job shows the object is present. However, currently, IP addresses and Interfaces will not auto-update to remove the `ssot-safe-delete` tag. The user is responsible for reviewing and updating accordingly. Safe delete tagging of objects works in an idempotent way. If an object has been tagged already, the custom field defining the last update will not be updated with a new sync date from IPFabric. So, if you re-run your sync job days apart and, you'd expect the date to change, but the object has been flagged as safe to delete; you will not see an updated date on the object custom field unless the status changed, in which case the tag (depending on the object) would be removed followed by updating the last date of sync.
+By design, a Nautobot SSoT app using DiffSync will Create, Update or Delete when synchronizing two data sources. However, this may not always be what we want to happen with our Source of Truth (Nautobot). A job configuration option is available and enabled by default to prevent deleting objects from the database and instead, update the `Status` of said object alongside assigning a default Tag, `SSoT Safe Delete`. For example, if an additional snapshot is created from IPFabric, synchronized with Nautobot and, it just so happens that a device was unreachable, down for maintenance, etc., This doesn't `always` mean that our Source of Truth should delete this object, but we may need to bring attention to this matter. We let you decide what should happen. One thing to note is that some of the objects will auto recover from the changed status if a new job shows the object is present. However, currently, IP addresses and Interfaces will not auto-update to remove the `SSoT Safe Delete` Tag. The user is responsible for reviewing and updating accordingly. Safe delete tagging of objects works in an idempotent way. If an object has been tagged already, the custom field defining the last update will not be updated with a new sync date from IPFabric. So, if you re-run your sync job days apart and, you'd expect the date to change, but the object has been flagged as safe to delete; you will not see an updated date on the object custom field unless the status changed, in which case the tag (depending on the object) would be removed followed by updating the last date of sync.
 
 The default status change of an object were to be `deleted` by SSoT DiffSync operations, will be specified below. These are the default transitions states, unless otherwise specified in the configuration options of the integration by a user.
 
 - Device -> Offline (Auto deletes tag upon recovery)
 - IPAddresses -> Deprecated (Does not auto-delete tag upon recovery)
 - VLAN -> Deprecated (Auto deletes tag upon recovery)
 - Site -> Decommissioning (Auto deletes tag upon recovery)
-- Interfaces -> Tagged with `ssot-safe-delete` (Does not auto-delete tag upon recovery)
+- Interfaces -> Tagged with `SSoT Safe Delete` (Does not auto-delete Tag upon recovery)
 
 If you would like to change the default status change value, ensure you provide a valid status name available for the referenced object. Not all objects share the same `Status`.
 
 ![Safe Delete](../../images/ipfabric-safe-delete.png)
 
-An example object that's been modified by SSoT App and tagged as `ssot-safe-delete` and `ssot-synced-from-ipfabric`. Notice the Status and child object, IPAddress has also changed to Deprecated and, it's status changed and tagged as well.
+An example object that's been modified by SSoT App and tagged as `SSoT Safe Delete` and `SSoT Synced from IPFabric`. Notice the Status and child object, IPAddress has also changed to Deprecated and, it's status changed and tagged as well.
 
 ![Safe Delete Address](../../images/ipfabric-safe-delete-ipaddress.png)
 

nautobot_ssot/integrations/device42/utils/nautobot.py

@@ -222,15 +222,16 @@ def update_custom_fields(new_cfields: dict, update_obj: object):
             )
             removed_cf.delete()
     for new_cf, new_cf_dict in new_cfields.items():
+        new_key = new_cf_dict["key"].replace(" ", "_").replace("-", "_")
         if new_cf not in current_cf:
             _cf_dict = {
-                "key": new_cf_dict["key"],
+                "key": new_key,
                 "type": CustomFieldTypeChoices.TYPE_TEXT,
                 "label": new_cf_dict["key"],
             }
             field, _ = CustomField.objects.get_or_create(key=_cf_dict["key"], defaults=_cf_dict)
             field.content_types.add(ContentType.objects.get_for_model(type(update_obj)).id)
-        update_obj.custom_field_data.update({new_cf_dict["key"]: new_cf_dict["value"]})
+        update_obj.custom_field_data.update({new_key: new_cf_dict["value"]})
 
 
 def verify_circuit_type(circuit_type: str) -> CircuitType:

nautobot_ssot/integrations/infoblox/diffsync/adapters/nautobot.py

@@ -30,7 +30,7 @@ class NautobotMixin:
 
     def tag_involved_objects(self, target):
         """Tag all objects that were successfully synced to the target."""
-        # The ssot-synced-to-infoblox tag *should* have been created automatically during plugin installation
+        # The ssot_synced_to_infoblox tag *should* have been created automatically during plugin installation
         # (see nautobot_ssot/integrations/infoblox/signals.py) but maybe a user deleted it inadvertently, so be safe:
         tag, _ = Tag.objects.get_or_create(
             name="SSoT Synced to Infoblox",
@@ -40,10 +40,10 @@ def tag_involved_objects(self, target):
                 "color": TAG_COLOR,
             },
         )
-        # Ensure that the "ssot-synced-to-infoblox" custom field is present; as above, it *should* already exist.
+        # Ensure that the "ssot_synced_to_infoblox" custom field is present; as above, it *should* already exist.
         custom_field, _ = CustomField.objects.get_or_create(
             type=CustomFieldTypeChoices.TYPE_DATE,
-            name="ssot-synced-to-infoblox",
+            name="ssot_synced_to_infoblox",
             defaults={
                 "label": "Last synced to Infoblox on",
             },
@@ -132,8 +132,8 @@ def load_prefixes(self):
         default_cfs = get_default_custom_fields(cf_contenttype=ContentType.objects.get_for_model(Prefix))
         for prefix in all_prefixes:
             self.prefix_map[str(prefix.prefix)] = prefix.id
-            if "ssot-synced-to-infoblox" in prefix.custom_field_data:
-                prefix.custom_field_data.pop("ssot-synced-to-infoblox")
+            if "ssot_synced_to_infoblox" in prefix.custom_field_data:
+                prefix.custom_field_data.pop("ssot_synced_to_infoblox")
             current_vlans = get_prefix_vlans(prefix=prefix)
             _prefix = self.prefix(
                 network=str(prefix.prefix),
@@ -169,8 +169,8 @@ def load_ipaddresses(self):
                 )
                 continue
 
-            if "ssot-synced-to-infoblox" in ipaddr.custom_field_data:
-                ipaddr.custom_field_data.pop("ssot-synced-to-infoblox")
+            if "ssot_synced_to_infoblox" in ipaddr.custom_field_data:
+                ipaddr.custom_field_data.pop("ssot_synced_to_infoblox")
             _ip = self.ipaddress(
                 address=addr,
                 prefix=str(prefix),
@@ -191,8 +191,8 @@ def load_vlangroups(self):
         default_cfs = get_default_custom_fields(cf_contenttype=ContentType.objects.get_for_model(VLANGroup))
         for grp in VLANGroup.objects.all():
             self.vlangroup_map[grp.name] = grp.id
-            if "ssot-synced-to-infoblox" in grp.custom_field_data:
-                grp.custom_field_data.pop("ssot-synced-to-infoblox")
+            if "ssot_synced_to_infoblox" in grp.custom_field_data:
+                grp.custom_field_data.pop("ssot_synced_to_infoblox")
             _vg = self.vlangroup(
                 name=grp.name,
                 description=grp.description,
@@ -210,8 +210,8 @@ def load_vlans(self):
             if vlan.group.name not in self.vlan_map:
                 self.vlan_map[vlan.vlan_group.name] = {}
             self.vlan_map[vlan.vlan_group.name][vlan.vid] = vlan.id
-            if "ssot-synced-to-infoblox" in vlan.custom_field_data:
-                vlan.custom_field_data.pop("ssot-synced-to-infoblox")
+            if "ssot_synced_to_infoblox" in vlan.custom_field_data:
+                vlan.custom_field_data.pop("ssot_synced_to_infoblox")
             _vlan = self.vlan(
                 vid=vlan.vid,
                 name=vlan.name,

nautobot_ssot/integrations/infoblox/signals.py

@@ -43,7 +43,7 @@ def nautobot_database_ready_callback(sender, *, apps, **kwargs):  # pylint: disa
     )
     custom_field, _ = CustomField.objects.get_or_create(
         type=CustomFieldTypeChoices.TYPE_DATE,
-        key="ssot-synced-to-infoblox",
+        key="ssot_synced_to_infoblox",
         defaults={
             "label": "Last synced to Infoblox on",
         },

nautobot_ssot/integrations/infoblox/utils/diffsync.py

@@ -89,7 +89,7 @@ def get_default_custom_fields(cf_contenttype: ContentType) -> dict:
     customfields = CustomField.objects.filter(content_types=cf_contenttype)
     default_cfs = {}
     for customfield in customfields:
-        if customfield.key != "ssot-synced-to-infoblox":
+        if customfield.key != "ssot_synced_to_infoblox":
             if customfield.key not in default_cfs:
                 default_cfs[customfield.key] = None
     return default_cfs

nautobot_ssot/integrations/ipfabric/constants.py

@@ -22,3 +22,4 @@
 SAFE_DELETE_LOCATION_STATUS = CONFIG.get("ipfabric_safe_delete_location_status", "Decommissioning")
 SAFE_DELETE_VLAN_STATUS = CONFIG.get("ipfabric_safe_delete_vlan_status", "Deprecated")
 SAFE_DELETE_IPADDRESS_STATUS = CONFIG.get("ipfabric_safe_delete_ipaddress_status", "Deprecated")
+LAST_SYNCHRONIZED_CF_NAME = "last_synced_from_sor"

nautobot_ssot/integrations/ipfabric/diffsync/adapter_nautobot.py

@@ -207,7 +207,7 @@ def load_data(self):
                     location = self.location(
                         diffsync=self,
                         name=location_record.name,
-                        site_id=location_record.custom_field_data.get("ipfabric-site-id"),
+                        site_id=location_record.custom_field_data.get("ipfabric_site_id"),
                         status=location_record.status.name,
                     )
                 except AttributeError:

nautobot_ssot/integrations/ipfabric/diffsync/diffsync_models.py

@@ -15,7 +15,7 @@
 from nautobot.extras.models.statuses import Status
 from nautobot.ipam.models import VLAN
 from nautobot.core.choices import ColorChoices
-
+from nautobot_ssot.integrations.ipfabric.constants import LAST_SYNCHRONIZED_CF_NAME
 import nautobot_ssot.integrations.ipfabric.utilities.nbutils as tonb_nbutils
 from nautobot_ssot.integrations.ipfabric.constants import (
     DEFAULT_DEVICE_ROLE,
@@ -76,12 +76,12 @@ def safe_delete(self, nautobot_object: Any, safe_delete_status: Optional[str] =
                 # No exception raised for empty iterator, safe to do this any
                 if not any(obj_tag for obj_tag in object_tags if obj_tag.name == ssot_safe_tag.name):
                     nautobot_object.tags.add(ssot_safe_tag)
-                    logger.warning(f"Tagging {nautobot_object} with `ssot-safe-delete`.")
+                    logger.warning(f"Tagging {nautobot_object} with `SSoT Safe Delete`.")
                     update = True
             if update:
-                tonb_nbutils.tag_object(nautobot_object=nautobot_object, custom_field="ssot-synced-from-ipfabric")
+                tonb_nbutils.tag_object(nautobot_object=nautobot_object, custom_field=LAST_SYNCHRONIZED_CF_NAME)
             else:
-                logger.warning(f"{nautobot_object} has previously been tagged with `ssot-safe-delete`. Skipping...")
+                logger.warning(f"{nautobot_object} has previously been tagged with `SSoT Safe Delete`. Skipping...")
 
         return self
 
@@ -120,7 +120,7 @@ def update(self, attrs):
         """Update Location Object in Nautobot."""
         location = NautobotLocation.objects.get(name=self.name)
         if attrs.get("site_id"):
-            location.custom_field_data["ipfabric-site-id"] = attrs.get("site_id")
+            location.custom_field_data["ipfabric_site_id"] = attrs.get("site_id")
             location.validated_save()
         if attrs.get("status") == "Active":
             safe_delete_tag, _ = Tag.objects.get_or_create(name="SSoT Safe Delete")
@@ -129,7 +129,7 @@ def update(self, attrs):
             device_tags = location.tags.filter(pk=safe_delete_tag.pk)
             if device_tags.exists():
                 location.tags.remove(safe_delete_tag)
-        tonb_nbutils.tag_object(nautobot_object=location, custom_field="ssot-synced-from-ipfabric")
+        tonb_nbutils.tag_object(nautobot_object=location, custom_field=LAST_SYNCHRONIZED_CF_NAME)
         return super().update(attrs)
 
 
@@ -198,7 +198,7 @@ def create(cls, diffsync, ids, attrs):
         )
         try:
             # Validated save happens inside of tag_objet
-            tonb_nbutils.tag_object(nautobot_object=new_device, custom_field="ssot-synced-from-ipfabric")
+            tonb_nbutils.tag_object(nautobot_object=new_device, custom_field=LAST_SYNCHRONIZED_CF_NAME)
         except ValidationError as error:
             message = f"Unable to create device: {ids['name']}. A validation error occured. Enable debug for more information."
             if diffsync.job.debug:
@@ -246,7 +246,7 @@ def update(self, attrs):
                     role_name=attrs.get("role", DEFAULT_DEVICE_ROLE), role_color=DEFAULT_DEVICE_ROLE_COLOR
                 )
                 _device.role = device_role_object
-            tonb_nbutils.tag_object(nautobot_object=_device, custom_field="ssot-synced-from-ipfabric")
+            tonb_nbutils.tag_object(nautobot_object=_device, custom_field=LAST_SYNCHRONIZED_CF_NAME)
             # Call the super().update() method to update the in-memory DiffSyncModel instance
             return super().update(attrs)
         except NautobotDevice.DoesNotExist:
@@ -384,7 +384,7 @@ def update(self, attrs):  # pylint: disable=too-many-branches
                         device.primary_ip6 = interface_obj
                         device.save()
             interface.save()
-            tonb_nbutils.tag_object(nautobot_object=interface, custom_field="ssot-synced-from-ipfabric")
+            tonb_nbutils.tag_object(nautobot_object=interface, custom_field=LAST_SYNCHRONIZED_CF_NAME)
             return super().update(attrs)
 
         except NautobotDevice.DoesNotExist:
@@ -447,7 +447,7 @@ def update(self, attrs):
         if attrs.get("description"):
             vlan.description = vlan.description
 
-        tonb_nbutils.tag_object(nautobot_object=vlan, custom_field="ssot-synced-from-ipfabric")
+        tonb_nbutils.tag_object(nautobot_object=vlan, custom_field=LAST_SYNCHRONIZED_CF_NAME)
 
 
 Location.update_forward_refs()

nautobot_ssot/integrations/ipfabric/jobs.py

@@ -111,7 +111,7 @@ class IpFabricDataSource(DataSource):
     sync_ipfabric_tagged_only = BooleanVar(
         default=True,
         label="Sync Tagged Only",
-        description="Only sync objects that have the 'ssot-synced-from-ipfabric' tag.",
+        description="Only sync objects that have the 'SSoT Synced from IPFabric' Tag.",
     )
     location_filter = OptionalObjectVar(
         description="Only sync Nautobot records belonging to a single Location. This does not filter IPFabric data.",

nautobot_ssot/integrations/ipfabric/signals.py

@@ -76,6 +76,7 @@ def nautobot_database_ready_callback(sender, *, apps, **kwargs):  # pylint: disa
     loc_type.content_types.add(ContentType.objects.get_for_model(apps.get_model("ipam", "Prefix")))
     loc_type.content_types.add(ContentType.objects.get_for_model(VLAN))
     synced_from_models = [Device, DeviceType, Interface, Manufacturer, Location, VLAN, Role, IPAddress]
-    create_custom_field("ssot-synced-from-ipfabric", "Last synced from IPFabric on", synced_from_models, apps=apps)
-    create_custom_field("ipfabric-site-id", "IPFabric Location ID", [Location], apps=apps, cf_type="type_text")
+    create_custom_field("system_of_record", "System of Record", synced_from_models, apps=apps, cf_type="type_text")
+    create_custom_field("last_synced_from_sor", "Last sync from System of Record", synced_from_models, apps=apps)
+    create_custom_field("ipfabric_site_id", "IPFabric Location ID", [Location], apps=apps, cf_type="type_text")
     create_custom_field("ipfabric_type", "IPFabric Type", [Role], apps=apps, cf_type="type_text")