Diode overriding unwanted data

[dga-nagra]

Hi,

First, thank you for diode. Data ingestion is indeed a must have for most people, including us.

We were trying to use diode with diode-napalm-agent and we got the following unwanted change:

As we can see, the following fields were overriden:

• rack, role, tenant, location, position, vitual_chassis
  Here, the change is unexpected as the code doesn't serialize any of these values:

And I think it is the same for the site field if we don't define its value in the discovery policy.

• platform: Found on the device, but replacing a more accurate information.
  Here, we expect diode to do the change, but he choose to replace "IOS-XE" with "IOS" which is less precise.

We expect Diode to ignore fields that are not provided instead of overriding them with null, and keep the more exact value
when possible.
Another behavior is that, if we don't provide the site, Diode will not find the device and create a new one a new site called undefined. Again, we expected Diode to find an existing device in any existing site even if not provided in the discovery policies.

Is there an option to change these behaviors ?
Basically, we expect Diode to only override information that are provided explicitly.

Thank you in advance for your response. We are also willing to help if you need someone to test or implement something.

[mfiedorowicz]

Thanks @dga-nagra, before we dig deeper with troubleshooting, could you share the following:

• diode server version
• diode netbox plugin version

[dga-nagra]

Hi @mfiedorowicz

• netboxlabs/diode-ingester: c4ad17fc4981
• netboxlabs/diode-reconciler: 4f4bae7078bf
• netboxlabs-diode-netbox-plugin==0.3.0 (We are still on Netbox 4.0.x)

For diode images, I wasn't able to find the corresponding tag as I was on latest. It should be between 0.4.0 <= diode <= 0.5.0 seeing their release dates.

Let me know if you need anything else. I guess I should retry with a newer version ?

[dga-nagra]

Hi,

If the information is usefull, I with the latest version of diode ingester and reconcilier as of today and the issue is still here.

Also, I tried to not import the devices information hoping it would just create the interfaces/Prefixes/IPAddresses on the existing device without trying to update it.

But it does try to create a new device anyway:

From previous observations, Diode need to know the site where to find the device. Here the Device instance is not created, so it it not retrieving the site and therefore create a new device in site "undefined".
It would maybe work if the search made by Diode was not depending on the site, but only using the site in case of ambiguity.

Let me know if I can help.
Have a nice day.

[mfiedorowicz]

Hi @dga-nagra, thanks for sharing more information. Diode works around NetBox constraints and i.e. device requires site, and interface requires device, so they way we go around it is to create required but not provided nested objects as undefined so you can still succeed with an interface ingestion, even when you don't have all required details. Hope that helps.

[dga-nagra]

Hi @mfiedorowicz

Sorry I thought I already responded last week..

The reason why the site is required on creation is understood, this is not the issue.
There are multiple issues, Diode should:

• only update data that were explicitly passed to it. In the screenshot, we see that it overwritten chassis but this was never provided to Device(...) in the first place.
• be able to only update.
  As most people, we have the data in Netbox already, so we really only want to update and not create.
• consider an absent value differently from a null value.
  We don't want to override the site/tenant/chassis/platform/... of existing devices
  (or make the site/tenant/... required in the discovery policy and allow null as a value)
• There should be a way to update(/create) interfaces only without impacting the device.
  In my last screen shot, I tried to remove the Entity(Device(...)) hoping it would only consider the interfaces/ip/prefixes, but it did create a new device.

In short: better control over consolidation (e.g. how we find the matching device), the operations we want (update_or_create, update_only, ...) and the data to use (absent != null)

Right now, we are unable to use Diode because of these reasons. This could be solved with a script launched from Netbox, but we are willing to support Diode project and would be glad to contribute. Let me know if I can help by any mean.

Have a nice day.

[dga-nagra]

Hi again @mfiedorowicz

I was wondering if there was any advancement on this issue?

We are reaching some deadlines and, if we are not able to fixe these issues soon, we will need to consider writing a script..
This would be a shame because we were really enthusiastic about using Diode and trying to help it grow.

Thank you in advance for the feedback.

[mfiedorowicz]

Hi @dga-nagra, we're looking into this issue. Would it possible to extract example ingestion input from i.e. diode-reconciler container logs to help us reproduce this behaviour leading to i.e. overriding chassis object to null?