Icinga2 Object Action Module Performance Issue

[lucagubler]

Hi all

I'm facing some performance issues with the icinga2_object.py action module and the task roles/icinga2/tasks/objects.yml at line 26 (unfortunately it doesn't have a name).

Problem Statement

1st Issue

I think one issue arises because we call this action plugin with a loop. So if we have n Icinga objects defined, we call this plugin n times.

2nd Issue

The second issue is that for each Icinga object we created, we simply copy it to the target host using the copy_action. The copy module is quite slow and since we copy each file individually, execution time increases significantly.

Possible Solutions

Here are some possible solutions

1st Issue

Instead of calling the icinga2_object.py action module from within a loop, we could call it just once and pass the whole tmp_objects. This means that we only call this plugin once and calculate all Icinga objects within that call. I was able to test that locally and the Ansible execution time was 50% faster.

2nd Issue

Here I'm not quite sure how this could be solved. I'm not even 100% sure if I fully understand the existing code. But basically we create all config fragments locally on our Ansible control node. Once everything is ready, we copy each file individually to the Icinga2 Host using the add custom config to assemble task in roles/icinga2/tasks/objects.yml. Each Icinga object is a file, so this could quickly scale up to several thousand files that need to be copied. Wouldn't it be faster and easier if we generate the fragments directly on the Icinga2 Host? This way we don't even have to copy them...

I hope my problem statement is clear. I'm looking forward to your comments and ideas how we can improve this :)

[Donien]

Hi,

thanks for your input!
A colleague and I actually talked about this just yesterday. I also started changing the icinga2_object action plugin to:

• Loop over the objects internally
• Merge config objects that got to the same file in RAM
• Write once per file

For what I am trying to change I think it would be best to get rid off the order key. This should not be a problem since Icinga 2 does not actually care whether you define the template or the object that imports the template first, for example.

Not writing a file per destination and per order would reduce the amount of write operations.
Additionally, if we decided to get rid off specific file destinations, we could reduce the time further I believe.

What I mean by file destinations: Currently you can choose whatever file you wish for the object to end up in.
If we make it more static, we can for example put all objects of a certain type (hosts.conf) in a destination directory. Basically: All hosts that go to local.d/ will end up in local.d/hosts.conf.

You could of course still use e.g. local.d/linux/ and local.d/windows/ if you really wanted separation here.

Once I rework the icinga2 role to work with the new approach completely, I will share the PoC.

Side note - 2nd Issue: icinga2_object actually does create the files on the target host directly. They are just located at /var/tmp/icinga/ first, for verification.
I have to look into the whole logic more deeply myself though.

[lucagubler]

Hi Donien

I added all my changes and created a PR #339. I tested it locally and it's working as expected. However, I'm not sure if it's 100% finished. It mainly fixes the 1st issue. I hope this helps :)