Enhance audit log

[stonezdj]

No description provided.

[Vad1mo]

This proposal not only introduces a new auditlogo but also add much more events.

• The volume of transactions that the new system can cause in a high pull/push environments. One crucial part of the enhanced auditlog should be the decoupling of event producer action and event writing. In other words, an artifact pull/push should not be affects by the eventlog.

• The Database is currently the bottelneck of Harbor, especially on large installations, with many artifacts, polcies, voulnerability data. Reducing the pressure on the Database should be priority in the new auditlogs. Adding Redis here will not improve the situation, unless we use/implement a Write-Back Caching strategy.

I suggest also making the auditlog plugable, meaning that in the future it would be possible to add other types of backends apart from postgres, like opentelemtry or loki..

---

I am looking into few options and Unlogged tables directly or in combination with stored procedures can provide a huge improvement in write performance.

[stonezdj]

This proposal not only introduces a new auditlogo but also add much more events.

• The volume of transactions that the new system can cause in a high pull/push environments. One crucial part of the enhanced auditlog should be the decoupling of event producer action and event writing. In other words, an artifact pull/push should not be affects by the eventlog.
• The Database is currently the bottelneck of Harbor, especially on large installations, with many artifacts, polcies, voulnerability data. Reducing the pressure on the Database should be priority in the new auditlogs. Adding Redis here will not improve the situation, unless we use/implement a Write-Back Caching strategy.

I suggest also making the auditlog plugable, meaning that in the future it would be possible to add other types of backends apart from postgres, like opentelemtry or loki..

I am looking into few options and Unlogged tables directly or in combination with stored procedures can provide a huge improvement in write performance.

Previous audit log has the log forward option, it can avoid log in the database

[Vad1mo]

Previous audit log has the log forward option, it can avoid log in the database

The problem with the existing approach is that when this is enabled. Audit logs aren't displayed in Harbor anymore. All events are forwarded to syslog.

What I had in mind was had the option to implement a different storage backend instead of the postgres.

[stonezdj]

Previous audit log has the log forward option, it can avoid log in the database

The problem with the existing approach is that when this is enabled. Audit logs aren't displayed in Harbor anymore. All events are forwarded to syslog.

What I had in mind was had the option to implement a different storage backend instead of the postgres.

When you configured forward log to LogInsight or ELK, you should search the audit log in the ELK's query interface.

[reasonerjt]

The problem with the existing approach is that when this is enabled. Audit logs aren't displayed in Harbor anymore. All events are forwarded to syslog.

What I had in mind was had the option to implement a different storage backend instead of the postgres.

@Vad1mo This is essentially a query module in Harbor that works with different backends.
The requirement to provide a pluggable query module in Harbor for audit logs should be tracked separately.
And as Stone has suggested, when the user chooses to store Harbor's audit logs to a 3rd party system, normally he will query the information from the 3rd party system instead from Harbor.

[wy65701436]

lgtm

[stonezdj]

Previous audit log has the log forward option, it can avoid log in the database

The problem with the existing approach is that when this is enabled. Audit logs aren't displayed in Harbor anymore. All events are forwarded to syslog.

What I had in mind was had the option to implement a different storage backend instead of the postgres.

Replace postgres is a big task, we are focusing on urgent requirements, have no bandwidth to handle so far.