[FR] Add Debian/Ubuntu errata information via ATIX AG errata_server

[pandel]

Hi!

It would be nice if you could add the Errata data for Debian/Ubuntu, which can be provided via the errata_server in JSON from ATIX AG:

https://github.com/ATIX-AG/errata_server
https://github.com/ATIX-AG/errata_parser

I made a docker stack to fire up a locally hosted solution in no time:

https://github.com/pandel/docker-atix-errata

You find the API description at the end of the README.md here.

Regards,
Holger

[furlongm]

There is native support for Debian errata in #633 if you want to test that branch. Support for Ubuntu errata is also in the works.

[pandel]

AH! Many thanks for the hint! Debian covers 99.9% of all I need atm actually.

I'd love to try, but after starting the container with a fresh (empty) setup the server returns the following error multiple times and does not come up:

/app # ./manage.py makemigrations
Traceback (most recent call last):
  File "/app/manage.py", line 39, in <module>
    execute_from_command_line(sys.argv)
  File "/usr/local/lib/python3.9/site-packages/django/core/management/__init__.py", line 419, in execute_from_command_line
    utility.execute()
  File "/usr/local/lib/python3.9/site-packages/django/core/management/__init__.py", line 413, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/usr/local/lib/python3.9/site-packages/django/core/management/base.py", line 354, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/usr/local/lib/python3.9/site-packages/django/core/management/base.py", line 398, in execute
    output = self.handle(*args, **options)
  File "/usr/local/lib/python3.9/site-packages/django/core/management/base.py", line 89, in wrapped
    res = handle_func(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/django/core/management/commands/migrate.py", line 92, in handle
    executor = MigrationExecutor(connection, self.migration_progress_callback)
  File "/usr/local/lib/python3.9/site-packages/django/db/migrations/executor.py", line 18, in __init__
    self.loader = MigrationLoader(self.connection)
  File "/usr/local/lib/python3.9/site-packages/django/db/migrations/loader.py", line 53, in __init__
    self.build_graph()
  File "/usr/local/lib/python3.9/site-packages/django/db/migrations/loader.py", line 259, in build_graph
    self.graph.validate_consistency()
  File "/usr/local/lib/python3.9/site-packages/django/db/migrations/graph.py", line 195, in validate_consistency
    [n.raise_error() for n in self.node_map.values() if isinstance(n, DummyNode)]
  File "/usr/local/lib/python3.9/site-packages/django/db/migrations/graph.py", line 195, in <listcomp>
    [n.raise_error() for n in self.node_map.values() if isinstance(n, DummyNode)]
  File "/usr/local/lib/python3.9/site-packages/django/db/migrations/graph.py", line 58, in raise_error
    raise NodeNotFoundError(self.error_message, self.key, origin=self.origin)
django.db.migrations.exceptions.NodeNotFoundError: Migration hosts.0004_remove_host_tags_host_tags dependencies reference nonexistent parent node ('taggit', '0006_rename_taggeditem_content_type_object_id_taggit_tagg_content_8fc721_idx')

[furlongm]

I added some fixes, can you try again?

How are you running it?

[pandel]

Hey! What exactly do you mean by "How are you running it?"

• during my tests, I always remove the Mysql DB and the Redis DB completely for a clean start
• using this as basic Docker stack: https://github.com/tigattack/Patchman-Docker
• I modified the Dockerfile to use the errata branch
• the (startup) container entry script is this: https://github.com/tigattack/Patchman-Docker/blob/main/configs/entry.sh

But I think, there is no problem here, it is working fine so far in principle...

Nevertheless, your changes made some progress!

Now, the main error message changed to:

MySQLdb.OperationalError: (1091, "Can't DROP COLUMN tags; check that it exists")

See first part of log:

No changes detected
Operations to perform:
  Synchronize unmigrated apps: admindocs, bootstrap3, celery, django_extensions, django_filters, humanize, messages, rest_framework, staticfiles, util
  Apply all migrations: admin, arch, auth, contenttypes, domains, hosts, modules, operatingsystems, packages, reports, repos, sessions, sites, tagging, taggit
Synchronizing apps without migrations:
  Creating tables...
    Running deferred SQL...
Running migrations:
  Applying contenttypes.0001_initial... OK
  Applying auth.0001_initial... OK
  Applying admin.0001_initial... OK
  Applying admin.0002_logentry_remove_auto_add... OK
  Applying admin.0003_logentry_add_action_flag_choices... OK
  Applying arch.0001_initial... OK
  Applying contenttypes.0002_remove_content_type_name... OK
  Applying auth.0002_alter_permission_name_max_length... OK
  Applying auth.0003_alter_user_email_max_length... OK
  Applying auth.0004_alter_user_username_opts... OK
  Applying auth.0005_alter_user_last_login_null... OK
  Applying auth.0006_require_contenttypes_0002... OK
  Applying auth.0007_alter_validators_add_error_messages... OK
  Applying auth.0008_alter_user_username_max_length... OK
  Applying auth.0009_alter_user_last_name_max_length... OK
  Applying auth.0010_alter_group_name_max_length... OK
  Applying auth.0011_update_proxy_permissions... OK
  Applying auth.0012_alter_user_first_name_max_length... OK
  Applying domains.0001_initial... OK
  Applying operatingsystems.0001_initial... OK
  Applying packages.0001_initial... OK
  Applying repos.0001_initial... OK
  Applying modules.0001_initial... OK
  Applying operatingsystems.0002_initial... OK
  Applying hosts.0001_initial... OK
  Applying hosts.0002_initial... OK
  Applying hosts.0003_host_modules... OK
Traceback (most recent call last):
  File "/app/manage.py", line 39, in <module>
    execute_from_command_line(sys.argv)
  File "/usr/local/lib/python3.9/site-packages/django/core/management/__init__.py", line 419, in execute_from_command_line
    utility.execute()
  File "/usr/local/lib/python3.9/site-packages/django/core/management/__init__.py", line 413, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/usr/local/lib/python3.9/site-packages/django/core/management/base.py", line 354, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/usr/local/lib/python3.9/site-packages/django/core/management/base.py", line 398, in execute
    output = self.handle(*args, **options)
  File "/usr/local/lib/python3.9/site-packages/django/core/management/base.py", line 89, in wrapped
    res = handle_func(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/django/core/management/commands/migrate.py", line 244, in handle
    post_migrate_state = executor.migrate(
  File "/usr/local/lib/python3.9/site-packages/django/db/migrations/executor.py", line 117, in migrate
    state = self._migrate_all_forwards(state, plan, full_plan, fake=fake, fake_initial=fake_initial)
  File "/usr/local/lib/python3.9/site-packages/django/db/migrations/executor.py", line 147, in _migrate_all_forwards
    state = self.apply_migration(state, migration, fake=fake, fake_initial=fake_initial)
  File "/usr/local/lib/python3.9/site-packages/django/db/migrations/executor.py", line 227, in apply_migration
    state = migration.apply(state, schema_editor)
  File "/usr/local/lib/python3.9/site-packages/django/db/migrations/migration.py", line 126, in apply
    operation.database_forwards(self.app_label, schema_editor, old_state, project_state)
  File "/usr/local/lib/python3.9/site-packages/django/db/migrations/operations/fields.py", line 104, in database_forwards
    schema_editor.add_field(
  File "/usr/local/lib/python3.9/site-packages/django/db/backends/mysql/schema.py", line 98, in add_field
    super().add_field(model, field)
  File "/usr/local/lib/python3.9/site-packages/django/db/backends/base/schema.py", line 487, in add_field
    if field.many_to_many and field.remote_field.through._meta.auto_created:
AttributeError: 'str' object has no attribute '_meta'
  Applying hosts.0004_remove_host_tags_host_tags...No changes detected
Operations to perform:
  Synchronize unmigrated apps: admindocs, bootstrap3, celery, django_extensions, django_filters, humanize, messages, rest_framework, staticfiles, util
  Apply all migrations: admin, arch, auth, contenttypes, domains, hosts, modules, operatingsystems, packages, reports, repos, sessions, sites, tagging, taggit
Synchronizing apps without migrations:
  Creating tables...
    Running deferred SQL...
Running migrations:
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File "/usr/local/lib/python3.9/site-packages/django/db/backends/mysql/base.py", line 73, in execute
    return self.cursor.execute(query, args)
  File "/usr/local/lib/python3.9/site-packages/MySQLdb/cursors.py", line 179, in execute
    res = self._query(mogrified_query)
  File "/usr/local/lib/python3.9/site-packages/MySQLdb/cursors.py", line 330, in _query
    db.query(q)
  File "/usr/local/lib/python3.9/site-packages/MySQLdb/connections.py", line 261, in query
    _mysql.connection.query(self, query)
MySQLdb.OperationalError: (1091, "Can't DROP COLUMN `tags`; check that it exists")

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/app/manage.py", line 39, in <module>
    execute_from_command_line(sys.argv)
  File "/usr/local/lib/python3.9/site-packages/django/core/management/__init__.py", line 419, in execute_from_command_line
    utility.execute()
  File "/usr/local/lib/python3.9/site-packages/django/core/management/__init__.py", line 413, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/usr/local/lib/python3.9/site-packages/django/core/management/base.py", line 354, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/usr/local/lib/python3.9/site-packages/django/core/management/base.py", line 398, in execute
    output = self.handle(*args, **options)
  File "/usr/local/lib/python3.9/site-packages/django/core/management/base.py", line 89, in wrapped
    res = handle_func(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/django/core/management/commands/migrate.py", line 244, in handle
    post_migrate_state = executor.migrate(
  File "/usr/local/lib/python3.9/site-packages/django/db/migrations/executor.py", line 117, in migrate
    state = self._migrate_all_forwards(state, plan, full_plan, fake=fake, fake_initial=fake_initial)
  File "/usr/local/lib/python3.9/site-packages/django/db/migrations/executor.py", line 147, in _migrate_all_forwards
    state = self.apply_migration(state, migration, fake=fake, fake_initial=fake_initial)
  File "/usr/local/lib/python3.9/site-packages/django/db/migrations/executor.py", line 227, in apply_migration
    state = migration.apply(state, schema_editor)
  File "/usr/local/lib/python3.9/site-packages/django/db/migrations/migration.py", line 126, in apply
    operation.database_forwards(self.app_label, schema_editor, old_state, project_state)
  File "/usr/local/lib/python3.9/site-packages/django/db/migrations/operations/fields.py", line 170, in database_forwards
    schema_editor.remove_field(from_model, from_model._meta.get_field(self.name))
  File "/usr/local/lib/python3.9/site-packages/django/db/backends/base/schema.py", line 559, in remove_field
    self.execute(sql)
  File "/usr/local/lib/python3.9/site-packages/django/db/backends/base/schema.py", line 145, in execute
    cursor.execute(sql, params)
  File "/usr/local/lib/python3.9/site-packages/django/db/backends/utils.py", line 98, in execute
    return super().execute(sql, params)
  File "/usr/local/lib/python3.9/site-packages/django/db/backends/utils.py", line 66, in execute
    return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
  File "/usr/local/lib/python3.9/site-packages/django/db/backends/utils.py", line 75, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File "/usr/local/lib/python3.9/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File "/usr/local/lib/python3.9/site-packages/django/db/utils.py", line 90, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/usr/local/lib/python3.9/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File "/usr/local/lib/python3.9/site-packages/django/db/backends/mysql/base.py", line 73, in execute
    return self.cursor.execute(query, args)
  File "/usr/local/lib/python3.9/site-packages/MySQLdb/cursors.py", line 179, in execute
    res = self._query(mogrified_query)
  File "/usr/local/lib/python3.9/site-packages/MySQLdb/cursors.py", line 330, in _query
    db.query(q)
  File "/usr/local/lib/python3.9/site-packages/MySQLdb/connections.py", line 261, in query
    _mysql.connection.query(self, query)
django.db.utils.OperationalError: (1091, "Can't DROP COLUMN `tags`; check that it exists")
  Applying hosts.0004_remove_host_tags_host_tags...No changes detected
Operations to perform:
  Synchronize unmigrated apps: admindocs, bootstrap3, celery, django_extensions, django_filters, humanize, messages, rest_framework, staticfiles, util
  Apply all migrations: admin, arch, auth, contenttypes, domains, hosts, modules, operatingsystems, packages, reports, repos, sessions, sites, tagging, taggit
Synchronizing apps without migrations:
  Creating tables...
    Running deferred SQL...
Running migrations:
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File "/usr/local/lib/python3.9/site-packages/django/db/backends/mysql/base.py", line 73, in execute
    return self.cursor.execute(query, args)
  File "/usr/local/lib/python3.9/site-packages/MySQLdb/cursors.py", line 179, in execute
    res = self._query(mogrified_query)
  File "/usr/local/lib/python3.9/site-packages/MySQLdb/cursors.py", line 330, in _query
    db.query(q)
  File "/usr/local/lib/python3.9/site-packages/MySQLdb/connections.py", line 261, in query
    _mysql.connection.query(self, query)
MySQLdb.OperationalError: (1091, "Can't DROP COLUMN `tags`; check that it exists")
...

[furlongm]

Ok, that taggit commit needs some work.. I've removed it from the errata branch, as it's not necessary there. Can you try again?

[pandel]

Many thanks! The server is starting now and seems to work so far. I then added a Debian client and issued the following command: patchman -r

The server starts to download the correct package repos, but I get one message:~
~/usr/local/lib/python3.9/site-packages/debian/deb822.py:726: UserWarning: Parsing of Deb822 data with python3-apt's apt_pkg was requested but this package is not importable. Is python3-apt installed?

Do I have to add something to the requirements.txt perhaps?

EDIT:
My current Dockerfile was based on Alpine Linux. I did rewrite the image to use Debian bookworm as base image, so that I could add python3-apt. But sadly, the error message is still the same... I'll try to figure out what's happening.

[furlongm]

That is just a warning, it can be ignored. python-apt was yanked from pypi with no plans to reinstate it, so it can only be installed from deb:

https://answers.launchpad.net/ubuntu/+source/python-apt/+question/707491

There is a binary upload here: https://pypi.org/project/python-apt-binary/#history but I have not tested it.

[furlongm]

Debian bug report (closed): https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=932838

[pandel]

Yes, you're right! I did manage to get it to load using the Debian base image and a little bit of tweaking, but I get a warning with some repositories still and am not sure how to deal with it::

/usr/local/lib/python3.9/site-packages/debian/deb822.py:734: UserWarning: Parsing of Deb822 data with python3-apt's apt_pkg was requested but this cannot be done on non-file input.
  warnings.warn(msg)

[pandel]

So, this is how it looks today:

According to this we have two machines with security relevant updates available. That looks very fine so far. Many thanks!

Question: Is it possible to actually see the associated CVE or get a link to the security message in the Debian mailing list? I thought that I might get the detail information when I click on the package or as a separate column in the table shown but I don't or maybe I simply did not find it. It would be very helpful to "enrich" the single "Security" information with the official data describing the security problem in detail.