Run() sqlalchemy.exc.PendingRollbackError

[knightdby]

🐛 Bug

Error encountered when running Run() class instance with Python API.

To reproduce

Expected behavior

Environment

• Aim Version (e.g., 3.0.1)
• Python version
• pip version
• OS (e.g., Linux)
• Any other relevant information

Additional context

sqlalchemy.exc.PendingRollbackError: This Session's transaction has been rolled back due to a previous exception during flush. To begin a new transaction with this Session, first issue Session.rollback(). Original exception was: (sqlite3.IntegrityError) UNIQUE constraint failed: run_tag.run_id, run_tag.tag_id
[SQL: INSERT INTO run_tag (run_id, tag_id) VALUES (?, ?)]
[parameters: (14, 7)]
(Background on this error at: https://sqlalche.me/e/20/gkpj)

[migranram]

I also have this problem. Just to add some information: I usually encounter this problem when there is already an open run logging to that AIM Server repo. Then if I try to start a new run (with another machine), or get access to a run from that Repo (like in the code below) I get this error.

Traceback (most recent call last):
  File "/home/miguel/dotfiles/tools/aim/list_runs.py", line 36, in <module>
    print(r.name)
  File "/home/miguel/miniconda3/envs/dl/lib/python3.9/site-packages/aim/sdk/run.py", line 124, in name
    return self.props.name
  File "/home/miguel/miniconda3/envs/dl/lib/python3.9/site-packages/aim/sdk/run.py", line 475, in props
    self._props = self.repo.request_props(self.hash, self.read_only)
  File "/home/miguel/miniconda3/envs/dl/lib/python3.9/site-packages/aim/sdk/repo.py", line 355, in request_props
    return StructuredRunProxy(self._client, hash_, read_only, created_at)
  File "/home/miguel/miniconda3/envs/dl/lib/python3.9/site-packages/aim/storage/structured/proxy.py", line 30, in __init__
    handler = self._rpc_client.get_resource_handler(self, self.resource_type, args=self.init_args)
  File "/home/miguel/miniconda3/envs/dl/lib/python3.9/site-packages/aim/ext/transport/client.py", line 204, in get_resource_handler
    raise_exception(response_json.get('exception'))
  File "/home/miguel/miniconda3/envs/dl/lib/python3.9/site-packages/aim/ext/transport/message_utils.py", line 70, in raise_exception
    raise exception(*args) if args else exception()
sqlalchemy.exc.PendingRollbackError: This Session's transaction has been rolled back due to a previous exception during flush. To begin a new transaction with this Session, first issue Session.rollback(). Original exception was: (sqlite3.IntegrityError) UNIQUE constraint failed: run_tag.run_id, run_tag.tag_id
[SQL: INSERT INTO run_tag (run_id, tag_id) VALUES (?, ?)]
[parameters: (267, 2)]
(Background on this error at: https://sqlalche.me/e/20/gkpj)

[mihran113]

Hey @migranram, @knightdby! Thanks a lot for opening the issue. Can I ask you to check if there are some error logs for the aim server command? Based on the logs provided it seems that the server encountered some error which it can't recover from, so restarting the server will resolve the issue at the moment, but I'll need some more details to tackle the underlying issue. Could you please also specify what version of sqlalchemy are you using?