Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DBConnection.ConnectionError causing Oban telemetry handler to detach #322

Open
davydog187 opened this issue May 1, 2024 · 0 comments
Open
Labels
bug Something isn't working

Comments

@davydog187
Copy link
Contributor

Describe the bug
Seems like there is a case where the Oban telemetry handler doesn't handle a reraised exception from DBConnection

[error] Handler "Elixir.OpentelemetryOban.PluginHandler.plugin_exception" has failed and has been detached. Class=:error
Reason=:function_clause
Stacktrace=[
  {OpentelemetryOban.PluginHandler, :handle_plugin_exception,
   [
     [:oban, :plugin, :exception],
     %{monotonic_time: -576452278018730500, duration: 7500619507},
     %{
       reason: %DBConnection.ConnectionError{
         message: "connection not available and request was dropped from queue after 608ms. This means requests are coming in and your connection pool cannot serve them fast enough. You can address this by:\n\n  1. Ensuring your database is available and that you can connect to it\n  2. Tracking down slow queries and making sure they are running fast enough\n  3. Increasing the pool_size (although this increases resource consumption)\n  4. Allowing requests to wait longer by increasing :queue_target and :queue_interval\n\nSee DBConnection.start_link/2 for more information\n",
         severity: :error,
         reason: :queue_timeout
       },
       stacktrace: [
         {DBConnection, :transaction, 3,
          [
            file: ~c"lib/db_connection.ex",
            line: 1059,
            error_info: %{module: Exception}
          ]},
         {Oban.Plugins.Pruner, :"-handle_info/2-fun-0-", 2,
          [file: ~c"lib/oban/plugins/pruner.ex", line: 109]},
         {:telemetry, :span, 3,
          [
            file: ~c"/truncateddeps/telemetry/src/telemetry.erl",
            line: 321
          ]},
         {Oban.Plugins.Pruner, :handle_info, 2,
          [file: ~c"lib/oban/plugins/pruner.ex", line: 108]},
         {:gen_server, :try_handle_info, 3,
          [file: ~c"gen_server.erl", line: 1095]},
         {:gen_server, :handle_msg, 6, [file: ~c"gen_server.erl", line: 1183]},
         {:proc_lib, :init_p_do_apply, 3, [file: ~c"proc_lib.erl", line: 241]}
       ],
       conf: %Oban.Config{
         dispatch_cooldown: 5,
         engine: Oban.Pro.Engines.Smart,
         get_dynamic_repo: nil,
         insert_trigger: true,
         log: false,
         name: Oban,
         node: "truncated",
         notifier: {Oban.Notifiers.PG, []},
         peer: {Oban.Peers.Postgres, []},
         plugins: [
           {Oban.Plugins.Cron,
            [
              timezone: "America/New_York",
              crontab: [
                # truncated
              ]
            ]},
           {Oban.Pro.Plugins.DynamicLifeline, []},
           {Oban.Plugins.Reindexer, []},
           {Oban.Plugins.Pruner, [max_age: 300]}
         ],
         prefix: "public",
         queues: [default: [limit: 10], automation: [limit: 100]],
         repo: Truncated.Repo,
         shutdown_grace_period: 15000,
         stage_interval: 1000,
         testing: :disabled
       },
       kind: :error,
       telemetry_span_context: #Reference<0.1581702706.460587010.193061>,
       plugin: Oban.Plugins.Pruner
     },
     []
   ], [file: ~c"lib/opentelemetry_oban/plugin_handler.ex", line: 54]},
  {:telemetry, :"-execute/3-fun-0-", 4,
   [
     file: ~c"/truncated/deps/telemetry/src/telemetry.erl",
     line: 160
   ]},
  {:lists, :foreach_1, 2, [file: ~c"lists.erl", line: 1686]},
  {:telemetry, :span, 3,
   [
     file: ~c"/truncated/deps/telemetry/src/telemetry.erl",
     line: 333
   ]},
  {Oban.Plugins.Pruner, :handle_info, 2,
   [file: ~c"lib/oban/plugins/pruner.ex", line: 108]},
  {:gen_server, :try_handle_info, 3, [file: ~c"gen_server.erl", line: 1095]},
  {:gen_server, :handle_msg, 6, [file: ~c"gen_server.erl", line: 1183]},
  {:proc_lib, :init_p_do_apply, 3, [file: ~c"proc_lib.erl", line: 241]}
]

Expected behavior
Ignore the error rather than detaching

Additional context
I don't have a way to reproduce but I wanted to report and I will try to address later if I can

@davydog187 davydog187 added the bug Something isn't working label May 1, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

1 participant