PLEASE HELP!!! Seeing some twisted-python error in my carbon-cache console.log file

[rkr-23]

Hello community.

I'm seeing below errors in my console.log file. Its concerning that when this error occurs my carbon-cache will stop writing to the disk which causes my memory to be filled up and system invokes OOM. Please guide me how to fix this issue.

==> /var/log/carbon/console.log <==
21/11/2024 14:50:33 :: Unhandled Error
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/twisted/python/threadpool.py", line 250, in inContext
result = inContext.theWork()
File "/usr/lib/python3/dist-packages/twisted/python/threadpool.py", line 266, in
inContext.theWork = lambda: context.call(ctx, func, *args, **kw)
File "/usr/lib/python3/dist-packages/twisted/python/context.py", line 122, in callWithContext
return self.currentContext().callWithContext(ctx, func, *args, **kw)
File "/usr/lib/python3/dist-packages/twisted/python/context.py", line 85, in callWithContext
return func(*args,**kw)
--- ---
File "/usr/lib/python3/dist-packages/carbon/writer.py", line 189, in writeForever
writeCachedDataPoints()
File "/usr/lib/python3/dist-packages/carbon/writer.py", line 98, in writeCachedDataPoints
(metric, datapoints) = cache.drain_metric()
File "/usr/lib/python3/dist-packages/carbon/cache.py", line 187, in drain_metric
metric = self.strategy.choose_item()
File "/usr/lib/python3/dist-packages/carbon/cache.py", line 84, in choose_item
metric_name, _ = max(self.cache.items(), key=lambda x: len(itemgetter(1)(x)))
builtins.RuntimeError: dictionary changed size during iteration

[deniszh]

What's your version? I believe fix was released with version 1.1.5 - #822

[rkr-23]

@deniszh

Hello I can see below 2 packages.

My OS is Ubuntu 20.04.6

pip3 list | grep carbon
carbon 1.1.10

dpkg -l | grep carbon
ii graphite-carbon 1.1.4-2 all backend data caching and persistence daemon for Graphite

[rkr-23]

@deniszh
I removed the apt package and now only have package from pip. But now I'm facing the below issue. Can you please help me. I'm new to these stuffs.

/usr/local/bin/carbon-cache.py
/usr/lib/python3/dist-packages/requests/init.py:89: RequestsDependencyWarning: urllib3 (2.2.3) or chardet (3.0.4) doesn't match a supported version!
warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "
Traceback (most recent call last):
File "/usr/local/bin/carbon-cache.py", line 28, in
from carbon.util import run_twistd_plugin # noqa
ModuleNotFoundError: No module named 'carbon'

[deniszh]

It's not carbon or graphite issue, it's generic python dependency issue. Quick googling says that 'pip3 install --upgrade requests' probably should fix this