tmp_dir is misleading as /tmp is still used by anchore engine container for a read-only container

[ericbrumfield]

Is this a request for help?:
Not a request for help, it's more a bug or at best reporting misleading documentation.

Is this a BUG REPORT or a FEATURE REQUEST? (choose one):
BUG REPORT

Version of Anchore Engine and Anchore CLI if applicable:
anchore-cli, version 0.9.3
Engine DB Version: 0.0.15
Engine Code Version: 1.0.0

What happened:
I ran anchore as a container with a read-only filesystem as well as a config.yaml that was setup with a tmp_dir specified to a volume I was mounting to the container instance. My tmp_dir path is setup to be /analysis_scratch. When I attempted to analyze an image after adding it via anchore-cli I got an analysis_failed message. Upon investigating and tailing the anchore-worker.log file, I saw error messages where /tmp was still being used by anchore. The tmp_dir directory is being used, but there is still code in anchore that is using /tmp during the unpacking during analysis it looks like, which in a read-only file system mode on the container, it fails to analyze images. The work-around I've done for now is to mount an additional volume to /tmp on the container, which is misleading and kind of makes the tmp_dir configuration pointless.

What did you expect to happen:
I expected the scratch directory to be fully used by anchore engine with what I had specified in tmp_dir, shifting anything that was happening under /tmp by anchore to the path specified in tmp_dir configuration like the documentation suggests tmp_dir is for.

Any relevant log output from /var/log/anchore:

2022-02-23 14:07:27+0000 [-] [Thread-747] [anchore_engine.utils/run_check()] [ERROR] {"level":"debug","msg":"application config:\noutput: json\nquiet: false\ncheck-for-app-update: false\nanchore:\n  host: \"\"\n  path: \"\"\n  dockerfile: \"\"\n  overwrite-existing-image: false\n  import-timeout: 30\ndev:\n  profile-cpu: false\n  profile-mem: false\nlog:\n  structured: true\n  level: \"\"\n  file: \"\"\npackage:\n  cataloger:\n    enabled: true\n    scope: Squashed\nfile-metadata:\n  cataloger:\n    enabled: true\n    scope: Squashed\n  digests:\n  - sha256\nfile-classification:\n  cataloger:\n    enabled: true\n    scope: Squashed\nfile-contents:\n  cataloger:\n    enabled: true\n    scope: Squashed\n  skip-files-above-size: 1048576\n  globs: []\nsecrets:\n  cataloger:\n    enabled: true\n    scope: AllLayers\n  additional-patterns: {}\n  exclude-pattern-names: []\n  reveal-values: false\n  skip-files-above-size: 1048576\nregistry:\n  insecure-skip-tls-verify: false\n  insecure-use-http: false\n  auth: []\n","time":"2022-02-23 14:07:27"}
2022-02-23 14:07:27+0000 [-] [Thread-747] [anchore_engine.utils/run_check()] [ERROR] {"from-lib":"stereoscope","level":"debug","msg":"image: source=OciDirectory location=/analysis_scratch/ddbca746-c256-407d-9d7d-b5ec9419b41e/raw","time":"2022-02-23 14:07:27"}
2022-02-23 14:07:27+0000 [-] [Thread-747] [anchore_engine.utils/run_check()] [ERROR] 1 error occurred:
2022-02-23 14:07:27+0000 [-] [Thread-747] [anchore_engine.utils/run_check()] [ERROR]    * failed to determine image source: could not fetch image '/analysis_scratch/ddbca746-c256-407d-9d7d-b5ec9419b41e/raw': could not create temp dir: mkdir /tmp/stereoscope-cache428267815: read-only file system
2022-02-23 14:07:27+0000 [-] [Thread-747] [anchore_engine.utils/run_check()] [ERROR]
2022-02-23 14:07:27+0000 [-] [Thread-747] [anchore_engine.utils/run_check()] [ERROR]
2022-02-23 14:07:27+0000 [-] [Thread-747] [anchore_engine.clients.localanchore_standalone/analyze_image()] [WARN] failed to analyze image redacted@256:133b7eaae2906c3af39aa60d92b719fc986f2919841f466db905e54435dcba7a, may retry analyzing the image with another tag
2022-02-23 14:07:27+0000 [-] Traceback (most recent call last):
2022-02-23 14:07:27+0000 [-]   File "/usr/local/lib/python3.8/site-packages/anchore_engine/services/analyzer/analysis.py", line 311, in process_analyzer_job
2022-02-23 14:07:27+0000 [-]     image_data = perform_analyze(
2022-02-23 14:07:27+0000 [-]   File "/usr/local/lib/python3.8/site-packages/anchore_engine/services/analyzer/analysis.py", line 167, in perform_analyze
2022-02-23 14:07:27+0000 [-]     analyzed_image_report, manifest_raw = localanchore_standalone.analyze_image(
2022-02-23 14:07:27+0000 [-]   File "/usr/local/lib/python3.8/site-packages/anchore_engine/clients/localanchore_standalone.py", line 1163, in analyze_image
2022-02-23 14:07:27+0000 [-]     raise AnalysisError(
2022-02-23 14:07:27+0000 [-] anchore_engine.clients.localanchore_standalone.AnalysisError: failed to download, unpack, analyze, and generate image export (redacted@sha256:133b7eaae2906c3af39aa60d92b719fc986f2919841f466db905e54435dcba7a) - exception: Non-zero exit status code when running subprocess: cmd=syft -vv -o json oci-dir:/analysis_scratch/ddbca746-c256-407d-9d7d-b5ec9419b41e/raw, rc=1
2022-02-23 14:07:27+0000 [-] [Thread-747] [anchore_engine.services.analyzer.analysis/process_analyzer_job()] [ERROR] problem analyzing image - exception: failed to download, unpack, analyze, and generate image export (redacted@sha256:133b7eaae2906c3af39aa60d92b719fc986f2919841f466db905e54435dcba7a) - exception: Non-zero exit status code when running subprocess: cmd=syft -vv -o json oci-dir:/analysis_scratch/ddbca746-c256-407d-9d7d-b5ec9419b41e/raw, rc=1

What docker images are you using:
registry1.dso.mil/ironbank/anchore/engine/engine:1.0.0

How to reproduce the issue:
An anchore engine setup with --read-only specified when launching the container instance. You will also need to have tmp_dir specified to a mounted volume using a path other than /tmp to reproduce.

Anything else we need to know:
No, think this covers it, but let me know if you need more info.

[ajvpot]

It looks like this issue comes from this line in stereoscope which uses os.MkdirTemp. os.MkdirTemp uses the value of the TMPDIR env var, so you may be able to set TMPDIR to your writable directory as a workaround.