Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Loki won't start if openstack swift storage is configured and swift user doesn't have permissions to create bucket #13980

Open
piotrrotter opened this issue Aug 27, 2024 · 0 comments

Comments

@piotrrotter
Copy link

Describe the bug
Loki won't start if openstack swift storage is configured and swift user doesn't have permissions to create bucket

Operation forbidden
error initialising module: ruler-storage
github.com/grafana/dskit/modules.(*Manager).initModule
        /src/loki/vendor/github.com/grafana/dskit/modules/modules.go:138
github.com/grafana/dskit/modules.(*Manager).InitModuleServices
        /src/loki/vendor/github.com/grafana/dskit/modules/modules.go:108
github.com/grafana/loki/v3/pkg/loki.(*Loki).Run
        /src/loki/pkg/loki/loki.go:458
main.main
        /src/loki/cmd/loki/main.go:129
runtime.main
        /usr/local/go/src/runtime/proc.go:271
runtime.goexit
        /usr/local/go/src/runtime/asm_amd64.s:1695

I believe this line is causing the problem: https://github.com/grafana/loki/blob/v3.1.1/pkg/storage/chunk/client/openstack/swift_object_client.go#L64

I could prepare a MR to fix this, but I don't want to make the decision on how to proceed.
Here are the solutions I see:

  1. Do not attempt to create the bucket; only check if it exists or is writable.
  2. Add a configuration parameter like CreateBucketIfNotExists, defaulting to true to preserve existing behavior.

To Reproduce
Steps to reproduce the behavior:

  1. Started Loki (SHA or version) loki, version release-3.1.x-89fe788 (branch: release-3.1.x, revision: 89fe788d)
  2. Started Promtail (SHA or version) N/A
  3. Query: N/A

The Swift user doesn't have permission to create a bucket. Helm values:

       loki:
          storage:
            type: 'swift'
            swift:
              auth_version: 3
              internal: false
              auth_url: "{{ ocs_url }}"
              username: "{{ ocs_username }}"
              password: "{{ ocs_password }}"
              user_domain_name:  "{{ ocs_domain_name }}"
              project_id: "{{ ocs_project_id }}"
              region_name: "{{ ocs_region }}"
              container_name: "{{ ocs_bucket_name }}"
              max_retries: 5
              connect_timeout: 10s
              request_timeout: 10s

Expected behavior
Loki should not crash on startup if it doesn't have permission to create a bucket, assuming the bucket already exists.
For security reasons, it's not ideal to grant Loki these permissions.

Environment:

  • Infrastructure: Kubernetes
  • Deployment tool: helm
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants