Support for s3 object storage

[KiaraGrouwstra]

What whould you like to see?

hetzner recently introduced their S3-compatible object storage, offering immutable storage cheaper than their regular shared volumes.
it would be cool if this provider would facilitate configuring hetzner object storage as well, tho it seems given it's in beta there is currently still a manual step to request access involved as well.

[apricote]

Hey @KiaraGrouwstra,

right now we do not plan to add support for the Object Storage S3 API in this terraform provider. You can use any S3-compatible provider like the Minio provider instead.

If using other providers does not work for you, could you explain the issues you have with them and the benefits you see with adding the APIs to this provider?

[KiaraGrouwstra]

i'll try that one - thank you for your response!

[c33s]

i would have prefered to have all from one hand. it would also feel strange if i could create server, firewalls, ... via hetzner cli but not the object storage.
also s3 user and secret management would be nice to have in this provider.

i would vote for reopening the issue

[BerndDA]

As I understood, creating buckets and access keys must be done using Hetzner API. We already successfully configured existing buckets (object lifecycle ruels) using other existing providers, but it would be nice to be able to create the buckets using this provider (saves a manual step in the UI)

[jooola]

Hello all 👋

All our integrations rely on the Hetzner Cloud public API, which is available with a certain level of stability. Since the features you are requesting are not in the public API, we cannot implement them.

Therefore, for the time being, we do not plan to support:

• Generating credentials using the Public API.
• Creating buckets using the Public API: You can use an S3-compatible tool instead, for example using curl or the minio terraform provider.
• Embedding a S3 client in our integrations: You can use any S3-compatible tool instead.

Note that only a subset of the Amazon S3 features are currently supported.

We will leave this ticket open to increase its visibility. If you have questions, reach out to us using the Support Center.

[c33s]

please correct me if i am wong, as i assume that hcloud cli code is the core for the terrform provider, excuse the crosspost:

let us vote for hetznercloud/cli#918 maybe this awesome hetzner developers ❤️ get a bigger budget if we vote for the issue, which i see as voting for them (the hetzner developers).

cheers

[3deep5me]

@apricote Just to let you know a bunch of resources are not supported by the minio terraform in combination with hetzner object storage. E.g. setting public acl on a bucket or create a lifecycle rule.

[jooola]

@apricote Just to let you know a bunch of resources are not supported by the minio terraform in combination with hetzner object storage. E.g. setting public acl on a bucket or create a lifecycle rule.

Do you have some code example to show your use case? Have you tried the aws terraform provider?

[3deep5me]

@jooola thanks for your response.
I tried the aws provider but i was not able to change the region to something non aws-specific and had some issue with auth. If someone has a working config it would be great!

This (at least) does not work right now with Hetzner:

resource "minio_ilm_policy" "bucket-lifecycle-rules" {
  bucket = minio_s3_bucket.bucket.bucket

  rule {
    id         = "expire-7d"
    expiration = "7d"
  }
}

Creating a public bucket with the terraform example fails - Reddit

resource "minio_s3_bucket" "state_terraform_s3" {
  bucket = "state-terraform-s3"
  acl    = "public"
}

All the IAM stuff from minio doesn't work either.

[3deep5me]

This leads me right now to do something like this 😢

resource "null_resource" "install_minio" {
  provisioner "local-exec" {
    command = <<EOT
      curl -o /usr/local/bin/mc https://dl.min.io/client/mc/release/linux-amd64/mc
      chmod +x /usr/local/bin/mc
    EOT
  }
}
# Import ILM rule using MinIO client
resource "null_resource" "import_lifecycle_rule" {
  provisioner "local-exec" {
    command = <<EOT
      echo '${jsonencode(var.bucket_lifecycle_rule)}' > expiry.json
      mc alias set myminio https://${var.hetzner_s3_fqdn} $MINIO_USER $MINIO_PASSWORD
      mc ilm rule import myminio/${minio_s3_bucket.bucket.bucket} < expiry.json
    EOT
  }
  depends_on = [null_resource.install_minio, minio_s3_bucket.bucket]
}

[Keisir]

Do you have some code example to show your use case?

The problem starts with the IAM stuff in MinIO. It's not possible to create a user in the first place.

eg.

resource "minio_iam_user" "some-user" {
  name = "some-custom-name"
}

It's not necessary for Hetzner to duplicate functionality into the hcloud Terraform provider. However, functionality that is distinct and cannot be achieved with third-party providers should be implemented.

In some comment here, it was mentioned that other tools can be used for different use cases, but no other methods of creating users (in general IAM) were stated.

[3deep5me]

Another limitation is that you can not delete minio_s3_bucket_policy with the terraform minio provider.
Only creation works

minio_s3_bucket_policy.access_control_to_bucket: Destroying... [id=hetzner-pls-782yasd]
╷
│ Error: [FATAL] error deleting bucket (hetzner-pls-782yasd): 200 OK
│
│
╵

Would be great to have at least a list which features are supported.
Does some got the aws provider working?

[jooola]

@3deep5me The following configuration should get yourself started using the aws terraform provider:

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.0"
    }
  }
}

provider "aws" {
  skip_credentials_validation = true
  skip_metadata_api_check     = true
  skip_requesting_account_id  = true
  skip_region_validation      = true

  endpoints {
    s3 = "https://fsn1.your-objectstorage.com"
  }

  region = "fsn1"

  # Please checks the docs on how to store those credentials safely.
  access_key = "<YOUR-ACCESS-KEY>"
  secret_key = "<YOUR-SECRET-KEY>"
}

resource "aws_s3_bucket" "main" {
  bucket = "my-bucket-a9c8ae4e"
}

resource "aws_s3_bucket_acl" "main" {
  bucket = aws_s3_bucket.main.id
  acl    = "private"
}

resource "aws_s3_bucket_versioning" "main" {
  bucket = aws_s3_bucket.main.id

  versioning_configuration {
    status = "Enabled"
  }
}

resource "aws_s3_bucket_lifecycle_configuration" "main" {
  bucket = aws_s3_bucket.main.id

  rule {
    id     = "expire-7d"
    status = "Enabled"

    expiration {
      days = 7
    }
  }
}

resource "aws_s3_bucket_policy" "main" {
  bucket = aws_s3_bucket.main.id

  policy = jsonencode({
    Version = "2012-10-17",
    Statement = [
      {
        Effect    = "Allow",
        Principal = "*",
        Action    = ["s3:GetObject"],
        Resource  = ["arn:aws:s3:::${aws_s3_bucket.main.bucket}/*"]
      }
    ]
  })
}