TiKV for both object store and meta

[thaines]

So I'm trying to format juicefs to run with TiKV using the following command:

sudo juicefs format --trash-days 7 --storage tikv --bucket '192.168.1.1:2379,192.168.1.2:2379,192.168.1.3:2379?ca=/opt/tidb_certs/lime_ca.crt&cert=/opt/tidb_certs/jeannie_juice.crt&key=/opt/tidb_certs/jeannie_juice.key' 'tikv://192.168.1.1:2379,192.168.1.2:2379,192.168.1.3:2379/rind?ca=/opt/tidb_certs/lime_ca.crt&cert=/opt/tidb_certs/jeannie_juice.crt&key=/opt/tidb_certs/jeannie_juice.key' lime

(I've changed the IPs as don't like leaking info about servers onto internet; sudo so it can read the .key)

And it's not working:

2022/06/24 10:36:45.085163 juicefs[956983] <INFO>: Meta address: tikv://192.168.1.1:2379,192.168.1.2:2379,192.168.1.3:2379/rind?ca=/opt/tidb_certs/lime_ca.crt&cert=/opt/tidb_certs/jeannie_juice.crt&key=/opt/tidb_certs/jeannie_juice.key [interface.go:400]
2022/06/24 10:36:45.129357 juicefs[956983] <FATAL>: object storage: strconv.ParseBool: parsing "": invalid syntax [format.go:435]

Anyone know what I could be doing wrong? (or otherwise have any thoughts on doing this!)

juicefs version 1.0.0-rc2+2022-06-24.fc6b120. TiKV has three pd's and three storage nodes - created for this so fresh, using v5.4.1 and happy according to usual checks. Certificates definitely work as tried using them with pd-ctl.

Extra info: I'm not a go coder, but had a look anyway: error message appears to be generated after the return of the call to createStorage (in format.go), which only has one call to ParseBool: Line 208, where it requests tls-insecure-skip-verify from the URL query values that make up the bucket. So I tried setting that to false, i.e.

sudo juicefs format --trash-days 7 --storage tikv --bucket '192.168.1.1:2379,192.168.1.2:2379,192.168.1.3:2379?ca=/opt/tidb_certs/lime_ca.crt&cert=/opt/tidb_certs/jeannie_juice.crt&key=/opt/tidb_certs/jeannie_juice.key&tls-insecure-skip-verify=false' 'tikv://192.168.1.1:2379,192.168.1.2:2379,192.168.1.3:2379/rind?ca=/opt/tidb_certs/lime_ca.crt&cert=/opt/tidb_certs/jeannie_juice.crt&key=/opt/tidb_certs/jeannie_juice.key' lime

Would consider not having a default value for that to be a bug, but won't create a bug report yet in case I'm misunderstanding something.

That got it past that error, but I've now got object storage: [pd] failed to get cluster id instead. Will now dig into that!

Further info: As far as I can tell the values extracted (i.e. the certificates) from the bucket are ignored. In fact, if I switch the logging level to the highest I can see that the meta data address works, i.e.

2022/06/24 12:13:27.480054 juicefs[968317] <INFO>: Meta address: tikv://192.168.1.1:2379,192.168.1.2:2379,192.168.1.3:2379/rind?ca=/opt/tidb_certs/lime_ca.crt&cert=/opt/tidb_certs/jeannie_juice.crt&key=/opt/tidb_certs/jeannie_juice.key [interface.go:400]
[2022/06/24 12:13:27.480 +00:00] [INFO] [client.go:378] ["[pd] create pd client with endpoints"] [pd-address="[192.168.1.1:2379,192.168.1.2:2379,192.168.1.3:2379]"]
[2022/06/24 12:13:27.498 +00:00] [INFO] [base_client.go:332] ["[pd] update member urls"] [old-urls="[http://192.168.1.1:2379,http://192.168.1.2:2379,http://192.168.1.3:2379]"] [new-urls="[https://192.168.1.1:2379,https://192.168.1.2:2379,https://192.168.1.3:2379]"]
[2022/06/24 12:13:27.498 +00:00] [INFO] [base_client.go:350] ["[pd] switch leader"] [new-leader=https://192.168.1.1:2379] [old-leader=]
[2022/06/24 12:13:27.498 +00:00] [INFO] [base_client.go:105] ["[pd] init cluster id"] [cluster-id=7112750979116025088]

but when it gets to the object storage server you get

[2022/06/24 12:15:06.620 +00:00] [WARN] [base_client.go:251] ["[pd] failed to get cluster id"] [url=http://192.168.1.2:2379] [error="[PD:client:ErrClientGetMember]error:rpc error: code = Unavailable desc = connection closed before server preface received target:192.168.1.2:2379 status:TRANSIENT_FAILURE: error:rpc error: code = Unavailable desc = connection closed before server preface received target:192.168.1.2:2379 status:TRANSIENT_FAILURE"]
2022/06/24 12:15:07.621455 juicefs[968317] <FATAL>: object storage: [pd] failed to get cluster id [format.go:435]

noting that the first line is repeated many times due to retries. Not being able to see the certificates being placed in the tls bit combined with the above error saying "http" when it should say "https" makes me think that something is very wrong here.

I guess my question now is: does JuiceFS support certificates when using TiKV as the object storage server? Manual tells you how to do it, yet as far as I can tell that code appears to be missing.

[thaines]

To finish this conversation with myself, I decided that as I had started looking at code I might as well start compiling code. It is a bug and I've submitted a bug report #2311 Was easy to fix, so included a patch with it that I've confirmed to work!

[davies]

it's fixed by #2317, thanks!