Fly.io for self hosting CouchDB

[vrtmrz]

Hi, everyone.
I heard that somebody get troubled with making account of IBM Couldant. So I looked around for alternatives, but there are not, unfortunately.

Therefore, perhaps we have to serve CouchDB ourselves. But setting up the server is a little complicated and dangerous.

I found this: fly.io.
I still don't understand how it will cost I used it a day, it looks like it is almost free when using as like using IBM Couldant. (Pricing).

Does somebody know this?

I registered and set it up to try it out. I felt this is so awesome.
The only thing I worried about is how it costs. But it might be ok.
EDITED
I wrote a Colab note for making a CouchDB instance with some configuration on fly.io. This includes several patches for fly.io v2. If you do not remember enabling the swap, possibly you should create the instance again.

FYI, the setup procedure is below:

1. Install the CLI (Docs)

$ curl -L https://fly.io/install.sh | sh

or

iwr https://fly.io/install.ps1 -useb | iex

2. Sign up (Credit card is required)

flyctl auth signup

3. Create the working directory

mkdir flyio
cd flyio
mkdir couchdb
cd couchdb

5. Create an app by CouchDB's image

flyctl launch --image couchdb

Wizard will be launched, answer as you like

$ flyctl launch --image couchdb:latest
Creating app in ..../flyio/couchdb
Using image couchdb:latest
? App Name (leave blank to use an auto-generated name):
Automatically selected personal organization: vorotamoroz
? Select region: nrt (Tokyo, Japan)
Created app xxxxxx-yyyyyyy-0000 in organization personal
Wrote config file fly.toml
? Would you like to setup a Postgresql database now? No
? Would you like to deploy now? No

6. Create data volume

fly volumes create --region nrt couchdata --size 2

(This means make 2 GB data volume in the Tokyo region, you have to adjust region you selected)

7. Modify created fly.toml as like below. Please change COUCHDB_USER

# fly.toml app configuration file generated for xxxxxx-yyyyyyy-0000
#
# See https://fly.io/docs/reference/configuration/ for information about how to use this file.
#

app = "xxxxxx-yyyyyyy-0000"
primary_region = "nrt"

[http_service]
  internal_port = 5984
  force_https = true
  auto_stop_machines = true
  auto_start_machines = true
  min_machines_running = 0
  processes = ["app"]

[env]
  COUCHDB_USER = "your_username"

  ERL_FLAGS="-couch_ini /opt/couchdb/etc/default.ini /opt/couchdb/etc/default.d/ /opt/couchdb/etc/local.d /opt/couchdb/etc/local.ini /opt/couchdb/data/persistence.ini"

[mounts]
  source="couchdata"
  destination="/opt/couchdb/data"

[build]
  dockerfile = "./Dockerfile"

8. Prepare the dockerfile like this.

FROM couchdb:latest
RUN sed -i '2itouch /opt/couchdb/data/persistence.ini && chmod +w /opt/couchdb/data/persistence.ini && fallocate -l 512M /swapfile && chmod 0600 /swapfile && mkswap /swapfile && echo 10 > /proc/sys/vm/swappiness && swapon /swapfile && echo 1 > /proc/sys/vm/overcommit_memory' /docker-entrypoint.sh

10. Set your password as you like.

flyctl secrets set COUCHDB_PASSWORD=your_password

11. Deploy

flyctl deploy

12. Open in browser.

flyctl open

If you get CouchDB's information, It's ok.

13. Open /_utils, Set up CouchDB (Just hit Configure a Single Node)
    https://xxxxxx-yyyyyyy-0000.fly.dev/_utils/#/setup
    If you have been asked username and password, use COUCHDB_USER and COUCHDB_PASSWORD.

14. Set up Self-hosted LiveSync

key	value
URI	https://xxxxxx-yyyyyyy-0000.fly.dev
Username	COUCHDB_USER
Password	COUCHDB_PASSWORD
Database name	anything_you_like

15. Hit the Check database configuration button and Every Fix button.

16. Hit the Test database connection button.
    If you get Connected to ..., it works!

17. If you are worried about how it will cost too. be sure to stop the app. (We can run it again with count 1.)

fly scale count 0

If you tried, let me know how it is.

[yfffy]

TLDR: it works perfectly!

I signed up with a free fly.io trial account, which is more limited than the hobby plan but you don't need to add payment method, they were good with the hobby plan and now even more generous with the free trial plan!

I spun up a couchdb instance and attached 1gb volume to it, then replicating my self-hosted couchdb to this fly.io instance as a backup, it works perfectly. Now I have my self-hosted couchdb replicating to cloudant, fly.io and one other self-hosted one, enough for data security I suppose.

[vrtmrz]

Thank you for the trial and information! I'm sorry for being late.

The trial plan is worth mentioning.
I had already added a payment method and could not confirm how it works without it. Is no card required?
(It looks like making a second account is not permitted)

I wasn't confident that I understood the terms of use enough, but it seems okay. I will write the document about using fly.io!

[yzlnew]

New to CouchDB. Would you mind sharing how to do replications between CouchDB instances?

[vrtmrz]

Sorry for being late! Sure, I'll write the article!

[NeedsAdjustment]

Can confirm this works perfectly; and both of the free plans on fly.io (that is, with/without a payment method added) should be sufficient for 100% uptime. For peace of mind I generated a virtual card number, but it's probably not necessary.

[janzheng]

This might help others who stumble across this. For me to get CouchDB to work on Fly.io with PouchDB connections, I had to make sure my ports looks like thisfly.toml:

...


  [[services.ports]]
    force_https = true
    handlers = ["http"]
    port = 80
    
  [[services.ports]]
    handlers = ["tls", "http"]
    port = 443

  [[services.ports]]
    handlers = ["tls", "http"]
    port = 5984

...

• Additionally, I had to turn CORS on with credentials: false and headers: accept, authorization, content-type, origin, referer, x-csrf-token.
• I also went to the interface at https://xxxxx-yyyyy.fly.dev/_utils/#/_all_dbs and created the databases: _replicator, _global_changes, _users so Fly would stop complaining in the maintenance log.

[SilencedGoose]

How did you turn on CORS? As in, where did you add those lines?

[janzheng]

once you've set up fly.toml and deployed you should be able to open CouchDB and log into the Fauxton interface. You can now go to Config > CORS, through a similar address: https://aaa-bbb-1234.fly.dev/_utils/#_config/nonode@nohost/cors

[vrtmrz]

Yes, we can fix it manually in the _config.
But also, now we can fix it in the Check database configuration of the Setting dialogue. We do not have to leave Obsidian.

cors.origins and httpd.enable_cors is the configuration that mentioned by @janzheng. a Fix button will be shown If it is not configurated correctly.

[SilencedGoose]

Thanks!

[sarojbelbase]

I followed the process above using India Madras Region, but they don't allow deployment, my configuration is ready 😞

$ flyctl deploy
==> Verifying app config
--> Verified app config
==> Building image
Searching for image 'couchdb:latest' remotely...
image found: img_r7hjdjeb9adhfjf
==> Creating release
Error We need your payment information to continue! Add a credit card or buy credit: https://fly.io/dashboard/someusername/billing

My fly.toml:

[build]
  image = "couchdb:latest"

[env]
COUCHDB_USER = "someusername"

[mounts]
  source="couchdata"
  destination="/opt/couchdb/data"

[experimental]
  allowed_public_ports = []
  auto_rollback = true

[[services]]
  http_checks = []
  internal_port = 8080
  processes = ["app"]
  protocol = "tcp"
  script_checks = []
  [services.concurrency]
    hard_limit = 25
    soft_limit = 20
    type = "connections"

  [[services.ports]]
    force_https = true
    handlers = ["http"]
    port = 80

  [[services.ports]]
    handlers = ["tls", "http"]
    port = 443

  [[services.tcp_checks]]
    grace_period = "1s"
    interval = "15s"
    restart_limit = 0
    timeout = "2s"

[sarojbelbase]

Provided that I don't have credit or debit card access.

[vrtmrz]

Oh, I had heard that we do not need credit cards, something had been changed again at fly.io.
I had already registered with my card so I could not check the behaviour. But still, they only sent me a bill for $0.

I looked for the hints and I found this and this, They might be worried about crypto mining or eviler some.

I also think $25 prepaid is a little expensive and not matched for their free tier (I would never use all of them even if I bought $25 credit).

[vrtmrz]

  internal_port = 8080

This should be

  internal_port = 5984

[sarojbelbase]

I tried changing port but yes I would also go against cryptomining, like always :)

[vrtmrz]

I found that all steps can be run from the Google Colaboratory. We do not have to instal the client locally.

https://gist.github.com/vrtmrz/6715f01bf038c9f66e9d2768c3fdee36

SECURITY WARNING
If you created the machine in this way before 2023-03-29, delete the user ${CouchUser} from the administration or create a machine again.
I really apologise for this.

[sarojbelbase]

So how do we host and stuff?

[NeedsAdjustment]

This has no impact on hosting; it's just to skip local-installing the fly.io CLI.

[Osirus2019]

but this does not support e2e encryption? the OP method does rit?

[bilke]

I have used the collab and it worked perfectly! Thanks a lot! @Osirus2019 You can use with e2e.

[joey00072]

looks like credit card is necessary (location india)

[vrtmrz]

Thank you for your information!

[Osirus2019]

does this plugin sync other plugins 2 way street too?

[kurakurakantor]

thank you so much for this! in my opinion this is the best option if you want to use livesync for free, even if it still requires a credit/debit card, since every free option do need a credit or debit card anyway.

the registration process is super easy and it's not as fussy about card details and personal information as ibm cloudant, my debit card work perfectly on this! (my attempt at ibm lead to my account getting banned without even doing anything...) and the hobby plan on fly.io seems to be really good, 3 gb maximum, dozens of gigabytes of maximum outbound transfer per month, i don't think even big obsidian vaults will exceed the free plan.

plus the setup is pretty simple, especially with the google collab deployment you made.

before this i tried using google cloud since it accepts way more cards than any other platforms, but it's 100 times more complicated, and the "free" plan are not really free, usually it's a couple of cents per month but it keeps growing bigger and bigger as the vault gets bigger (from my understanding google's free outbound transfer is only a gigabyte per month). and the documentation on google about the free plan is not as clear cut which can be really confusing sometimes... and the billing is all over the place, sometimes the bill goes to a negative amount which is very concerning HAHAHA. but with fly.io the cost stays 0 as long as you don't exceed the outbound transfer (which is again very unlikely), the vm selection is very simple, so is the persistent volume storage.

so once again, thank you so much for sharing it!

[Osirus2019]

the first step gives me this error
:PowerShell tried to load a malicious resource detected as Heur.BZC.ZFV.Boxter.341.0F164678 and was blocked. Your device is safe.

Has anyone has the same issue?

[vrtmrz]

This could be a false positive (Heur means heuristic), but it is not clear at the moment.
It is possible that a newer version is not detected, in which case time will tell.

Alternatively, you should ask fly.io to submit a sample to the vendor that makes the antivirus software that notified you of the message.

[Moaness]

Just wanted to say thank you! Fly.io works perfect with zero issues

(couldn't create an account on IBM although using gmail or corporate email address with real physical credit cards and uk address - ended up having banned account for no reason!)

Hands down, LiveSync is the best plugin for Obsidian so far - and probably one of the most underrated plugins!

Thank you again.

[dxmanoo]

I deployed couchdb on flyio with docker by following this post. But it always requires me to click the fix button in livesync ‘cause it always auto turn off the CORS setting and restore /opt/couchdb/etc/ocal.ini on couchdb .
I wonder there's something wrong I've make.

[john-walks-slow]

Hi! I have the same problem, so I make a customized Dockerfile as a workaround. Change couchdb:latest to johnnren/obsidian-couchdb:latest in your fly.toml and everything should work!

[hakuoku]

Hello, I also faced same problem although I faithfully followed the instruction. Even region was same cause I'm Japanese.
Yet I have no knowledge of db or CORS, I realize that CORS is always disabled when I leave https://xxxxxx-yyyyyyy-0000.fly.dev/_utils/ and LiveSync constantly requires me to click the Fix buttons.
Now I got "Replication Completed" on my second device(perhaps replication has been done gradually through try & error?), but in my first trial to sync blank vault in second device to first one LiveSync constantly threw "TypeError: Load Failed" error.

Is this a problem caused by some circumstance of fly.io? I wish this problem figured out and any workaround found.

[Sahuta]

I had struggled with the same problem to realize that I'd forgotten to hit Configure a Single Node. Now everything is fine.

[vrtmrz]

Maybe the last step of the note of colab is useful for performing Configure a Single Node automatically.

And, Fixing all things might be also able to be written up as the curl request. (It might be a bit more.).
I will try it later.

[Sahuta]

Thanks for your response. Recently, the trouble recurred and I found that

CORS is always disabled when I leave https://xxxxxx-yyyyyyy-0000.fly.dev/_utils/

is also triggered by pushing "Fetch" button of "Fetch rebuilt DB" and "Check" button of "Check database configuration".
Moreover, the value

 "origins": "app://obsidian.md,capacitor://localhost,http://localhost"

for cors which overwritten by this app is recognized as a invalid domain, even if directly input them at https://xxxxxx-yyyyyyy-0000.fly.dev/_utils/)#_config/nonode@nohost/cors because they does't begins with http/https .
Thus I think that this may be due to the overwrite settings of this app. Can you check out?

[aparaschiv]

I've put together a Google Colaboratory version that has persistent configurations in /opt/couchdb/etc/local.ini.
We're using fly's built in functionality to build from a Dockerfile, i.e. from couchdb:latest and add our own local.ini with the values that we're interested (I've taken them from setup_own_server)
Fly remote builders are free, so it doesn't cost us anything.

https://gist.github.com/aparaschiv/d878e6aa886d21a41ca1a9794707f61f

Feel free to adapt it to your own needs.

[Douko]

It took me 3 days to configure it, but I am enormously grateful.
Livesync is awesome.

[mammuth]

FYI I had the issue that my fly.io machine ran out of memory all the time during the initial sync (Out of memory messages in the monitoring tab on fly.io).

In case the same happens to others, I did the following things which then ended up successfully syncing my complete vault

• Temporarily increase the memory of the machine from 256MB to 1GB (fly scale memory 1024)
• Decreased batch sizes in the livesync remote DB settings (batch size, batch limit, on-demand batch size)

[mammuth]

Temporarily increase the memory of the machine from 256MB to 1GB

I still face OOM errors with 256MB of memory after reducing batch sizes (the free tier on fly.io). So I'm now trying a machine that has 512MB permanently (due to the auto shut downs costs should be negligible I think)