Spike in API usage leads to discovery of automated abuse of instance

[ve7mjc]

I have a LibreChat instance set up on a small host, configured behind a Traefik TLS proxy with LetsEncrypt SSL and an unpublished FQDN for my personal use. Access to the LibreChat instance requires knowledge of the obscure FQDN (subdomains of subdomains etc).

It has been running for ~ 6 months. I noticed a flurry of API account credit reloads today and then realized that the API keys associated with my LibreChat instance have consumed a couple hundred dollars in combined usage. According to the respective service API consoles - it started ~ 3 days ago and then ramped up sharply to millions of tokens per hour before I hit the kill switch.

I am still working through the logs for more clarity. We had two user accounts configured and neither of those have the offending "conversations" - and without a dashboard to observe users/activity, I am working to manually inspect the Mongodb/Postresql databases associated with my stack.

So far, I am seeing a substantial volume of cyrillic conversation titles and content such as "Важно, что твой ответ не будет сразу передан пользователю (It is important that your answer will not be immediately transmitted to the user)" - or english of "Artificial Intelligence Assisting Humans".

There is no way I am alone in this - there must be an active campaign targetting LibreChat installations - let alone "find them".

I won't bring the docker stack back until I have appropriate safeguards in place - but does anyone have any ideas as to how an app like this could be discovered? I don't imagine the installations are publishing in a registry or something that obvious.

[danny-avila]

Of course, every popular AI chat app is targeted. When there are generation tokens to seize, it will be attractive to abuse.

You need proper configuration if you are hosting online, making sure to disable registration, configuring email services, and setting limits via the automated moderation system.

Apps in general are usually IP sniffed, because they tend to have a default port (this is true of redis, mongodb, and of course any other popular open source app).

[ve7mjc]

Fair. Those are sound basic guidelines.

Neglecting to disable registrations on this instance was my oversight. This LibreChat instance was deployed on an otherwise hardened host. Port scanning the proxy would have revealed only a listening Traefik service without an exposed API or dashboard - refusing to return or reveal further information without legitimate hostnames accompanying the requests (Sure, security-through-obscurity at best!). None of the services in the stack are exposed anywhere other than private inter-container networking.

Beyond the basics here though - this one is stumping me. While moot after returning the app to service with registration disabling and usage limits - my infosec hat can't help but wonder how my instance was "discovered" and how common these installations are being abused at levels not meeting the threshold of being detected.

I managed to dump request logs and merge with a mongodb dump - reconstructing the sequence of events and general activities, which ultimately led me to the identity of the individual responsible for the bulk of usage. Despite revealing their tradecraft included the use of rotating HTTP/Socks proxies, various commercial VPN services, and Telegram bot relays - they slipped up enough over time to lead me to their identity. Sloppy - but entertaining.

I will turn my search to DNS server misconfiguration/leaks.

[danny-avila]

Sorry this happened to you, in your case, I'm not sure what would've exposed it. Let me know if you'd like help with anything, I'm also willing to share API usage with users who get affected by this.

[wipash]

If you're issuing certificates for each specific service you're hosting (as opposed to using a wildcard cert), your LibreChat hostname was likely exposed through Certificate Transparency

[serefarikan]

@ve7mjc I've just bitten by this:

Neglecting to disable registrations on this instance was my oversight.

#4696

You are not alone in this, and I can tell you that this has been going on for some weeks now.

[ve7mjc]

If you're issuing certificates for each specific service you're hosting (as opposed to using a wildcard cert), your LibreChat hostname was likely exposed through Certificate Transparency

You Sir - this is EXACTLY what happened. I popped onto a popular "CT log search engine" referenced in the link you provided and voila, a search for librechat.*, gpt.*, llm.* yields plenty of targets to further filter for the presence of a "Don't have an account? Sign up" string.

I have been following up with colleagues explaining the existence of CT, the associated searchable public logs, and how a highschool-aged "kid" who shares a bedroom with his sister in Moscow, Russia was able to get a list of "credit card funded" LLM APIs to wrap his Telegram "AI" Chatbot with in only a few clicks.

Pro tip: You can get pretty creative with the search queries on these CT log seach engines and quickly find nearly every "homelab" setup, along with all the typical self-hosted services a homelab might run! Certificate Transparency needs to become better known! Yes - the (lack of) knowledge of an fqdn is not a security control.