Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

use minIO/S3 compatible storage for match blobs #2812

Open
howardchung opened this issue Nov 13, 2024 · 1 comment
Open

use minIO/S3 compatible storage for match blobs #2812

howardchung opened this issue Nov 13, 2024 · 1 comment

Comments

@howardchung
Copy link
Member

howardchung commented Nov 13, 2024

Instead of using Cassandra as a blob store we can use minio (which is s3 compatible) or another s3 compatible object storage. This should help relieve some of the load on the Cassandra cluster (and make it easier to migrate to Scylla when that makes sense)

However, the downsides are:

  • Some additional latency to fetch matches
  • S3 provider will bill per read operation, while Cassandra will not
@howardchung
Copy link
Member Author

howardchung commented Nov 13, 2024

Steps:

  • Set up minIO local Docker image (local tests can use this instead of remote blob store)
  • Update gcdata blobs to contain account_id (avoid need for identity blob)
  • Dual-write anything writing to match_blobs to write to blobstore
  • Configure allowing reading from match_blobs to read from blobstore instead
  • Archive any matches with parsed data in current match_blobs table (use postgres as record, go up to ID where we started dual-write)
  • Backfill old api data from Steam API Can start from a specific seq num date (e.g. 2020) to save time, we can backfill the oldest matches later
  • Migrate api, gcdata blobs from match_blobs to blobstore (since they have ability_upgrades data that expires from API after 2 months)
  • Drop identity blobs (we can't update identity right now anyway due to Steam API outage)
  • Drop ranks blobs (we never filled this data in anyway)
  • After backfill complete, remove reading from match_blobs
  • Drop cassandra table match_blobs

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant