How to Store Large Mesh Files

[rbridges12]

Problem: we currently store our large mesh files on git LFS, which has a free bandwidth limit of 1GB per month. Each time someone clones the repo it uses about 150MB of bandwidth, so obviously that free limit isn’t nearly enough. This needs to be figured out before we start recruiting.

Possible Solutions

• Buy more LFS bandwidth
  • We can buy more bandwidth in 50GB/month increments, each costing $5 a month (details here)
  • at our current file size, we could get about 1000 clones per month for $15/month
  • we could make this better by using Dan’s lower resolution blender files, which should cut us down to ~100MB per clone, might be able to improve that further, FYI this will require someone to realign the files in blender
• Switch to Google Drive API
  • store all our large files on MRover google drive and use the google drive API to make a script that will download them into the repo
  • none of us have used this, so someone will have to spend some time figuring it out and setting it up properly
  • credentials/access tokens might be a problem
  • https://developers.google.com/drive/api
  • https://developers.googleblog.com/2022/02/announcing-authorization-support-for.html
• Abandon fancy .dae meshes and go back to the old ones (at least for now)
  • get rid of our .dae files and just use the original mesh files
  • these still exist in the repo but they might be outdated
  • we can solve this problem and switch back to the nice ones later when we have more time
  • rover will look less cool for demos and stuff but won't actually reduce functionality at all

[CameronTressler]

If someone is interested in figuring out the Google Drive API answer, I'd be fine letting them try. Otherwise, I think it would probably be worth the money to just pay for bandwidth. Do we really need 1000 clones per month? Maybe I'm missing something but it seems to me like $5 a month might cover us.

[rbridges12]

I think that's fair, I suppose we probably don't need 1000. Say we have like 100 computers that the repo needs to be cloned onto in the fall, and maybe some people mess it up once so they have to clone it again, so that puts us well within the 50GB for $5 territory.

[niwhsa9]

@CameronTressler by any chance would you be willing to try out the GDrive API for us to see if we can make that work across the board for software team's LFS needs?

Drive would be an ideal solution, since the UM accts have unlimited free storage and we can use this across software for all kinds of data files not just these meshes.

I'd offer to look into it myself but I currently have too many auton specific ops things I'm trying to dispatch so I don't have the bandwidth atm.