Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enhancement: Add everything from a WARC under one pin instead of many #830

Open
machawk1 opened this issue May 7, 2024 · 0 comments
Open

Comments

@machawk1
Copy link
Member

machawk1 commented May 7, 2024

From @ProximaNova in #810, they suggested (summarized):

Could you provide an option for "ipwb index" that adds everything added from .warc under one pin instead of many pins. Use: for those who want to have as few CIDs in their pinset as possible. It wouldn't matter to those who don't care about having hundreds/thousands of pins. For now, I did this:

  1. edit .cdxj with vim to just get CIDs, on per line, save to a different file (don't overwrite the CDXJ)
  2. run cat f1.cdxj.cid | xargs -d "\n" sh -c 'for args do ipfs files cp /ipfs/$args /a/f1cid/$args && ipfs pin rm $args; done' _
  3. run ipfs files ls --long /a then pin the CID you see for that
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant