URL web scrapper to provide content of the website (articles mostly) as YT

[martincik]

Hey team,

is there an command (cli) I can use to consume content from URL and pass it to summarize?

Example:

• For YT I do: yt https://yoututube.com/xxx | tweet | pbcopy
• And I like to do something similar with URL containing article urlt https://article.com/xxx | tweet | pbcopy

Thank you.

[atljoseph]

lol I had the same thought last night!

Also would be good for the cli to be able to answer this question for us, and suggest a command to create such a pattern with improved prompt. #676 (comment)

It would be really nice to have something extract the useful parts of html to divorce it from the markup for downstream consumption.

[johnconnor-sec]

Here you go. I think this might help a bit. At least I hope so... Jina AI GO Tool

[mseigal]

I use curl to extract the content from a webpage. I share a webpage from my iPhone to my raspberry pi running fabric that extracts the page’s content and a custom pattern summarises the content to meet my needs. Here is the shell command that is executed on my pi: curl -s webpageurl | sed -n '/<body/,/</body>/p' | sed 's/<[^>]*>//g' | fabric --pattern news_summary