Group Project, Idea Suggestion #1

I made a habit of noting down any project idea that popped up in my mind for the last year, I ended up with a bunch of them written in notion xD

Here's a sneak peek

Idea 1: on-demand smart web scraper using LLMs

usually, when writing a scraping script it will be kind of hardcoded for one specific source and you need another script if you want to handle another website, but thanks to recent LLMs like gpt4 who can understand web content you can describe the data you want to be scraped and we'll handle the rest

It would let you scrape any data from any website through an interface, you can select which parts you want to be scraped, and the output format,...
You can set tracking for those fields so that if any changes are detected on the website you are notified. (useful in e-commerce)
You may export the data in JSON for example but you can also ingest the data through an API endpoint which gets updated automatically if you set the tracking mode
No need to worry about IP blocking, we handle those
Can become a SaaS product, I would pay if something like this existed

Looking forward to your feedback

I'll make sure to post other ideas before the call, stay tuned

3 comments