Jkroeger123

Hey HN,

A few months ago I built a web scraper to analyze reddit posts for side project ideas.

I decided to double down on finding insights from reddit, and build an entire platform for exploring insights from reddit data

I use a combination of LLMs for semantic metadata extraction, topic analysis / clustering, semantic analysis, and custom classifiers to derive insights from.

I hope a few of you may find this useful, would appreciate any feedback

~Justin

yea definitely not cheap, to break it down further:

I first ran each post title through gpt-3 to classify as either likely or unlikely to be insightful to founders. Then, if it is likely to be insightful I pass the entire post content including up to 100 comments into gpt 4.

This comes down to around 13-14 million tokens used for the 15k posts, which for the latest gpt-4 model is ~$150.

Im sure this could be done cheaper with something like Llama 2 but tbh I am fine with this for now since I was aiming for speed with the initial list to gauge interest :)

ah sorry poor formatting. insights found was 1k. api costs were $150 for the 15k posts processed

the current list is ["startups, SaaS, bigdata, analytics, content_marketing, web_design, UXDesign, growthhacking, telecommuting, remotework, BusinessIntelligence, indiebiz, advertising, agency, PRConsulting, sales, AskMarketing, eventplanners, EventProduction, EventProfessionals, CustomerSuccess, userexperience, Marketresearch, VirtualAssistants, OfficeManagers, devops, sysadmin, webdev, LawFirm, legaltech, LegalAdvice, sustainability, Green, enviroaction, business, travelhacks, humanresources, InstructionalDesign, socialmedia, consulting, AskManagement, DataScience, networking, B2BForHire, retail, warehouse, QualityAssurance, AskHR, projectmanagement, Construction, ITManagers, SmallBusiness, CustomerService, Automate, cybersecurity, AskNetsec, finance, FinancialPlanning, Emailmarketing, marketing, TechRecruitment, BusinessProcess, techsupport, ITdept, copywriting, TeamBuilding, RemoteWork, smallbusiness, dataisbeautiful, visualization, ecommerce, EntrepreneurRideAlong, LeadGeneration"] and adding a bunch more in the coming weeks :)

happy to hear! hoping to 10x the value of the product over the next few months so stay tuned :)

thanks for the insight! - yes absolutely this is not a list of validated ideas by any means, more so a feed of inspiration and hints towards market interest (if you find 5-6 reddit posts complaining about a similar thing it has higher likely hood of validation then random idea you think of in the shower). I'm working on a free plan which gives 10 a day to scroll through so currently looking at ways of increasing the value here beyond just the static list

yea great point, I chose those in the screenshot since they caught my eye when scrolling through. But in general yes I think i need to work on the database more to only contain the super niche opportunities

Hey HN,

Im an indiehacker who loves to solve for niche pain points

I realized I spend a lot of time just scrolling through subreddits looking for user complaints, insights, etc

I decided to have ChatGPT do the heavy lifting for me, and consolidate a more curated list of posts with insights for me to read

Stats:

Posts Processed: ~15k Insights Found: ~1K API Costs: ~$150

I hope a few of you may find this useful, would appreciate any feedback

~Justin

Contigo is a tool that provides voice to voice conversations powered by GPT-3 and Azure speech services. It corrects your grammar in real time and will support a number of role play scenarios with goals that a classifier will determine if you have succeeded