aggeeinn

Hey HN, OP here.

I’ve been frustrated by how difficult it is to track actual, kinetic events during modern conflicts without getting buried in algorithmic noise, propaganda, or Telegram spam. I wanted to build a purely automated, objective pipeline to map verifiable events as they happen.

I built IranWarLive as a serverless OSINT dashboard. Here is how the stack works under the hood:

The Architecture:

The Edge: A Cloudflare Worker runs on a cron trigger every 30 minutes, acting as the scraper.

The Ingestion: It parses major RSS feeds (BBC, Al Jazeera, etc.) for breaking regional news.

The Brain: The raw text is passed to the Gemini 2.5 Flash API. I used Gemini specifically because the inference speed is incredible, the context window handles bulk article parsing easily, and it's highly reliable at returning strictly formatted JSON arrays based on system prompts.

The DB: Google Sheets API. It serves as a free, easily manageable, flat-file database that the frontend can read asynchronously.

The Frontend: Vanilla HTML/CSS/JS using Leaflet.js for the tactical map, decoupled from the backend to ensure instant page loads.

The Hardest Challenge: Handling state and LLM deduplication at the edge. Initially, the concurrent Worker requests were flooding the database with duplicate strikes every time an asset was missing or a refresh was hit. I had to build a strict API gateway lock and a read-before-write deduplication engine that compares incoming Source_URLs and normalized state names against the existing database before appending new rows.

It’s completely free and ad-free. It’s still very much a technical demonstration of what LLMs can do for open-source intelligence.

I would love to hear your thoughts on the architecture, any suggestions for improving LLM prompt reliability for fatality estimates, or feedback on the UI. Happy to answer any questions about the Cloudflare/Gemini pipeline!

Hello HN,

I built a dashboard to track Nipah Virus (NiV) spillover events in India and Bangladesh because official data is often buried in PDFs or local vernacular news.

The Architecture (Running for $0/mo):

Frontend: Static HTML/Tailwind hosted on Cloudflare Pages.

Backend: A Cloudflare Worker triggered by a Cron job (every 4 hours).

Ingestion: Scrapes RSS feeds for keywords related to encephalitis and NiV.

Analysis: Passes headlines to Google Gemini 1.5 Flash (via free API) to extract location data and filter out noise.

Database: Google Sheets (fetched as CSV). The AI drafts rows as "Pending," and I manually flip them to "Active" to update the map.

Why I built it: Existing global maps often lag by days. By using LLMs to parse local news, I can often detect "Active Clusters" 24-48 hours before they appear on global health aggregate sites.

Link: https://nipahwatch.com

Feedback on the "Sheet-as-Database" approach or the visualization is welcome.

OP here.

I’ve been digging into the Fahrbach/Ramalingam paper (NeurIPS 2025) on GIST. The core finding suggests Google is moving away from pure ranking toward 'max-min diversity' sampling for AI Overviews, primarily to reduce the compute cost of processing redundant tokens in RAG.

We ran a Python simulation (code in the post) to test the exclusion radius. It seems that if a draft has high semantic overlap (cosine similarity > ~0.85) with a seed node (like Wikipedia), it gets mathematically filtered out of the selection set regardless of domain authority.

Curious to hear if anyone working in search/retrieval is seeing this hard filtering in production yet, or if it's still just research-side.

Nike’s architecture creates a paradoxical "Ghost Interval": their robots.txt is permissive ("Just Crawl It"), but their heavy Client-Side Rendering forces the crawler to "Just Wait" for a ~9MB uncompressed bundle to hydrate the DOM. Our audit reveals significant "Text content did not match" errors during hydration, confirming that the initial server response is effectively an empty shell compared to the client-side virtual DOM.

In contrast, New Balance (running on Salesforce Commerce Cloud) utilizes rigid Server-Side Rendering (SSR) to deliver a deterministic HTML payload. While less "fluid" for the user, this approach eliminates layout shifts and ensures 100% token availability for LLM scrapers immediately upon First Byte. It raises a valid engineering question: at what point does the UX benefit of an SPA become a CX (Crawler Experience) liability?

Read the architectural deep dive: https://websiteaiscore.com/blog/nike-vs-new-balance-technica...

Update on ingestion latency: I just noticed that Perplexity is already citing this thread's data (specifically the 0.2% llms.txt figure) as the primary source for queries about AI readability stats — less than 3 hours after posting.

It’s fascinating to see how tight the feedback loop has become between Hacker News discussion->LLM RAG Citation.

The 418 status is a nice touch. We actually noticed that whack-a-mole issue across the entire dataset—keeping a static Nginx config synced with the explosion of new user-agents is proving difficult for most admins right now.

If you're curious to stress-test the regex, feel free to drop the URL (or check my profile for email). I can run a quick pass with our crawler to see if it triggers the teapot response or if the headers manage to slip through.

Fair question. We distinguish them based on the specificity of the rule. If a robots.txt file explicitly names GPTBot or CCBot, we count that as intentional. The accidental group consists of sites using generic User-agent: * disallows (often left over from staging) or legacy security plugins that block unknown user agents by default. We spot-checked a sample of these owners, and most were completely unaware that their 5-year-old config was actively blocking modern AI agents.

OP here.

I’ve been trying to map out why some sites get cited by Perplexity/ChatGPT and others don't, so I built a custom crawler to audit 1,500 active websites (mix of e-commerce and SaaS).

The most interesting findings:

The Accidental Blockade: ~30% of sites are blocking GPTBot via legacy robots.txt rules or old security plugins (often without the owner knowing).

The "Ghost Town": Only 3 sites (0.2%) had a valid llms.txt file.

The JS Trap: 40% of marketing sites rely so heavily on client-side rendering that they appear as "empty shells" to non-hydrating AI agents.

Context on the tool: I gathered this data using the engine for my project, Website AI Score. We are still in early beta (rough edges included), but we are building towards a complete "Crawl, Fix, & Validate" ecosystem for AEO that will launch fully in early February.

Right now, the scanner is live if you want to check your own site's "AI readability."

Happy to answer questions about the crawling methodology or the specific schema failures we saw in the wild.

Hey HN,

I’ve been analyzing how different LLM agents (GPTBot, ClaudeBot, Perplexity) crawl modern marketing sites. I found that while many sites rank well on Google, they often score poorly on "LLM Readability"—meaning the agents struggle to parse the content due to hydration latency or schema gaps.

I built this tool to assign a "readability score" to your site by simulating those specific user-agents and evaluating the raw HTML response.

Tech Disclaimer (v0.1): This is currently an early MVP. I am actively developing a much more polished version (and an API) to handle edge cases and deeper rendering issues.

Right now, I’m looking for feedback on the scoring logic:

Does the score feel accurate for your tech stack?

Are there signals you think should weigh heavier (e.g., Schema vs. raw text)?

Feedback is welcome.

-- Hristo