I Got Tired of Checking 15 AI Sites Every Morning, So I Built a Pipeline That Does It While I Sleep

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 3: Slow and Steady What I Built Global Intelligence Graph is a data ingestion engine that keeps an eye on the AI world so I don't have to. Staying current in AI means opening arXiv, Papers with Code, GitHub, a handful of news sites and several job boards, then copy-pasting the interesting bits somewhere. I dreaded that chore, so I automated it. One command now: Scrapes research: the latest AI papers from arXiv and Papers with Code, plus GitHub star counts for papers that have repos. Finds startups and products: it queries the GitHub API for repositories and extracts startup and product entities from them. Reads the news: it pulls articles from 5 major AI news sources and keeps only those published in the last 24 hours. Watches the job market: it scrapes 5 AI job boards with Playwright and converts relative dates ("3 hours ago") into absolute ISO timestamps, again keeping only the last 24 hours. Cleans up the mess: it resolves entity names against a canonical dictionary using fuzzy matching, so "Open AI" and "OpenAI" become one entity. Hands over tidy data: everything lands in SQLite and is exported as structured CSVs in output/. Anyone who tracks the AI ecosystem (students, researchers, recruiters, founders) gets to be a little lazier: the morning tab-hopping becomes opening a folder of CSVs. Code 23f3003672 / data-ingestion-engine Global Intelligence Graph: Data Ingestion Engine This repository contains the data ingestion pipeline for the AI Engineer demo task. It is designed for massive bulk extraction, 24-hour freshness tracking, and resilient LLM extraction using a multi-tier fallback architecture. Setup Instructions Python Environment: Ensure you have Python 3.11+ installed. Install Dependencies pip install -r requirements.txt playwright install Configure Environment Variables: Open the .env file in the root directory and add your API keys: GEMINI_API_KEY=your_gemini_key GROQ_API_KEY=your_groq_key DEEPSEEK_API_KEY=your_deepseek_key (Note: You can run the pipeline with just one of these keys, but the fallback chain works best when all three are present.) Running the Pipeline To execute the pipeline: python src/main.py Pipeline Flow Phase I (Arxiv, Papers with Code & GitHub): Scrapes the latest AI research papers from Arxiv and Papers with Code. Queries the GitHub API for repositories to extract Startup and Product… View on GitHub How I Built It The pipeline runs in four phases: Phase I, research and GitHub. Scrapers collect papers from arXiv and Papers with Code. The GitHub API supplies repositories (the source of startup and product entities) and star counts. Phase II, news and jobs. News articles are fetched from 5 sources. Job listings come from 5 JavaScript-heavy boards, so I used Playwright to render them. Phase III, LLM extraction. An LLMExtractor turns messy raw payloads into validated Pydantic models. It also enforces the 24-hour freshness rule, so stale items never reach the database. Phase IV, entity resolution. Company names across startups, products and jobs are normalized with fuzzy matching against a canonical dictionary. Results are stored in an intermediate SQLite database (data_ingestion.db) that preserves relationships and raw payloads, then exported to CSV. The multi-tier LLM fallback Free-tier LLM APIs hit 429 rate limits constantly when you're doing bulk extraction. Rather than crash or pay for a bigger plan, the extractor falls back across providers: Gemini → Groq (Llama 3) → DeepSeek If one provider rate-limits, the next takes over, and a 5-second processing delay keeps the load gentle. You can run it with just one key, but it's most resilient with all three. Two of the three tiers are open-weight model families (Llama 3 and DeepSeek), so the extraction layer is not tied to a single closed vendor. pip install -r requirements.txt playwright install python src/main.py Why Does Open Innovation Matter? Bulk extraction is exactly where closed APIs hurt: thousands of small calls, strict quotas, and a bill that grows with every page scraped. No single point of failure. Because Llama 3 and DeepSeek are open-weight, many providers can serve them. My pipeline can swap models and hosts without rewriting the extraction logic, and a rate limit on one provider is a detour, not an outage. Cost stays near zero. Open-weight models on free or cheap tiers let me run a daily pipeline without a subscription. Swappable by design. The fallback chain is just an ordered list. Adding a locally hosted open-weight model (for example through Ollama) as another tier, or the first one, is a small change that would make the pipeline work fully offline with no data leaving my machine. Open tools all the way down. Playwright, SQLite, Pydantic and the GitHub and arXiv ecosystems are all open, which made the whole thing easy to build and inspect.