web Scrapper In Python
-
Updated
Sep 6, 2023 - Python
#
trafilatura
Here are 8 public repositories matching this topic...
🤖 Automated Q&A Dataset Generation Pipeline powered by LLMs. Multi-stage pipeline that searches, filters, extracts and transforms web content into high-quality question-answer datasets for LLM training. Supports multiple LLM providers (Groq, Mistral, Ollama) and search engines.
nlp machine-learning natural-language-processing web-scraping question-answering dataset-generation content-extraction mistral document-processing qa-dataset groq automated-pipeline llm llama-index trafilatura ollama semantic-chunking crawl4ai ai-training-data
-
Updated
Jun 7, 2025 - Python
🤖 Collection of AI agents for web search, RAG, and multi-agent collaboration. Features phi-agent + Groq integration, Ollama support, DuckDuckGo/Google search, web scraping, and local knowledge base querying with vector embeddings.
duckduckgo web-scraping knowledge-base semantic-search google-search multi-agent-systems ai-agents conversational-ai rag groq vector-database sentence-transformers llm retrieval-augmented-generation lancedb trafilatura ollama crawl4ai phi-agent
-
Updated
Jun 7, 2025 - Python
Summarize articles using NLTK, Gemini and Trafilatura
-
Updated
Apr 9, 2025 - Python
This project is a Python-based web scraping tool that uses the Trafilatura library to extract and save text content from a list of specified websites. The program is designed to process multiple URLs, extract their main content, and save each website's content to a separate .txt file.
-
Updated
Nov 1, 2024 - Jupyter Notebook
A web scraper with an LLM-powered document suggestion system that combines web crawling, data extraction, and advanced AI capabilities to recommend relevant documents.
-
Updated
May 10, 2025 - Python
Improve this page
Add a description, image, and links to the trafilatura topic page so that developers can more easily learn about it.
Add this topic to your repo
To associate your repository with the trafilatura topic, visit your repo's landing page and select "manage topics."