A system for crawling, processing, and querying documentation with AI-powered embedding generation and semantic search capabilities.
- Documentation Crawling: Automatically crawl documentation sites with customizable depth and rate limiting
- Content Processing: Convert HTML to clean Markdown with metadata extraction
- Vector Embeddings: Generate embeddings using AWS Bedrock for semantic searching
- Job Management: Track and manage document processing jobs with detailed progress reporting
- MCP Integration: Built-in MCP tools for AI agent integration
- SPA support: currently the crawler doesn't support SPAs
- Caching: crawled urls are directly added to the DB
- Docker (Install Guide)
- Docker Compose (Install Guide)
- Node.js 16+
- Git
- AWS Account with Bedrock access
- AWS CLI configured with appropriate credentials
-
Clone the Repository:
git clone https://github.com/visheshd/docmcp.git cd docmcp -
Configure Environment:
- Copy the example environment file:
cp .env.example .env
- Edit the
.envfile:- Set
DATABASE_URLtopostgresql://postgres:postgres@localhost:5433/docmcp - Configure AWS Bedrock:
- Set
AWS_REGIONto your AWS region (e.g.,us-east-1) - Set
AWS_ACCESS_KEY_IDandAWS_SECRET_ACCESS_KEYwith your AWS credentials - Or ensure your AWS CLI is configured with appropriate credentials
- Set
- Adjust other settings like
LOG_LEVELif needed
- Set
- Copy the example environment file:
-
Start the Development Environment:
# Make the script executable chmod +x dev-start.sh # Start the development environment ./dev-start.sh
This script will:
- Start PostgreSQL with pgvector in a Docker container
- Install project dependencies
- Run database migrations
- Import seed data automatically
- The database will be accessible on port 5433
-
Add Documentation: Use the
add-docsscript to crawl and process documentation:# Basic usage npm run add-docs -- --url https://example.com/docs --max-depth 3 # With additional options npm run add-docs -- \ --url https://example.com/docs \ --max-depth 3 \ --tags react,frontend \ --package react \ --version 18.0.0 \ --wait
Available options:
--url: Documentation URL to crawl (required)--max-depth: Maximum crawl depth (default: 3)--tags: Comma-separated tags for categorization--package: Package name this documentation is for--version: Package version (defaults to "latest")--wait: Wait for processing to complete--verbose: Enable detailed logging- See
npm run add-docs -- --helpfor all options
-
Query Documentation: Once documentation is added, you can query it using the MCP tools. See the "Querying Documentation" section below.
-
Stop the Development Environment:
docker-compose -f docker-compose.dev.yml down
This setup provides a lightweight development environment with just the required PostgreSQL database and pre-loaded seed data. For production deployments or if you prefer a fully containerized setup, see the "Production Docker Setup" section below.
To use DocMCP with Cursor IDE, you'll need to configure the MCP transport. Add the following configuration to your Cursor settings:
{
"docmcp-local-stdio": {
"transport": "stdio",
"command": "node",
"args": [
"<DOCMCP_DIR>/dist/stdio-server.js"
],
"clientInfo": {
"name": "cursor-client",
"version": "1.0.0"
}
}
}Replace <DOCMCP_DIR> with the absolute path to your DocMCP installation directory.
For example, if DocMCP is installed in /home/user/projects/docmcp, your configuration would be:
"args": ["/home/user/projects/docmcp/dist/stdio-server.js"]After adding this configuration, restart Cursor for the changes to take effect.
The system consists of several core services:
- CrawlerService: Handles documentation site crawling with robots.txt support
- DocumentProcessorService: Processes documents (HTML→Markdown, chunking, embedding)
- JobService: Manages asynchronous processing jobs with detailed status tracking
- ChunkService: Stores and retrieves document chunks with vector search capabilities
- MCP Tools: Agent-friendly interface for adding and querying documentation
The DocMCP system processes documentation through the following pipeline:
-
Documentation Input
- User provides a URL through the
add_documentationMCP tool - System creates a Job record with "pending" status
- Job is assigned tags for categorization and future filtering
- User provides a URL through the
-
Web Crawling (CrawlerService)
- Crawler respects robots.txt restrictions
- Follows links up to specified maximum depth
- Captures HTML content and metadata
- Creates Document records linked to the parent Job
-
Document Processing (DocumentProcessorService)
- Cleans HTML and converts to structured Markdown
- Extracts metadata (package info, version, document type)
- Establishes parent-child relationships between documents
- Updates Job progress as processing continues
-
Chunking & Embedding (ChunkService)
- Splits documents into semantic chunks for better retrieval
- Generates vector embeddings using AWS Bedrock
- Stores embeddings in PostgreSQL with pgvector extension
- Preserves chunk metadata and document references
-
Job Finalization (JobService)
- Updates Job status to "completed"
- Calculates and stores document statistics
- Makes documents available for querying
-
Querying & Retrieval
- User sends query through
query_documentationMCP tool - System converts query to vector embedding
- Performs similarity search to find relevant chunks
- Returns formatted results with source information
- Supports filtering by tags, status, and metadata
- User sends query through
This pipeline enables efficient storage, processing, and retrieval of documentation with semantic understanding capabilities. All steps are tracked through the job system, allowing detailed progress monitoring and error handling.
docmcp/
├── prisma/ # Database schema and migrations
│ └── schema.prisma # Prisma model definitions and database configuration
├── src/
│ ├── config/ # Application configuration
│ │ └── database.ts # Database connection setup
│ ├── generated/ # Generated code (Prisma client)
│ ├── services/ # Core service modules
│ │ ├── crawler.service.ts # Website crawling functionality
│ │ ├── document.service.ts # Document management
│ │ ├── document-processor.service.ts # Document processing and transformation
│ │ ├── job.service.ts # Async job management
│ │ ├── chunk.service.ts # Document chunking and vector operations
│ │ └── mcp-tools/ # MCP integration tools
│ │ ├── add-documentation.tool.ts # Tool for adding new documentation
│ │ ├── get-job-status.tool.ts # Tool for checking job status
│ │ ├── list-documentation.tool.ts # Tool for listing available documentation
│ │ ├── query-documentation.tool.ts # Tool for querying documentation
│ │ ├── sample.tool.ts # Example tool implementation
│ │ └── index.ts # Tool registry and exports
│ ├── types/ # TypeScript type definitions
│ │ └── mcp.ts # MCP tool interface definitions
│ ├── utils/ # Utility functions
│ │ ├── logger.ts # Logging utilities
│ │ └── prisma-filters.ts # Reusable Prisma filtering patterns
│ └── __tests__/ # Test files
│ └── utils/ # Test utilities
│ └── testDb.ts # Test database setup and teardown
├── .env # Environment variables
└── package.json # Project dependencies and scripts