Web Search & Fetch

The Web MCP server provides web search and content fetching capabilities.

Authentication: None required

Tools

search

Search the web:

Input:
  query: string - Search query
  max_results?: number - Number of results (default: 10)

Output:
  results: array
    - title: string
    - url: string
    - snippet: string

Provider: DuckDuckGo (no API key required)

fetch

Fetch and parse a URL:

Input:
  url: string - URL to fetch
  timeout?: number - Timeout in seconds

Output:
  content: string - Page content as markdown
  title: string - Page title
  url: string - Final URL (after redirects)

Features:

• HTML to Markdown conversion

• Main content extraction

• 5-minute page cache

• User agent rotation

scrape

Extract structured data from a page:

Input:
  url: string - URL to scrape
  selectors?: object - CSS selectors for extraction

Output:
  data: object - Extracted structured data

Usage Examples

Research Task

Search for "latest developments in AI agents 2025" and summarize the findings

The agent will:

1. Use search to find relevant articles

2. Use fetch to get full content from top results

3. Synthesize a summary

Fact Checking

Verify this claim: "GPT-4 was released in March 2023"

The agent will:

1. Search for official announcements

2. Fetch and verify from authoritative sources

3. Provide confirmation with citations

Content Analysis

Analyze the homepage of https://example.com

The agent will:

1. Fetch the page content

2. Extract key information

3. Provide analysis

Caching

• TTL: 5 minutes

• Storage: Redis

• Key: URL hash

• Invalidation: Automatic on TTL expiry

Rate Limiting

• Requests are rate-limited to prevent abuse

• Automatic backoff on 429 responses

• Distributed rate limiting via Redis

User Agent Rotation

Multiple user agents are rotated to:

• Avoid blocking

• Appear as real browser traffic

• Support different site requirements

Content Extraction

The fetch tool:

1. Downloads raw HTML

2. Parses with DOM parser

3. Extracts main content (article body, etc.)

4. Removes navigation, ads, scripts

5. Converts to clean Markdown

Memory Integration

Search results can be stored to the memory system:

Search for AI news and remember the key findings

This stores summaries in the vector database for future retrieval.

Error Handling

• Timeout: Returns partial content or error

• 404: Returns "Page not found" error

• Rate limited: Automatic retry with backoff

• Parse error: Returns raw text fallback