Firecrawl MCP Server

Overview

Firecrawl MCP Server is a Model Context Protocol (MCP) server implementation developed by Mendable AI that provides advanced web scraping capabilities. This tool bridges the gap between AI language models and web content extraction, enabling models like Claude and other LLMs to efficiently scrape, process, and analyze web data. The server supports both cloud-based and self-hosted FireCrawl instances, making it flexible for different deployment requirements.

The project has seen regular updates, with the latest stable release being v1.3.3, which introduced official support for Cursor, Claude Desktop, and other MCP-compatible platforms. It can be quickly deployed using NPX or installed as a package for more permanent implementations.

Features

Core Scraping Capabilities

• Web scraping with JavaScript rendering support
• Mobile/Desktop viewport emulation
• Smart content filtering with tag inclusion/exclusion
• URL discovery and crawling functionality

Performance Optimization

• Efficient batch processing with built-in rate limiting
• Automatic retries with exponential backoff
• Queue system for batch operations with parallel processing
• Comprehensive logging system

Integration & Monitoring

• Credit usage monitoring with configurable thresholds
• Support for cloud and self-hosted FireCrawl instances
• Environment variable configuration
• Customizable retry behavior

Search & Discovery

• Web search with content extraction
• Site mapping with subdomain support
• Crawl depth management

Use Cases

Research & Data Collection

• Automate research data gathering from multiple sources
• Extract structured information from web pages using LLM capabilities
• Create comprehensive datasets for training AI models

Content Analysis

• Analyze content across multiple websites for competitive intelligence
• Extract specific information patterns across various web sources
• Monitor content changes on target websites

Website Development

• Clone website structures for redesign purposes
• Generate sitemaps for website auditing
• Extract content templates from existing sites

AI Assistance Enhancement

• Enable AI assistants to provide real-time web data in conversations
• Allow code assistants to reference documentation from the web
• Support content generation with up-to-date web information

Tools

fire_crawl_scrape

• Purpose: Single URL scraping with customizable options
• Features: Support for JavaScript-rendered content, viewport selection
• Example usage:

// Example of scraping a single URL with custom options
const result = await tools.fire_crawl_scrape({
  url: "https://example.com",
  viewport: "desktop",
  includeTags: ["article", "main"],
  excludeTags: ["nav", "footer"]
});

fire_crawl_batch

• Purpose: Batch processing of multiple URLs
• Features: Asynchronous operation with job tracking, parallel processing
• Example usage:

// Example of batch processing multiple URLs
const jobId = await tools.fire_crawl_batch({
  urls: ["https://example.com/page1", "https://example.com/page2"],
  maxParallel: 2,
  outputFormat: "markdown"
});

fire_crawl_status

• Purpose: Check batch job status and results
• Features: Progress tracking with completion status
• Example usage:

// Example of checking batch job status
const status = await tools.fire_crawl_status({
  jobId: "batch_job_12345"
});

fire_crawl_search

• Purpose: Web search with content extraction
• Features: Multi-language support, customizable result limits
• Example usage:

// Example of performing a web search
const searchResults = await tools.fire_crawl_search({
  query: "artificial intelligence latest developments",
  limit: 5
});

Integration Complexity

3/5 - Moderate complexity requiring basic technical knowledge. The server is straightforward to set up with NPX for quick use, but proper configuration for production environments requires understanding of environment variables and MCP integration principles.

Setup Guide

Claude Desktop Setup

1. Set your API key as an environment variable:

export FIRECRAWL_API_KEY=your_api_key_here

2. Add this configuration to your claude_desktop_config.json:

{
  "mcpServers": {
    "mcp-server-firecrawl": {
      "command": "node",
      "args": ["path/to/dist/src/index.js"],
      "env": {
        "FIRE_CRAWL_API_KEY": "YOUR_API_KEY_HERE",
        "FIRE_CRAWL_RETRY_MAX_ATTEMPTS": "5",
        "FIRE_CRAWL_RETRY_INITIAL_DELAY": "2000",
        "FIRE_CRAWL_RETRY_MAX_DELAY": "30000",
        "FIRE_CRAWL_RETRY_BACKOFF_FACTOR": "3",
        "FIRE_CRAWL_CREDIT_WARNING_THRESHOLD": "2000",
        "FIRE_CRAWL_CREDIT_CRITICAL_THRESHOLD": "500"
      }
    }
  }
}

3. For debugging, view logs with:

tail -n 20 -f ~/Library/Logs/Claude/mcp*.log

Cursor Setup

1. Install the server:

npm install firecrawl-mcp-server

2. Build the project:

npm run build

3. Configure Cursor's MCP settings:
   • Add Firecrawl MCP Server to your Cursor MCP configuration
   • Select "node" as the command type
   • Add the path to the built index.js file
   • Include your API key in the environment variables

Quick Setup (For Both Claude and Cursor) Run directly with NPX (no installation required):

env FIRECRAWL_API_KEY=YOUR_KEY npx -y firecrawl-mcp

Additional Configuration Options

• Retry Configuration:
  • FIRE_CRAWL_RETRY_MAX_ATTEMPTS (default: 3)
  • FIRE_CRAWL_RETRY_INITIAL_DELAY (default: 1000ms)
  • FIRE_CRAWL_RETRY_MAX_DELAY (default: 10000ms)
  • FIRE_CRAWL_RETRY_BACKOFF_FACTOR (default: 2)
• Credit Monitoring:
  • FIRE_CRAWL_CREDIT_WARNING_THRESHOLD (default: 1000)
  • FIRE_CRAWL_CREDIT_CRITICAL_THRESHOLD (default: 100)