firecrawl-reference-architecture
Implement FireCrawl reference architecture with best-practice project layout. Use when designing new FireCrawl integrations, reviewing project structure, or establishing architecture standards for FireCrawl applications. Trigger with phrases like "firecrawl architecture", "firecrawl best practices", "firecrawl project structure", "how to organize firecrawl", "firecrawl layout".
Installation
Details
Usage
After installing, this skill will be available to your AI coding assistant.
Verify installation:
skills listSkill Instructions
name: firecrawl-reference-architecture description: | Implement FireCrawl reference architecture with best-practice project layout. Use when designing new FireCrawl integrations, reviewing project structure, or establishing architecture standards for FireCrawl applications. Trigger with phrases like "firecrawl architecture", "firecrawl best practices", "firecrawl project structure", "how to organize firecrawl", "firecrawl layout". allowed-tools: Read, Grep version: 1.0.0 license: MIT author: Jeremy Longshore jeremy@intentsolutions.io
FireCrawl Reference Architecture
Overview
Production-ready architecture patterns for FireCrawl integrations.
Prerequisites
- Understanding of layered architecture
- FireCrawl SDK knowledge
- TypeScript project setup
- Testing framework configured
Project Structure
my-firecrawl-project/
├── src/
│ ├── firecrawl/
│ │ ├── client.ts # Singleton client wrapper
│ │ ├── config.ts # Environment configuration
│ │ ├── types.ts # TypeScript types
│ │ ├── errors.ts # Custom error classes
│ │ └── handlers/
│ │ ├── webhooks.ts # Webhook handlers
│ │ └── events.ts # Event processing
│ ├── services/
│ │ └── firecrawl/
│ │ ├── index.ts # Service facade
│ │ ├── sync.ts # Data synchronization
│ │ └── cache.ts # Caching layer
│ ├── api/
│ │ └── firecrawl/
│ │ └── webhook.ts # Webhook endpoint
│ └── jobs/
│ └── firecrawl/
│ └── sync.ts # Background sync job
├── tests/
│ ├── unit/
│ │ └── firecrawl/
│ └── integration/
│ └── firecrawl/
├── config/
│ ├── firecrawl.development.json
│ ├── firecrawl.staging.json
│ └── firecrawl.production.json
└── docs/
└── firecrawl/
├── SETUP.md
└── RUNBOOK.md
Layer Architecture
┌─────────────────────────────────────────┐
│ API Layer │
│ (Controllers, Routes, Webhooks) │
├─────────────────────────────────────────┤
│ Service Layer │
│ (Business Logic, Orchestration) │
├─────────────────────────────────────────┤
│ FireCrawl Layer │
│ (Client, Types, Error Handling) │
├─────────────────────────────────────────┤
│ Infrastructure Layer │
│ (Cache, Queue, Monitoring) │
└─────────────────────────────────────────┘
Key Components
Step 1: Client Wrapper
// src/firecrawl/client.ts
export class FireCrawlService {
private client: FireCrawlClient;
private cache: Cache;
private monitor: Monitor;
constructor(config: FireCrawlConfig) {
this.client = new FireCrawlClient(config);
this.cache = new Cache(config.cacheOptions);
this.monitor = new Monitor('firecrawl');
}
async get(id: string): Promise<Resource> {
return this.cache.getOrFetch(id, () =>
this.monitor.track('get', () => this.client.get(id))
);
}
}
Step 2: Error Boundary
// src/firecrawl/errors.ts
export class FireCrawlServiceError extends Error {
constructor(
message: string,
public readonly code: string,
public readonly retryable: boolean,
public readonly originalError?: Error
) {
super(message);
this.name = 'FireCrawlServiceError';
}
}
export function wrapFireCrawlError(error: unknown): FireCrawlServiceError {
// Transform SDK errors to application errors
}
Step 3: Health Check
// src/firecrawl/health.ts
export async function checkFireCrawlHealth(): Promise<HealthStatus> {
try {
const start = Date.now();
await firecrawlClient.ping();
return {
status: 'healthy',
latencyMs: Date.now() - start,
};
} catch (error) {
return { status: 'unhealthy', error: error.message };
}
}
Data Flow Diagram
User Request
│
▼
┌─────────────┐
│ API │
│ Gateway │
└──────┬──────┘
│
▼
┌─────────────┐ ┌─────────────┐
│ Service │───▶│ Cache │
│ Layer │ │ (Redis) │
└──────┬──────┘ └─────────────┘
│
▼
┌─────────────┐
│ FireCrawl │
│ Client │
└──────┬──────┘
│
▼
┌─────────────┐
│ FireCrawl │
│ API │
└─────────────┘
Configuration Management
// config/firecrawl.ts
export interface FireCrawlConfig {
apiKey: string;
environment: 'development' | 'staging' | 'production';
timeout: number;
retries: number;
cache: {
enabled: boolean;
ttlSeconds: number;
};
}
export function loadFireCrawlConfig(): FireCrawlConfig {
const env = process.env.NODE_ENV || 'development';
return require(`./firecrawl.${env}.json`);
}
Instructions
Step 1: Create Directory Structure
Set up the project layout following the reference structure above.
Step 2: Implement Client Wrapper
Create the singleton client with caching and monitoring.
Step 3: Add Error Handling
Implement custom error classes for FireCrawl operations.
Step 4: Configure Health Checks
Add health check endpoint for FireCrawl connectivity.
Output
- Structured project layout
- Client wrapper with caching
- Error boundary implemented
- Health checks configured
Error Handling
| Issue | Cause | Solution |
|---|---|---|
| Circular dependencies | Wrong layering | Separate concerns by layer |
| Config not loading | Wrong paths | Verify config file locations |
| Type errors | Missing types | Add FireCrawl types |
| Test isolation | Shared state | Use dependency injection |
Examples
Quick Setup Script
# Create reference structure
mkdir -p src/firecrawl/{handlers} src/services/firecrawl src/api/firecrawl
touch src/firecrawl/{client,config,types,errors}.ts
touch src/services/firecrawl/{index,sync,cache}.ts
Resources
Flagship Skills
For multi-environment setup, see firecrawl-multi-env-setup.
More by jeremylongshore
View allOauth Callback Handler - Auto-activating skill for API Integration. Triggers on: oauth callback handler, oauth callback handler Part of the API Integration skill category.
Rabbitmq Queue Setup - Auto-activating skill for Backend Development. Triggers on: rabbitmq queue setup, rabbitmq queue setup Part of the Backend Development skill category.
evaluating-machine-learning-models: This skill allows Claude to evaluate machine learning models using a comprehensive suite of metrics. It should be used when the user requests model performance analysis, validation, or testing. Claude can use this skill to assess model accuracy, precision, recall, F1-score, and other relevant metrics. Trigger this skill when the user mentions "evaluate model", "model performance", "testing metrics", "validation results", or requests a comprehensive "model evaluation".
building-neural-networks: This skill allows Claude to construct and configure neural network architectures using the neural-network-builder plugin. It should be used when the user requests the creation of a new neural network, modification of an existing one, or assistance with defining the layers, parameters, and training process. The skill is triggered by requests involving terms like "build a neural network," "define network architecture," "configure layers," or specific mentions of neural network types (e.g., "CNN," "RNN," "transformer").