SiteGulp
v1

REST API Documentation

Complete programmatic access to SiteGulp. Manage sites, start crawls, retrieve results, and configure webhooks via our REST API.

Get API KeyCheck API Status

Quick Start

1Get Your API Key

Generate an API key from your Settings page. Keep it secure - it provides full access to your account.

2Make Your First Request

Include your API key in the Authorization header with every request.

cURL Example
curl -X GET "https://your-domain.com/api/v1/sites" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json"

3Create a Site and Start Crawling

Create a Site
curl -X POST "https://your-domain.com/api/v1/sites" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "My Website",
    "url": "https://example.com",
    "maxPages": 100
  }'
Start a Crawl
curl -X POST "https://your-domain.com/api/v1/sites/SITE_ID/crawls" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Initial Crawl",
    "prompt": "Explore all main pages and capture screenshots",
    "includeImages": true,
    "includeVideos": false,
    "includeDescriptions": true,
    "includeDesignImprovements": false,
    "includeBrowserConsoleLogs": true,
    "viewportDesktop": true,
    "viewportTablet": false,
    "viewportMobile": false,
    "recordScript": true,
    "recordScriptLanguage": "javascript"
  }'

Authentication

All API requests require authentication via Bearer token in the Authorization header:

Authorization: Bearer ak_your_api_key_here

Rate Limiting

1,000 requests per hour per API key

Secure Keys

Keys are hashed with SHA-256 and never stored in plain text

Keep It Secret

Your API key provides full access to your account. Never expose it in client-side code.

API Key Scopes

Every API key has a scope that controls which endpoints it can access.

full — Full Account Access (default)

Can access all projects, crawls, webhooks, and key management endpoints. Use for trusted server-side integrations.

project — Project Access (restricted)

Locked to a single project. Can only start crawls, run page analysis, and read crawl results for that one project. Cannot list other projects, manage webhooks, or create/delete keys. Use when distributing keys to third parties or CI pipelines.

Project-scoped key allowed endpoints:

GET /api/v1/health
GET /api/v1/sites/{projectId}
GET /api/v1/sites/{projectId}/crawls
POST /api/v1/sites/{projectId}/crawls
POST /api/v1/sites/{projectId}/analyze-page
GET /api/v1/crawls/{crawlId}
GET /api/v1/crawls/{crawlId}/pages
GET /api/v1/crawls/{crawlId}/artifacts
GET /api/v1/crawls/{crawlId}/logs
GET /api/v1/crawls/{crawlId}/results

Key Expiry

Keys can be set to expire automatically after a specified duration.

No expiry (default)

Key remains valid until manually deleted.

Time-limited keys

Set expiresInMinutes when creating a key. After expiry, the key is rejected with 401 Unauthorized. Expired keys show as Expired in the Settings page.

Key Management API

Use a full-scope API key to manage other keys programmatically. Project-scoped keys cannot call these endpoints.

GET
/api/v1/keys

List all API keys for your account (hashes not returned).

cURL
curl -X GET "https://your-domain.com/api/v1/keys" \
  -H "Authorization: Bearer YOUR_FULL_ACCESS_KEY"
Response
{
  "keys": [
    {
      "id": "key_abc123",
      "name": "Production CI",
      "scope": "full",
      "projectId": null,
      "expiresAt": null,
      "lastUsedAt": "2026-04-14T10:00:00.000Z",
      "createdAt": "2026-01-01T00:00:00.000Z"
    },
    {
      "id": "key_xyz789",
      "name": "Deploy Pipeline",
      "scope": "project",
      "projectId": "proj_abc123",
      "expiresAt": "2026-12-31T23:59:59.000Z",
      "lastUsedAt": null,
      "createdAt": "2026-04-01T00:00:00.000Z"
    }
  ]
}

POST
/api/v1/keys

Create a new API key. The plain key is returned only once.

Create full-access key with 1-day expiry
curl -X POST "https://your-domain.com/api/v1/keys" \
  -H "Authorization: Bearer YOUR_FULL_ACCESS_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Temporary CI Key",
    "scope": "full",
    "expiresInMinutes": 1440
  }'
Create project-scoped key
curl -X POST "https://your-domain.com/api/v1/keys" \
  -H "Authorization: Bearer YOUR_FULL_ACCESS_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Partner Access",
    "scope": "project",
    "projectId": "proj_abc123"
  }'
Response (201 Created)
{
  "id": "key_new123",
  "name": "Partner Access",
  "scope": "project",
  "projectId": "proj_abc123",
  "expiresAt": null,
  "key": "sk_live_abc123..."
}

Request body fields:

  • name — required, display name
  • scope — "full" (default) or "project"
  • projectId — required when scope is "project"
  • expiresInMinutes — optional positive integer; omit for no expiry

DELETE
/api/v1/keys/:keyId

Delete an API key permanently.

cURL
curl -X DELETE "https://your-domain.com/api/v1/keys/key_abc123" \
  -H "Authorization: Bearer YOUR_FULL_ACCESS_KEY"
Response (200 OK)
{
  "success": true,
  "deleted": "key_abc123"
}

API Endpoints

Click any endpoint to expand input parameters, example requests, responses, and status codes.

Sites

Crawls

Results

Webhooks

Adhoc Crawl

Webhooks

Webhook Events

Subscribe to events to receive real-time notifications when crawls change state.

crawl.startedFired when a crawl begins execution
crawl.completedFired when a crawl finishes successfully
crawl.failedFired when a crawl encounters an error
crawl.cancelledFired when a crawl is cancelled
*Subscribe to all events

Webhook Payload

Example payload sent when a crawl completes

{
  "event": "crawl.completed",
  "timestamp": "2025-11-30T06:00:00.000Z",
  "data": {
    "crawlId": "abc123",
    "projectId": "xyz789",
    "projectName": "My Website",
    "projectUrl": "https://example.com",
    "crawlName": "Initial Crawl",
    "status": "completed",
    "pagesFound": 50,
    "screenshotsTaken": 50,
    "scriptsGenerated": 25,
    "duration": 120000,
    "completedAt": "2025-11-30T06:02:00.000Z"
  }
}

Webhook Signatures

Verify webhook authenticity using HMAC-SHA256 signatures

Every webhook includes a signature header for verification:

X-SiteGulp-Signature: t=1732946400,v1=abc123...

Verify the signature by computing HMAC-SHA256 of the timestamp and payload using your webhook secret.

Node.js Verification Example
const crypto = require('crypto');

function verifyWebhookSignature(payload, signature, secret) {
  const [timestampPart, signaturePart] = signature.split(',');
  const timestamp = timestampPart.split('=')[1];
  const receivedSignature = signaturePart.split('=')[1];

  const signedPayload = `${timestamp}.${JSON.stringify(payload)}`;
  const expectedSignature = crypto
    .createHmac('sha256', secret)
    .update(signedPayload)
    .digest('hex');

  return crypto.timingSafeEqual(
    Buffer.from(receivedSignature),
    Buffer.from(expectedSignature)
  );
}

Response Format

{
  "success": true,
  "data": {
    "id": "site_abc123",
    "name": "My Website",
    "url": "https://example.com",
    "createdAt": "2025-11-30T06:00:00.000Z"
  },
  "meta": {
    "apiKey": "Production Key"
  }
}

Ready to Get Started?

Generate your API key and start automating your website crawls today.

Get Your API Key