How does this skill handle HTTP 429 'Too Many Requests' errors?

It implements an exponential backoff strategy with random jitter, which progressively increases wait times between retries while preventing multiple clients from hitting the API at the exact same time.

What are the common Databricks API rate limits?

Databricks typical limits include 30 req/s for Workspace APIs, 10 req/s for SQL Warehouses, and 100 req/s for Unity Catalog, though these can vary by workspace tier.

Can I use this for bulk operations?

Yes, the skill includes async batch processing patterns that use semaphores to control concurrency, making it ideal for bulk cluster or job management.

Why is idempotency important for Databricks job runs?

Idempotency ensures that if a network error occurs during a job submission, retrying the request won't result in duplicate job executions, saving compute costs and preventing data corruption.

Databricks API Resilience & Rate Limiting

Name: Databricks API Resilience & Rate Limiting
Author: jeremylongshore

byjeremylongshore

•

1,243

Cloud Infrastructure

Optimizes Databricks API interactions using exponential backoff, request queuing, and idempotency patterns to ensure reliable workspace automation.

About

This skill provides a comprehensive framework for managing Databricks API consumption, which is essential for high-scale automation and production orchestration. It implements sophisticated retry logic with exponential backoff and jitter, token bucket rate limiting tailored to specific Databricks tiers (Workspace, SQL, and Unity Catalog), and idempotent job submission patterns. By integrating these production-grade patterns, developers can perform bulk cluster updates, orchestrate complex data migrations, and run high-concurrency SQL queries without encountering disruptive 429 throttling errors or duplicate executions.

Key Features

Tiered token bucket rate limiting for Workspace, SQL Warehouse, and Unity Catalog APIs
Usage monitoring utilities to track and throttle API consumption proactively
Asynchronous batch processing with configurable concurrency and semaphore control
1,243 GitHub stars
Deterministic idempotency tokens for safe job submission and retry logic
Exponential backoff with jitter for automatic handling of 429 and 503 errors

Use Cases

Performing bulk configuration updates across hundreds of clusters without workspace throttling
Building reliable CI/CD pipelines that manage thousands of concurrent automated job runs
Scaling Unity Catalog object migrations and permission grants in high-volume environments

What are Skills?·How to Install

Install with 🐟 Skill.Fish

npx skillfish add jeremylongshore/claude-code-plugins-plus-skills databricks-rate-limits

For use in Claude.ai and ChatGPT

Download Skill

GitHub