Nov 20, 2026  ·  1 min read

Databricks swaps fixed rate limits for adaptive concurrency in cluster creation

Databricks, a cloud data and AI platform company, changed how it protects cluster creation from overload. It had used fixed request-rate limits, but an API call returns in milliseconds while provisioning virtual machines (VMs) can take minutes. The team now counts in-flight provisioning workflows, shrinks that concurrency limit when latency rises, and holds extra requests in a bounded queue with fair per-customer scheduling. The company says default limits rose 7.5x, rate-limit errors fell by more than 99.99%, and over 100 customer-specific overrides were removed.

The fixed limits failed both ways: they rejected work when capacity was free and accepted too much when cloud providers slowed. The new design has costs. Queued requests wait longer, and retries inside accepted work can still prolong overload; the authors list retry backoff as future work. The post omits absolute limit values and the exact scope of the error figure.

Source  ·  Databricks ↗ How we raised Databricks cluster rate limits by 7.5× Learn how adaptive concurrency limits helped Databricks raise cluster-creation request-rate limits by 7.5× while protecting long-running workflows.
  1. Nov 20Resiliocs founder turns climate risk research into a business
  2. Nov 20Google makes Genie 4 generally available in the paid Gemini API
  3. Nov 19AI drafts of heart ultrasound reports match human ones, a trial says
  4. Nov 19FCC sets December 17 compliance date for SIM swap fraud rules
  5. Nov 19GAO says four major AI chatbots produced viable attack plans

This is our sketchbook.

Two stoneware cups turned towards each other, as if in conversation

Got a good project?
Let’s work on it together.

Let’s talk →

info@ndot.io · New Jersey · Budapest · Biel