Paginate Unity Catalog REST API operations

Pagination bounds response sizes, making requests more predictable and reducing failures caused by increasingly large responses as your metastore grows. Hence, unpaginated requests to the Unity Catalog REST API operations listed on this page will soon be unsupported. These operations will remain available in paginated form. For more information, see Unpaginated requests to some Unity Catalog REST API operations will soon be rejected.

This page describes how to find and update your clients that call these operations.

Operations that require pagination

The following operations use their paginated form when max_results is specified. Set max_results=0 to let the server choose the page size, or set a positive value to request a page size:

Operation API reference
List metastores Metastores: List
List external locations External locations: List
List connections Connections: List
List catalogs Catalogs: List
Get workspace bindings Workspace bindings: Get bindings
List schemas Schemas: List
List tables Tables: List
List functions Functions: List
Get permissions Grants: Get
Get effective permissions Grants: Get effective
List system schemas System schemas: List
List storage credentials Storage credentials: List

To find clients that call these operations without pagination, see Find unpaginated callers in the audit log.

Other Unity Catalog list operations, such as List volumes and List table summaries, already return pages when you omit max_results. Operations that use a page_size parameter, such as List permissions, also return pages by default. Your client must still follow with page tokens for these operations to retrieve every result.

Use newer operations where available

For new integrations, Databricks recommends using the following newer operations. Existing clients can continue using the non-deprecated operations in paginated form. Get catalog workspace bindings is deprecated and requires migration. When switching operations, update your client to account for differences in URLs, request parameters, or response fields.

Older operation Newer operation What to do
List storage credentials List credentials Use List credentials for new integrations. Existing clients can continue to call List storage credentials with max_results=0 and follow with page tokens.
Get catalog workspace bindings Get workspace bindings Migrate. Get catalog workspace bindings is deprecated. Call Get workspace bindings with max_results=0 and follow with page tokens.
Get permissions List permissions Use List permissions for new integrations. Existing clients can continue to call Get permissions with max_results=0 and follow with page tokens.
Get effective permissions List effective permissions Use List effective permissions for new integrations. Existing clients can continue to call Get effective permissions with max_results=0 and follow with page tokens.

Use a current SDK or CLI version

The Databricks SDKs and Databricks CLI list commands paginate for you. Use the latest supported release of the Python, Java, or Go SDK, and the latest generally available release of the Databricks CLI.

Older SDK and CLI versions might not paginate for every Unity Catalog list operation. If your integration pins a version, verify that its list command or SDK iterator retrieves all pages, and upgrade if it returns only the first page.

Direct REST calls, including calls made with databricks api get, don't paginate automatically. Your client must follow with next_page_token as described in the next section.

Fetch all pages with a direct REST client

To retrieve a complete result set from an affected operation:

  1. On the first request, set max_results=0 and omit page_token. Setting max_results=0 lets the server choose the page size.
  2. Read next_page_token from the response. On the next request, pass its value as page_token and keep all other request parameters the same.
  3. Repeat until a response omits next_page_token or returns it as an empty value. Keep requesting pages even if a page contains no results.

The response field that contains the results varies by operation. Check the API reference for each operation.

The following example lists all tables in a schema. It reads the workspace URL and a personal access token from the DATABRICKS_HOST and DATABRICKS_TOKEN environment variables:

import os
import requests


def list_tables(catalog_name, schema_name):
    url = f"{os.environ['DATABRICKS_HOST'].rstrip('/')}/api/2.1/unity-catalog/tables"
    headers = {"Authorization": f"Bearer {os.environ['DATABRICKS_TOKEN']}"}
    params = {
        "catalog_name": catalog_name,
        "schema_name": schema_name,
        "max_results": 0,  # Let the server choose the page size.
    }

    tables = []
    while True:
        response = requests.get(url, headers=headers, params=params, timeout=30)
        response.raise_for_status()
        body = response.json()

        tables.extend(body.get("tables", []))

        # Stop when the response no longer includes a page token.
        page_token = body.get("next_page_token")
        if not page_token:
            return tables
        params["page_token"] = page_token

For more REST API guidance, including how to handle HTTP 429 responses and omit response fields you don't need, see Best practices.

Find unpaginated callers in the audit log

To find clients that call the affected operations without pagination, query the audit log system table for requests that omit max_results. The following query checks the last 30 days and groups requests by operation and caller. To identify and contact the owner of each client, use the OAuth app name when available, the user agent, and the sample request ID.

WITH unpaginated_calls AS (
  SELECT
    action_name,
    COALESCE(
      NULLIF(request_params['workspace_id'], '0'),
      NULLIF(CAST(workspace_id AS STRING), '0'),
      'account-level or unknown'
    ) AS originating_workspace,
    COALESCE(
      user_identity.email,
      user_identity.subject_name,
      'unknown'
    ) AS caller,
    identity_metadata.acting_resource_display_name AS oauth_app,
    user_agent,
    source_ip_address,
    event_time,
    request_id,
    request_params
  FROM system.access.audit
  WHERE event_date >= date_sub(current_date(), 30)
    AND service_name = 'unityCatalog'
    AND action_name IN (
      'listMetastores',
      'listExternalLocations',
      'listConnections',
      'listCatalogs',
      'GetWorkspaceBindings',
      'listSchemas',
      'listTables',
      'listFunctions',
      'getPermissions',
      'getEffectivePermissions',
      'listSystemSchemas',
      'listStorageCredentials'
    )
    AND request_params['max_results'] IS NULL
)
SELECT
  action_name,
  originating_workspace,
  caller,
  oauth_app,
  user_agent,
  COUNT(*) AS unpaginated_requests,
  MIN(event_time) AS first_seen,
  MAX(event_time) AS last_seen,
  MAX_BY(source_ip_address, event_time) AS sample_source_ip,
  MAX_BY(request_id, event_time) AS sample_request_id
FROM unpaginated_calls
GROUP BY
  action_name,
  originating_workspace,
  caller,
  oauth_app,
  user_agent
ORDER BY unpaginated_requests DESC;

If the query returns no rows, extend the time range to catch jobs that run infrequently. Most audit logs are available only in the region of the workspace, so run the query in each region that contains your workspaces. An empty result doesn't guarantee that all of your clients paginate.

  • Last updated on