Getting Started with Azure Databricks Serverless Private Git

Note

Azure Databricks Serverless Private Git is in Public Preview. There is a charge for compute and networking costs incurred from serverless compute resources connecting to external resources. For more information on billing, see Understand Databricks serverless networking costs.

What is Serverless Private Git?

Azure Databricks Serverless Private Git lets you connect a Databricks workspace to a private Git server using serverless compute and Azure Private Link. A Git server is private if it cannot be accessed from the internet.

The following diagram illustrates the overall system architecture:

Why use Serverless Private Git?

Compared to the Git proxy, Serverless Private Git offers the following advantages:

• Serverless Private Git acquires serverless compute only when it receives a Git request, and it can be inactive when not in use. In contrast, the Git proxy requires the proxy cluster to be active when the user submits a Git request.

• Serverless Private Git uses Azure Private Link to securely connect to the private Git instance.

Requirements

• Workspaces must be enabled for Serverless.
• Private Git Server must be in the same Azure VNet as the Standard Load Balancer.
• Private Git Server must have a signed certificate/valid HTTPS FQDN.
• The VNet is configured for the Standard Load Balancer (SLB) used for the Private Link service.

Set up Serverless Private Git

1. Follow the steps to set up Private Link for your private Git server. This allows you to create an Azure Private Link connection from Serverless to backends in your network behind an SLB.
2. Create a network connectivity configuration (NCC) to configure egress to a standard load balancer. Considerations for this step:
   • Only one NCC can be configured for a workspace for private Git. If the workspace needs to connect to multiple private Git servers, verify that they can be connected using the same NCC.
   • Limitations such as the number of NCCs supported in a region and the number of workspaces that can be attached to an NCC are documented at Requirements.

3. Obtain an account api token by using a service principal with account-level access.

curl
--location 'https://accounts.azuredatabricks.net/oidc/accounts/{accountid}/v1/token' \
--header 'Content-Type: application/x-www-form-urlencoded' \
--data-urlencode 'client_id=SP_CLIENT_ID_HERE' \
--data-urlencode 'grant_type=client_credentials' \
--data-urlencode 'scope=2ff814a6-3304-4ab8-85cb-cd0e6f879c1d/.default' \
--data-urlencode 'client_secret=SP_CLIENT_SECRET_HERE'

Response

{"access_token":"...","scope":"all-apis","token_type":"Bearer","expires_in":3600}

• Alternate way to obtain Account API Token:

  • Access Token for Authentication: To call the Databricks account REST API, you must perform authentication and obtain an access token.
  • Use a Microsoft Entra ID access token.

BEARER_TOKEN = `az account get-access-token --resource
2ff814a6-3304-4ab8-85cb-cd0e6f879c1d --query "accessToken" -o tsv

4. Add a private endpoint rule to define DNS logic via the API.

In the example, specify the following:

• Account ID
• NCC ID
• Account OAuth Token
• Private Link Service Resource ID
• Git Server’s FQDN in the domain_name’s list

curl --location 'https://accounts.azuredatabricks.net/api/2.0/accounts/{accountid}/network-connectivity-configs/{nccid}/private-endpoint-rules' \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer BEARER_TOKEN' \
--data '{
"resource_id": "/subscriptions/3f262328b/resourceGroups/rg/providers/Microsoft.Network/privateLinkServices/example",
"domain_names": [
"git-server.contoso.com"
]
}
'

Response

{"rule_id":"843ba2e5-bbbb-bbbb-bbbb-7f0d55555215","network_connectivity_config_id":"5a9bdc5f-c43d-41cd-9a6d-1b653e20c7d2","resource_id":"/subscriptions/3f262328b/resourceGroups/rg/providers/Microsoft.Network/privateLinkServices/example","endpoint_name":"databricks-5a9bdc5f-c43d-41cd-9a6d-1b653e20c7d2-pe-99cbbac3","connection_state":"PENDING","creation_time":1740000647980,"updated_time":1740000647949,"domain_names":["git-server.contoso.com"]}

5. Wait a few minutes after setting up the NCC private endpoint rules. The private endpoint rule should appear in the NCC without a specified subresource, and the status should be pending.
6. The Private Link service configured in Step 1 should also have a pending private endpoint connection ready for approval. Approve this connection.

7. Return to the NCC inside the account console and check that it was established.
8. Go to the workspace and try a Git operation. You should see a UI indicator for Serverless Private Git. This page may load for a few seconds while the serverless compute for the Git proxy is spinning up.

After you configure it, Serverless Private Git takes precedence over other forms of private Git connectivity you already provisioned, such as classic Git proxy. If you have a classic Git proxy cluster running, terminate it after setting up Serverless Private Git.

Additional Configurations

Customize your git operations using the config.json file.

1. Create a configuration file at /Workspace/.git_settings/config.json, following the specification below.
2. Grant all Git users View permissions to the configuration file and any CA cert files referenced by it.
3. Interact with Git to validate connectivity to the Git remote, such as cloning a Git folder for a remote repository on the server.
4. Changes to the configuration file may take up to 1 minute to be applied.

Top-Level Config File Structure

{
"default": { ... }, // Optional global settings
"remotes": \[ ... \] // Optional list of per-remote settings
}

`default` Section (Optional)

Global defaults are applied to all Git operations unless overridden by a specific remote.

Field	Type	Required	Default Value	Description
sslVerify	boolean	No	true	Whether to verify SSL certificates.
caCertPath	string	No	"" (empty)	Workspace path to a custom CA certificate.
httpProxy	string	No	"" (empty)	HTTP proxy to route Git traffic through.
customHttpPort	integer	No	Unspecified	Custom HTTP port of the Git server.

`remotes` Section (Optional)

A list of objects defining settings for individual remote Git servers. These settings override the `default` block on a per-remote basis.

Field	Type	Required	Default Value	Description
urlPrefix	string	Yes	—	Prefix to match Git remote URLs.
sslVerify	boolean	No	true	Whether to verify SSL certificates.
caCertPath	string	No	"" (empty)	Workspace path to a custom CA certificate path for this remote.
httpProxy	string	No	"" (empty)	HTTP proxy to route Git traffic through.
customHttpPort	integer	No	Unspecified	Custom HTTP port of the Git server.

Example Config With No Remote-Specific Configuration

"default": {
"sslVerify": false
}
}

Full Config Example

{
"default": {
"sslVerify": true,
"caCertPath": "/Workspace/my_ca_cert.pem",
"httpProxy": "https://git-proxy-server.company.com",
"customHttpPort": "8080",
},
"remotes": \[
{
"urlPrefix": "https://my-private-git.company.com/",
"caCertPath": "/Workspace/my_ca_cert_2.pem"
},
{
"urlPrefix": "https://another-git-server.com/project.git",
"sslVerify": false
}
\]
}

Notes

• The default section must be present, even if only partially.
• The remotes list is optional and can be omitted entirely.
• Each remote entry must contain at least the urlPrefix.
• If you don't specify a value for a field, it uses the default value.
• Unknown fields are ignored.

Limitations

• Serverless proxy log is currently unavailable.
• Available on Azure Serverless regions only.