這很重要
這項功能目前處於 公開預覽版。
GitHub Actions 會觸發你從 GitHub 倉庫執行 CI/CD 流程,並讓你自動化建置、測試和部署的 CI/CD 流程。
本文提供 Databricks 所開發的 GitHub Actions 相關信息,以及常見使用案例的範例。 如需 Databricks 上其他 CI/CD 功能和最佳做法的相關信息,請參閱 Azure Databricks 上的 CI/CD 和 最佳做法,以及 Databricks 上建議的 CI/CD 工作流程。
Databricks GitHub 動作
Databricks 已針對 GitHub 上的 CI/CD 工作流程開發下列 GitHub Actions 。 將 GitHub Actions YAML 檔案新增至存放庫的 .github/workflows 目錄。
備註
本文涵蓋由第三方開發的 GitHub Actions。 若要連絡提供者,請參閱 GitHub Actions 支援。
| GitHub 工作流程 | 說明 |
|---|---|
| databricks/setup-cli | 在 GitHub Actions 工作流程中設定 Databricks CLI 的複合動作。 |
執行一個 CI/CD 工作流程,更新 Git 資料夾
下列範例 GitHub Actions YAML 檔案會在遠端分支更新時更新工作區 Git 資料夾。 關於 CI/CD 的 Git 資料夾方法,請參見 「其他版本控制工具」。
需求
這個範例使用了 GitHub Actions 的工作負載身份聯合來增強安全性,並需要你已經建立聯合政策。 請參閱 啟用 GitHub Actions 的工作負載身分識別同盟。
創造行動
現在,將檔案 .github/workflows/sync_git_folder.yml 放入你的儲存庫,內容如下 YAML:
name: Sync Git Folder
concurrency: prod_environment
on:
push:
branches:
# Set your base branch name here
- git-folder-cicd-example
permissions:
id-token: write
contents: read
jobs:
deploy:
runs-on: ubuntu-latest
name: 'Update git folder'
environment: Prod
env:
DATABRICKS_AUTH_TYPE: github-oidc
DATABRICKS_HOST: ${{ vars.DATABRICKS_HOST }}
DATABRICKS_CLIENT_ID: ${{ secrets.DATABRICKS_CLIENT_ID }}
steps:
- uses: actions/checkout@v3
- uses: databricks/setup-cli@main
- name: Update git folder
# Set your workspace path and branch name here
run: databricks repos update /Workspace/<git-folder-path> --branch git-folder-cicd-example
執行包含管線更新的套件組合以運行 CI/CD 工作流程
以下範例 GitHub Actions YAML 檔案觸發測試部署,驗證、部署並執行套件中指定的工作,並於套件組設定檔中定義 dev 的預生產目標中執行。
需求
此範例需要有:
存放庫根目錄的套件組合組態檔,此套件組合組態檔是透過 GitHub Actions YAML 檔案的設定
working-directory: .明確宣告的。此配套組態檔應該定義名為sample_job的 Azure Databricks 工作流程,以及名為dev的目標。 例如:# This is a Databricks asset bundle definition for pipeline_update. bundle: name: pipeline_update include: - resources/*.yml variables: catalog: description: The catalog to use schema: description: The schema to use resources: jobs: sample_job: name: sample_job parameters: - name: catalog default: ${var.catalog} - name: schema default: ${var.schema} tasks: - task_key: refresh_pipeline pipeline_task: pipeline_id: ${resources.pipelines.sample_pipeline.id} environments: - environment_key: default spec: environment_version: '4' pipelines: sample_pipeline: name: sample_pipeline catalog: ${var.catalog} schema: ${var.schema} serverless: true root_path: '../src/sample_pipeline' libraries: - glob: include: ../src/sample_pipeline/transformations/** environment: dependencies: - --editable ${workspace.file_path} targets: dev: mode: development default: true workspace: host: <dev-workspace-url> variables: catalog: my_catalog schema: ${workspace.current_user.short_name} prod: mode: production workspace: host: <production-workspace-url> root_path: /Workspace/Users/someone@example.com/.bundle/${bundle.name}/${bundle.target} variables: catalog: my_catalog schema: prod permissions: - user_name: someone@example.com level: CAN_MANAGE欲了解更多關於套件組組的資訊,請參閱 Databricks 資產組組設定。
名為
SP_TOKEN的 GitHub 秘密,代表 Azure Databricks 服務主體 的 Azure Databricks 存取令牌,其與部署和執行此套件組合的 Azure Databricks 工作區相關聯。 要建立代幣:- 建立一個 Databricks 服務主體。 請參閱 將服務主體新增至您的帳戶。
- 為服務主體產生一個秘密。 請參閱 步驟 1:建立 OAuth 密碼。 複製秘密和客戶端 ID 的數值。
- 手動使用複製的密碼和客戶端 ID 值來生成 Databricks 存取權杖(帳戶或工作區)。 請參閱產生帳戶層級存取權杖。
- 複製
access_tokenJSON 回應的值。 在你的倉庫裡新增一個名為SP_TOKEN的 GitHub 機密至 Actions,並使用 Databricks 的存取權杖作為機密值。 請參閱 加密的秘密。
創造行動
現在,將檔案 .github/workflows/pipeline_update.yml 放入你的儲存庫,內容如下 YAML:
# This workflow validates, deploys, and runs the specified bundle
# within a pre-production target named "dev".
name: 'Dev deployment'
# Ensure that only a single job or workflow using the same concurrency group
# runs at a time.
concurrency: 1
# Trigger this workflow whenever a pull request is opened against the repo's
# main branch or an existing pull request's head branch is updated.
on:
pull_request:
types:
- opened
- synchronize
branches:
- main
jobs:
# Used by the "pipeline_update" job to deploy the bundle.
# Bundle validation is automatically performed as part of this deployment.
# If validation fails, this workflow fails.
deploy:
name: 'Deploy bundle'
runs-on: ubuntu-latest
steps:
# Check out this repo, so that this workflow can access it.
- uses: actions/checkout@v3
# Download the Databricks CLI.
# See https://github.com/databricks/setup-cli
- uses: databricks/setup-cli@main
# Deploy the bundle to the "dev" target as defined
# in the bundle's settings file.
- run: databricks bundle deploy
working-directory: .
env:
DATABRICKS_TOKEN: ${{ secrets.SP_TOKEN }}
DATABRICKS_BUNDLE_ENV: dev
# Validate, deploy, and then run the bundle.
pipeline_update:
name: 'Run pipeline update'
runs-on: ubuntu-latest
# Run the "deploy" job first.
needs:
- deploy
steps:
# Check out this repo, so that this workflow can access it.
- uses: actions/checkout@v3
# Use the downloaded Databricks CLI.
- uses: databricks/setup-cli@main
# Run the Databricks workflow named "sample_job" as defined in the
# bundle that was just deployed.
- run: databricks bundle run sample_job --refresh-all
working-directory: .
env:
DATABRICKS_TOKEN: ${{ secrets.SP_TOKEN }}
DATABRICKS_BUNDLE_ENV: dev
您也可以觸發生產部署。 下列 GitHub Actions YAML 檔案可以存在於與上述檔案相同的存放庫中。 此檔案會在名為 「prod」 的生產目標內驗證、部署及執行指定的套件組合,如套件組合組態檔中所定義。
# This workflow validates, deploys, and runs the specified bundle
# within a production target named "prod".
name: 'Production deployment'
# Ensure that only a single job or workflow using the same concurrency group
# runs at a time.
concurrency: 1
# Trigger this workflow whenever a pull request is pushed to the repo's
# main branch.
on:
push:
branches:
- main
jobs:
deploy:
name: 'Deploy bundle'
runs-on: ubuntu-latest
steps:
# Check out this repo, so that this workflow can access it.
- uses: actions/checkout@v3
# Download the Databricks CLI.
# See https://github.com/databricks/setup-cli
- uses: databricks/setup-cli@main
# Deploy the bundle to the "prod" target as defined
# in the bundle's settings file.
- run: databricks bundle deploy
working-directory: .
env:
DATABRICKS_TOKEN: ${{ secrets.SP_TOKEN }}
DATABRICKS_BUNDLE_ENV: prod
# Validate, deploy, and then run the bundle.
pipeline_update:
name: 'Run pipeline update'
runs-on: ubuntu-latest
# Run the "deploy" job first.
needs:
- deploy
steps:
# Check out this repo, so that this workflow can access it.
- uses: actions/checkout@v3
# Use the downloaded Databricks CLI.
- uses: databricks/setup-cli@main
# Run the Databricks workflow named "sample_job" as defined in the
# bundle that was just deployed.
- run: databricks bundle run sample_job --refresh-all
working-directory: .
env:
DATABRICKS_TOKEN: ${{ secrets.SP_TOKEN }}
DATABRICKS_BUNDLE_ENV: prod
運行 CI/CD 工作流程以建置 JAR 並部署套件組合
如果你的生態系統是基於 Java 的,你的 GitHub Action 需要先建置並上傳 JAR,才能部署套件。 下列範例 GitHub Actions YAML 檔案會觸發部署,以建置 JAR 並將其上傳至磁碟區,然後將套件組合驗證並部署至名為 “prod” 的生產目標,如套件組合組態檔中所定義。 它會編譯 Java 型 JAR,但 Scala 型專案的編譯步驟很類似。
需求
此範例需要有:
- 在儲存庫的根目錄中,有一個套件組合的組態檔,透過 GitHub Actions YAML 檔案的設定明確地指定
working-directory: . -
DATABRICKS_TOKEN環境變數,代表與部署和執行此套件組合之 Azure Databricks 工作區相關聯的 Azure Databricks 存取令牌。 - 環境變數
DATABRICKS_HOST,代表 Azure Databricks 主機工作區。
創造行動
現在,將檔案 .github/workflows/build_jar.yml 放入你的儲存庫,內容如下 YAML:
name: Build JAR and deploy with bundles
on:
pull_request:
branches:
- main
push:
branches:
- main
jobs:
build-test-upload:
runs-on: ubuntu-latest
steps:
- name: Checkout code
uses: actions/checkout@v4
- name: Set up Java
uses: actions/setup-java@v4
with:
java-version: '17' # Specify the Java version used by your project
distribution: 'temurin' # Use a reliable JDK distribution
- name: Cache Maven dependencies
uses: actions/cache@v4
with:
path: ~/.m2/repository
key: ${{ runner.os }}-maven-${{ hashFiles('**/pom.xml') }}
restore-keys: |
${{ runner.os }}-maven-
- name: Build and test JAR with Maven
run: mvn clean verify # Use verify to ensure tests are run
- name: Databricks CLI Setup
uses: databricks/setup-cli@v0.9.0 # Pin to a specific version
- name: Upload JAR to a volume
env:
DATABRICKS_TOKEN: ${{ secrets.DATABRICKS_TOKEN }}
DATABRICKS_HOST: ${{ secrets.DATABRICKS_HOST }} # Add host for clarity
run: |
databricks fs cp target/my-app-1.0.jar dbfs:/Volumes/artifacts/my-app-${{ github.sha }}.jar --overwrite
validate:
needs: build-test-upload
runs-on: ubuntu-latest
steps:
- name: Checkout code
uses: actions/checkout@v4
- name: Databricks CLI Setup
uses: databricks/setup-cli@v0.9.0
- name: Validate bundle
env:
DATABRICKS_TOKEN: ${{ secrets.DATABRICKS_TOKEN }}
DATABRICKS_HOST: ${{ secrets.DATABRICKS_HOST }}
run: databricks bundle validate
deploy:
needs: validate
if: github.event_name == 'push' && github.ref == 'refs/heads/main' # Only deploy on push to main
runs-on: ubuntu-latest
steps:
- name: Checkout code
uses: actions/checkout@v4
- name: Databricks CLI Setup
uses: databricks/setup-cli@v0.9.0
- name: Deploy bundle
env:
DATABRICKS_TOKEN: ${{ secrets.DATABRICKS_TOKEN }}
DATABRICKS_HOST: ${{ secrets.DATABRICKS_HOST }}
run: databricks bundle deploy --target prod