Use DevOps and CI/CD to publish APIs
With the strategic value of APIs in the enterprise, adopting DevOps continuous integration (CI) and deployment (CD) techniques has become an important aspect of API development. This article discusses the decisions you'll need to make to adopt DevOps principles for the management of APIs.
API DevOps consists of three parts:
Each part of the API DevOps pipeline is discussed below.
An API developer writes an API definition by providing a specification, settings (such as logging, diagnostics, and backend settings), and policies to be applied to the API. The API definition provides the information required to provision the API on an Azure API Management service. The specification may be based on a standards-based API specification (such as WSDL, OpenAPI, or GraphQL), or it may be defined using the Azure Resource Manager (ARM) APIs (for example, an ARM template describing the API and operations). The API definition will change over time and should be considered "source code". Ensure that the API definition is stored under source code control and has appropriate review before adoption.
There are several tools to assist producing the API definition:
- The Azure API Management DevOps Resource Toolkit includes two tools that provide an Azure Resource Manager (ARM) template. The extractor creates an ARM template by extracting an API definition from an API Management service. The creator produces the ARM template from a YAML specification. The DevOps Resource Toolkit supports SOAP, REST, and GraphQL APIs.
- The Azure APIOps Toolkit provides a workflow built on top of a git source code control system (such as GitHub or Azure Repos). It uses an extractor similar to the DevOps Resource Toolkit to produce an API definition that is then applied to a target API Management service. APIOps supports REST only at this time.
- The dotnet-apim tool converts a well-formed YAML definition into an ARM template for later deployment. The tool is focused on REST APIs.
- Terraform is an alternative to Azure Resource Manager to configure resources in Azure. You can create a Terraform configuration (together with policies) to implement the API in the same way that an ARM template is created.
You can also use IDE-based tools for editors such as Visual Studio Code to produce the artifacts necessary to define the API. For instance, there are over 30 plugins for editing OpenAPI specification files on the Visual Studio Code Marketplace. You can also use code generators to create the artifacts. The CADL language lets you easily create high-level building blocks and then compile them into a standard API definition format such as OpenAPI.
Once the API definition has been produced, the developer will submit the API definition for review and approval. If using a git-based source code control system (such as GitHub or Azure Repos), the submission can be done via Pull Request. A pull request informs others of changes that have been proposed to the API definition. Once the approval gates have been confirmed, an approver will merge the pull request into the main repository to signify that the API definition can be deployed to production. The pull request process empowers the developer to remediate any issues found during the approval process.
Both GitHub and Azure Repos allow approval pipelines to be configured that run when a pull request is submitted. You can configure the approval pipelines to run tools such as:
- API specification linters such as Spectral to ensure that the definition meets API standards required by the organization.
- Breaking change detection using tools such as openapi-diff.
- Security audit and assessment tools. OWASP maintains a list of tools for security scanning.
- Automated API test frameworks such as Newman, a test runner for Postman collections.
Once the automated tools have been run, the API definition is reviewed by the human eye. Tools won't catch all problems. A human reviewer ensures that the API definition meets the organizational criteria for APIs, including adherence to security, privacy, and consistency guidelines.
The API definition will be published to an API Management service through a release pipeline. The tools used to publish the API definition depend on the tool used to produce the API definition:
- If using the Azure API Management DevOps Resource Toolkit or dotnet-apim, the API definition is represented as an ARM template. Tasks are available for Azure Pipelines and GitHub Actions to deploy an ARM template.
- If using the Azure APIOps Toolkit, the toolkit includes a publisher that writes the API definition to the service.
- If using Terraform, CLI tools will deploy the API definition on your service. There are tasks available for Azure Pipelines and GitHub Actions
Can I use other source code control and CI/CD systems?
Yes. The process described works with any source code control system (although APIOps does require that the source code control system is git based). Similarly, you can use any CI/CD platform as long as it can be triggered by a check-in and run command line tools that communicate with Azure.
There's no industry standard for setting up a DevOps pipeline for publishing APIs, and none of the tools mentioned will work in all situations. However, we see that most situations are covered by using a combination of the following tools and services:
- Azure Repos stores the API definitions in a git repository.
- Azure Pipelines runs the automated API approval and API publication processes.
- Azure APIOps Toolkit provides tools and workflows for publishing APIs.
We've seen the greatest success in customer deployments, and recommend the following practices:
- Set up either GitHub or Azure Repos for your source code control system. This choice will determine your choice of pipeline runner as well. GitHub can use Azure Pipelines or GitHub Actions, whereas Azure Repos must use Azure Pipelines.
- Set up an Azure API Management service for each API developer so that they can develop API definitions along with the API service. Use the consumption or developer SKU when creating the service.
- Use policy fragments to reduce the new policy that developers need to write for each API.
- Use the Azure APIOps Toolkit to extract a working API definition from the developer service.
- Set up an API approval process that runs on each pull request. The API approval process should include breaking change detection, linting, and automated API testing.
- Use the Azure APIOps Toolkit publisher to publish the API to your production API Management service.
Review Automated API deployments with APIOps in the Azure Architecture Center for more details on how to configure and run a CI/CD deployment pipeline with APIOps.
- Azure DevOps Services includes Azure Repos and Azure Pipelines.
- Azure APIOps Toolkit provides a workflow for API Management DevOps.
- Spectral provides a linter for OpenAPI specifications.
- openapi-diff provides a breaking change detector for OpenAPI v3 definitions.
- Newman provides an automated test runner for Postman collections.
Submit and view feedback for