Concourse CI
Significant updated on 2018-6-21, see ChangeLog
Concourse is a CI system composed of simple tools and ideas. It can express entire pipelines, integrating with arbitrary resources, or it can be used to execute one-off tasks, either locally or in another CI system.
NOTE: When you deploy this template, you should choose the right location which supports DS series VMs for better performance. If you don't want to use DS series VMs, you need to update
Standard_DS
toStandard_D
in cloud-config.yml before deploying Concourse.
1 Prepare Azure Resources
Here we’ll create the following Azure resources that’s required for deploying BOSH and Concourse:
- A default Storage Account
- Two reserved public IPs
- For dev-box
- For Concourse
- A Virtual Network
- A Virtual Machine as your dev-box
- A Bosh director if you need
This ARM template can help you to deploy all the above resources on Azure. Just click the button below with the following parameters:
Name | Required | Default Value | Description |
---|---|---|---|
vmName | YES | Name of Virtual Machine | |
adminUsername | YES | Username for the Virtual Machines. Never use root as the adminUsername. | |
sshKeyData | YES | SSH RSA public key file as a string. | |
concourseUsername | YES | Login username for Concourse web portal or Fly CLI. | |
concoursePassword | YES | Login password for Concourse web portal or Fly CLI. | |
concourseWorkerDiskSize | YES | 30 | Disk size of Concourse worker instance in GB, value must be between 30GB to 1023GB |
environment | YES | AzureCloud | Different environments in Azure. Only AzureCloud is supported for now. |
tenantID | YES | ID of the tenant | |
clientID | YES | ID of the client | |
clientSecret | YES | secret of the client | |
boshVmSize | YES | Standard_D2_v2 | The vm size for bosh director. |
autoDeployBosh | YES | enabled | The flag allowing to deploy the Bosh director. |
autoDeployConcourse | YES | enabled | The flag allowing to deploy the Bosh concourse. |
NOTE:
- Currently BOSH can be only deployed from a Virtual Machine (dev-box) in the same virtual network on Azure.
1.1 Basic Configuration
1.1.1 Generate your Public/Private Key Pair
The parameter sshKeyData
should be a string which starts with ssh-rsa
.
For Linux/Mac Users
Use ssh-keygen to create a 2048-bit RSA public and private key files, and unless you have a specific location or specific names for the files, accept the default location and name.
ssh-keygen -t rsa -b 2048
Then you can find your public key in
~/.ssh/id_rsa.pub
, and your private key in~/.ssh/id_rsa
. Copy and paste the contents of~/.ssh/id_rsa.pub
assshKeyData
.Reference: How to Use SSH with Linux and Mac on Azure
For Windows Users
- Use
PuttyGen
to generate a public and private key pair. - Use the public key as
sshKeyData
. - Save the private key as a .ppk file.
- When you login the dev-box, use the .ppk file as the private key file for authentication.
Reference: How to Use SSH with Windows on Azure
- Use
1.1.2 Specify your Service Principal
The service principal includes tenantID
, clientID
and clientSecret
. If you have an existing service principal, you can reuse it. If not, you should click HERE to learn how to create one.
1.1.3 Deploy Bosh Director Automatically
By default, autoDeployBosh
is set to enabled
. The Bosh director will be deployed by the template, it will take about 30 minutes.
If you would like to deploy Bosh using a more customized bosh.yml
, you can set autoDeployBosh
to disabled
. After the deployment succeeds, you can login the dev-box, update ~/example_manifests/bosh.yml
, and run ./deploy_bosh.sh
.
1.1.4 Deploy Concourse Automatically
By default, autoDeployConcourse
is set to enabled
. The concourse cluster will be deployed by the template, it will take about 30 minutes.
If you would like to deploy Concourse using a more customized concourse.yml
, you can set autoDeployConcourse
to disabled
. After the deployment succeeds, you can login the dev-box, update ~/example_manifests/concourse.yml
, and run ./deploy_concourse.sh
.
1.2 Advanced Configurations
If you want to customize your bosh-setup
template, you can modify the following variables in azuredeploy.json.
Name | Default Value |
---|---|
storageAccountType | Standard_RAGRS |
virtualNetworkName | vnet |
virtualNetworkAddressSpace | 10.0.0.0/16 |
subnetNameForBosh | Bosh |
subnetAddressRangeForBosh | 10.0.0.0/24 |
subnetGatewayForBosh | 10.0.0.1 |
subnetNameForConcourse | Concourse |
subnetAddressRangeForConcourse | 10.0.16.0/20 |
subnetGatewayForConcourse | 10.0.16.1 |
devboxNetworkSecurityGroup | nsg-devbox |
boshNetworkSecurityGroup | nsg-bosh |
concourseNetworkSecurityGroup | nsg-concourse |
vmSize | Standard_D1 |
devboxPrivateIPAddress | 10.0.0.100 |
ubuntuOSVersion | 14.04.5-LTS |
keepUnreachableVMs | false |
NOTE:
- The default type of Azue storage account is "Standard_RAGRS" (Read access geo-redundant storage). For a list of available Azure storage accounts, their capacities and prices, check HERE. Please note Standard_ZRS account cannot be changed to another account type later, and the other account types cannot be changed to Standard_ZRS. The same goes for Premium_LRS accounts.
vmSize
is the instance type of the dev-box. For a list of available instance types, please check HERE.- If you change
subnetAddressRangeForBosh
and/orsubnetAddressRangeForConcourse
, you should changesubnetGatewayForBosh
and/orsubnetGatewayForConcourse
correspondingly. To get a subnet's gateway address, convert the subnet range's prefix to binary presentation, add1
to it, then convert back. For example, the defaultsubnetAddressRangeForBosh
is10.0.0.0/16
, its prefix is10.0.0.0
, the binary presentation of10.0.0.0
is0000 1010 0000 0000 0000 0000 0000 0000
, add1
to it:0000 1010 0000 0000 0000 0000 0000 0001
, then convert back:10.0.0.1
, which is the gateway address.- Set
keepUnreachableVMs
as true when you want to keep unreachable VMs when the deployment fails.
2 Login your dev-box
After the deployment succeeded, you can find the resource group with the name you specified on Azure Portal. The VM in the resource group with the name you specified is your dev-box.
For Linux/Mac Users
You can find the output
SSHDEVBOX
in the latest deployment.SSHDEVBOX
is a string just likessh <adminUsername>@<vm-name-in-lower-case>.<location>.cloudapp.azure.com
.The private key is
~/.ssh/id_rsa
if you accept the default location and name when you generate the ssh key. If so, please runssh <adminUsername>@<vm-name-in-lower-case>.<location>.cloudapp.azure.com
to login your dev-box.If you specified the location and name of the private key, please run
ssh <adminUsername>@<vm-name-in-lower-case>.<location>.cloudapp.azure.com -i <path-to-your-private-key>
.For Windows Users
- Open Putty.
- Fill in
Public IP address/DNS name label
of the dev-box. - Before selecting Open, click the
Connection > SSH > Auth
tab to choose your private key (.ppk).
After you login, you can check ~/install.log
to determine the status of the deployment. When the deployment succeeds, you will find Finish at the end of the log file and no ERROR message in it.
3 Deploy BOSH
If you have enabled the parameter autoDeployBosh
, you can skip this step.
3.1 Configure
The ARM template pre-creates and configure the deployment manifest file bosh.yml
in ~/example_manifests
. You do not need to update it unless you have other specific configurations.
3.2 Deploy
Run the following commands in your home directory to deploy bosh:
./deploy_bosh.sh
NOTE:
- Never use root to perform these steps.
- More verbose logs are written to
~/run.log
.
4 Deploy Concourse
If you have enabled the parameter autoConcourse
, you can skip this step.
4.1 Configure
The ARM template pre-creates and configure the cloud-config manifest file cloud-config.yml
and the deployment manifest file concourse.yml
in ~/example_manifests
. You do not need to update it unless you have other specific configurations.
4.2 Deploy
Run the following commands in your home directory to deploy concourse:
./deploy_concourse.sh
5 Connect to your Concourse
After Concourse had been successfully deployed, you can find the output CONCOURSEENDPOINT
in the latest deployment in your resource group, you can connect to it by the address using your favorite web browser or Fly CLI.
Tags: Microsoft.Network/networkSecurityGroups, Microsoft.Storage/storageAccounts, Microsoft.Network/publicIPAddresses, Microsoft.Network/virtualNetworks, Microsoft.Network/networkInterfaces, Microsoft.Resources/deployments, Microsoft.Compute/virtualMachines, Microsoft.Compute/virtualMachines/extensions, CustomScript