Concourse CI

Browse code

Significant updated on 2018-6-21, see ChangeLog

Concourse is a CI system composed of simple tools and ideas. It can express entire pipelines, integrating with arbitrary resources, or it can be used to execute one-off tasks, either locally or in another CI system.

NOTE: When you deploy this template, you should choose the right location which supports DS series VMs for better performance. If you don't want to use DS series VMs, you need to update Standard_DS to Standard_D in cloud-config.yml before deploying Concourse.

1 Prepare Azure Resources

Here we’ll create the following Azure resources that’s required for deploying BOSH and Concourse:

• A default Storage Account
• Two reserved public IPs
  • For dev-box
  • For Concourse
• A Virtual Network
• A Virtual Machine as your dev-box
• A Bosh director if you need

This ARM template can help you to deploy all the above resources on Azure. Just click the button below with the following parameters:

Name	Required	Default Value	Description
vmName	YES		Name of Virtual Machine
adminUsername	YES		Username for the Virtual Machines. Never use root as the adminUsername.
sshKeyData	YES		SSH RSA public key file as a string.
concourseUsername	YES		Login username for Concourse web portal or Fly CLI.
concoursePassword	YES		Login password for Concourse web portal or Fly CLI.
concourseWorkerDiskSize	YES	30	Disk size of Concourse worker instance in GB, value must be between 30GB to 1023GB
environment	YES	AzureCloud	Different environments in Azure. Only AzureCloud is supported for now.
tenantID	YES		ID of the tenant
clientID	YES		ID of the client
clientSecret	YES		secret of the client
boshVmSize	YES	Standard_D2_v2	The vm size for bosh director.
autoDeployBosh	YES	enabled	The flag allowing to deploy the Bosh director.
autoDeployConcourse	YES	enabled	The flag allowing to deploy the Bosh concourse.

NOTE:

• Currently BOSH can be only deployed from a Virtual Machine (dev-box) in the same virtual network on Azure.

1.1 Basic Configuration

1.1.1 Generate your Public/Private Key Pair

The parameter sshKeyData should be a string which starts with ssh-rsa.

• For Linux/Mac Users

  Use ssh-keygen to create a 2048-bit RSA public and private key files, and unless you have a specific location or specific names for the files, accept the default location and name.

  ssh-keygen -t rsa -b 2048
  

  Then you can find your public key in ~/.ssh/id_rsa.pub, and your private key in ~/.ssh/id_rsa. Copy and paste the contents of ~/.ssh/id_rsa.pub as sshKeyData.

  Reference: How to Use SSH with Linux and Mac on Azure

• For Windows Users

  1. Use PuttyGen to generate a public and private key pair.
  2. Use the public key as sshKeyData.
  3. Save the private key as a .ppk file.
  4. When you login the dev-box, use the .ppk file as the private key file for authentication.

  Reference: How to Use SSH with Windows on Azure

1.1.2 Specify your Service Principal

The service principal includes tenantID, clientID and clientSecret. If you have an existing service principal, you can reuse it. If not, you should click HERE to learn how to create one.

1.1.3 Deploy Bosh Director Automatically

By default, autoDeployBosh is set to enabled. The Bosh director will be deployed by the template, it will take about 30 minutes.

If you would like to deploy Bosh using a more customized bosh.yml, you can set autoDeployBosh to disabled. After the deployment succeeds, you can login the dev-box, update ~/example_manifests/bosh.yml, and run ./deploy_bosh.sh.

1.1.4 Deploy Concourse Automatically

By default, autoDeployConcourse is set to enabled. The concourse cluster will be deployed by the template, it will take about 30 minutes.

If you would like to deploy Concourse using a more customized concourse.yml, you can set autoDeployConcourse to disabled. After the deployment succeeds, you can login the dev-box, update ~/example_manifests/concourse.yml, and run ./deploy_concourse.sh.

1.2 Advanced Configurations

If you want to customize your bosh-setup template, you can modify the following variables in azuredeploy.json.

Name	Default Value
storageAccountType	Standard_RAGRS
virtualNetworkName	vnet
virtualNetworkAddressSpace	10.0.0.0/16
subnetNameForBosh	Bosh
subnetAddressRangeForBosh	10.0.0.0/24
subnetGatewayForBosh	10.0.0.1
subnetNameForConcourse	Concourse
subnetAddressRangeForConcourse	10.0.16.0/20
subnetGatewayForConcourse	10.0.16.1
devboxNetworkSecurityGroup	nsg-devbox
boshNetworkSecurityGroup	nsg-bosh
concourseNetworkSecurityGroup	nsg-concourse
vmSize	Standard_D1
devboxPrivateIPAddress	10.0.0.100
ubuntuOSVersion	14.04.5-LTS
keepUnreachableVMs	false

NOTE:

• The default type of Azue storage account is "Standard_RAGRS" (Read access geo-redundant storage). For a list of available Azure storage accounts, their capacities and prices, check HERE. Please note Standard_ZRS account cannot be changed to another account type later, and the other account types cannot be changed to Standard_ZRS. The same goes for Premium_LRS accounts.
• vmSize is the instance type of the dev-box. For a list of available instance types, please check HERE.
• If you change subnetAddressRangeForBosh and/or subnetAddressRangeForConcourse, you should change subnetGatewayForBosh and/or subnetGatewayForConcourse correspondingly. To get a subnet's gateway address, convert the subnet range's prefix to binary presentation, add 1 to it, then convert back. For example, the default subnetAddressRangeForBosh is 10.0.0.0/16, its prefix is 10.0.0.0, the binary presentation of 10.0.0.0 is 0000 1010 0000 0000 0000 0000 0000 0000, add 1 to it: 0000 1010 0000 0000 0000 0000 0000 0001, then convert back: 10.0.0.1, which is the gateway address.
• Set keepUnreachableVMs as true when you want to keep unreachable VMs when the deployment fails.

2 Login your dev-box

After the deployment succeeded, you can find the resource group with the name you specified on Azure Portal. The VM in the resource group with the name you specified is your dev-box.

• For Linux/Mac Users

  You can find the output SSHDEVBOX in the latest deployment. SSHDEVBOX is a string just like ssh <adminUsername>@<vm-name-in-lower-case>.<location>.cloudapp.azure.com.

  

  

  The private key is ~/.ssh/id_rsa if you accept the default location and name when you generate the ssh key. If so, please run ssh <adminUsername>@<vm-name-in-lower-case>.<location>.cloudapp.azure.com to login your dev-box.

  If you specified the location and name of the private key, please run ssh <adminUsername>@<vm-name-in-lower-case>.<location>.cloudapp.azure.com -i <path-to-your-private-key>.

• For Windows Users

  1. Open Putty.
  2. Fill in Public IP address/DNS name label of the dev-box.
  3. Before selecting Open, click the Connection > SSH > Auth tab to choose your private key (.ppk).

After you login, you can check ~/install.log to determine the status of the deployment. When the deployment succeeds, you will find Finish at the end of the log file and no ERROR message in it.

3 Deploy BOSH

If you have enabled the parameter autoDeployBosh, you can skip this step.

3.1 Configure

The ARM template pre-creates and configure the deployment manifest file bosh.yml in ~/example_manifests. You do not need to update it unless you have other specific configurations.

3.2 Deploy

Run the following commands in your home directory to deploy bosh:

./deploy_bosh.sh

NOTE:

• Never use root to perform these steps.
• More verbose logs are written to ~/run.log.

4 Deploy Concourse

If you have enabled the parameter autoConcourse, you can skip this step.

4.1 Configure

The ARM template pre-creates and configure the cloud-config manifest file cloud-config.yml and the deployment manifest file concourse.yml in ~/example_manifests. You do not need to update it unless you have other specific configurations.

4.2 Deploy

Run the following commands in your home directory to deploy concourse:

./deploy_concourse.sh

5 Connect to your Concourse

After Concourse had been successfully deployed, you can find the output CONCOURSEENDPOINT in the latest deployment in your resource group, you can connect to it by the address using your favorite web browser or Fly CLI.

Tags: Microsoft.Network/networkSecurityGroups, Microsoft.Storage/storageAccounts, Microsoft.Network/publicIPAddresses, Microsoft.Network/virtualNetworks, Microsoft.Network/networkInterfaces, Microsoft.Resources/deployments, Microsoft.Compute/virtualMachines, Microsoft.Compute/virtualMachines/extensions, CustomScript