How to always use new docker named volume for every web app deployment?

Question

How to always use new docker named volume for every web app deployment?

Ting Chou 1

Hi,

I am using docker compose for my web app and using docker named volume for file sharing between containers. Every time the images are built, the data will be written into the volume. My problem is after I rebuild the images (get new data and write into the volume) and redeploy, the web app still use the old volume with old data. But what I want is always using the new volume (same name to the old one but having different data) after deployment.

To achieve this, in local I will need to run docker volume rm cronData to remove the old local named volume before I run docker-compose up. But, how can we do this using in the Azure DevOps pipeline?

This is the current yml file for web app deployment in Azure DevOps release pipeline.

steps:
- task: AzureWebAppContainer@1
  displayName: 'Azure Web App on Container Deploy: '
  inputs:
    azureSubscription: 'xxx
    appName: 'xxx'
    multicontainerConfigFile: '$(System.DefaultWorkingDirectory)/xxx/xxx/docker-compose-configure.yml'
    appSettings: ' -WEBSITE_LOCAL_CACHE_OPTION Never'

And, this is my docker-compose.yml file. docker-compose-configure.yml is same to the docker-compose.yml execpt for the image field (pull image from ACR) and the no build field.

version: "3"
services:
  shiny:
    build: ./shiny
    image: shiny
    restart: always
    volumes:
      - cronData:/srv/shiny-server/cronData
    expose: 
      - 3838
    ports: 
      - 3838:3838

  cron:
    build: ./cron
    image: cron
    volumes:
      - cronData:/task/cronData

volumes:
  cronData:

Thanks in advance!!

ajkuma 28,036 Reputation points Microsoft Employee Moderator

2020-11-06T19:36:46.937+00:00

@Ting Chou , Thanks for posting this good question with a detailed description. I'm checking on this internally and will get back to you soon.
ajkuma 28,036 Reputation points Microsoft Employee Moderator

2020-11-07T19:24:39.733+00:00

@Ting Chou , While I was reviewing the scenario, I wish to highlight a couple of important App Service container process and how the images are stored for adding better clarity.

The first time you run a custom Docker image in App Service, App Service does a docker pull and pulls all image layers. These layers are stored on the disk (just like if you were using Docker on-premises), so every time the app restarts, App Service does a docker pull, but only pulls layers that have altered. If there have been no changes, then in this case, the App Service uses existing layers on the local disk.
If the app changes compute instances for any reason (typically- scaling-up/down the tiers or scale-out (instances), App Service will pull down all layers again.

You can enable persistent storage setting the WEBSITES_ENABLE_APP_SERVICE_STORAGE app setting.
In your docker-compose.yml file, you can map the volumes option to ${WEBAPP_STORAGE_HOME}.
ajkuma 28,036 Reputation points Microsoft Employee Moderator

2020-11-07T19:41:03.79+00:00

@Ting Chou , Adding more to the above (if you're referring to BYOS)

The files will be picked-up from Azure Storage being used for BYOS. If we change the File share, the content will also change.

Here, myvol would be the custom-id. So, you could upload the new contents to another "Storage Container" and update this Storage Mount. The custom-id would remain the same,
if the older content is not required then all the files in it can be removed.

Kindly refer this document for more details.
Custom Container FAQ

Just to understand more on the requirement, added the above info, kindly let us know if this is the case you're trying to accomplish.

2 answers

Your answer

ajkuma 28,036 Reputation points Microsoft Employee Moderator

2020-11-06T19:36:46.937+00:00

@Ting Chou , Thanks for posting this good question with a detailed description. I'm checking on this internally and will get back to you soon.
ajkuma 28,036 Reputation points Microsoft Employee Moderator

2020-11-07T19:24:39.733+00:00

@Ting Chou , While I was reviewing the scenario, I wish to highlight a couple of important App Service container process and how the images are stored for adding better clarity.

The first time you run a custom Docker image in App Service, App Service does a docker pull and pulls all image layers. These layers are stored on the disk (just like if you were using Docker on-premises), so every time the app restarts, App Service does a docker pull, but only pulls layers that have altered. If there have been no changes, then in this case, the App Service uses existing layers on the local disk.
If the app changes compute instances for any reason (typically- scaling-up/down the tiers or scale-out (instances), App Service will pull down all layers again.

You can enable persistent storage setting the WEBSITES_ENABLE_APP_SERVICE_STORAGE app setting.
In your docker-compose.yml file, you can map the volumes option to ${WEBAPP_STORAGE_HOME}.
ajkuma 28,036 Reputation points Microsoft Employee Moderator

2020-11-07T19:41:03.79+00:00

@Ting Chou , Adding more to the above (if you're referring to BYOS)

The files will be picked-up from Azure Storage being used for BYOS. If we change the File share, the content will also change.

Here, myvol would be the custom-id. So, you could upload the new contents to another "Storage Container" and update this Storage Mount. The custom-id would remain the same,
if the older content is not required then all the files in it can be removed.

Kindly refer this document for more details.
Custom Container FAQ

Just to understand more on the requirement, added the above info, kindly let us know if this is the case you're trying to accomplish.

Answer 1

Ting Chou 1

hi @Anonymous ,

Thank you for the prompt response. Appreciate that. :)

I have tried to use ${WEBAPP_STORAGE_HOME} to map the volumes using ${WEBAPP_STORAGE_HOME}/cronData:/task/cronData ), but I found that since in the beginning the directory ( ${WEBAPP_STORAGE_HOME}/cronData ) will be empty, so if I mount my volume to that directory, then my original data in /task/cronData will be wiped out. But I do need the initial data in /task/cronData which is written in in the building stage.

I guess the things I want to achieve is files from the container mounted to the host(https://stackoverflow.com/questions/42395748/docker-compose-volume-is-empty-even-from-initialize) rather than the other way around. And it seems that there is no way for us to run the docker commands to the web app host machine, so I don't think I can use the walk around suggested from the above link.

I think BYOS can be a good solution. I can write the data into blob storage in the building stage and mount it back to the containers. However, after reading the documents, I am still not quite sure how to properly configure this. I can create a New Azure Storage Mount like how you did, then what I should specify in my docker-compose.yml and docker-compose-configure.yml file?

Thank you!

ajkuma 28,036 Reputation points Microsoft Employee Moderator

2020-11-10T21:00:26.82+00:00

@Ting Chou , Thanks for the follow-up and sharing additional details.

I was trying to clarify if you were already leveraging BYOS. In that case the Azure Storage in App Service is in preview for App Service on Linux and Web App for Containers. It's not supported for production scenarios. Just to highlight, you can explicitly control which container is the web container in your multi-container deployment by using the WEBSITES_WEB_CONTAINER_NAME app setting.

In a Docker Compose YAML file, the value for this app setting should be the service name listed in your YAML file.

yes by default, we don't persist files in your site. To persist files with Storage, with this we mount the /home directory to Azure Storage. If your app is restarted, any files saved within the /home directory and sub-directories will persist. However, any files or directories outside of the /home directory will not be persisted between site recycles.
ajkuma 28,036 Reputation points Microsoft Employee Moderator

2020-11-10T21:09:41.73+00:00

@Ting Chou , Just to add more clarity, yes App service overwrite anything that's in the /home directory with the content from the Azure Storage location. (For a brand new app, it will remove what's currently in /home and the /home directory will then be empty.)

If you copy data to the /home directory in the build phase of your Docker image, that data will be removed when the Docker container is started. Therefore, you should make sure that any data you write to the file system in your build phase is written outside of the /home directory structure. (this applies for persisting files.)

Also the property 'build' of the docker-compose is not support in Azure App Service. If your requirement fits, you create the image locally and then push them to a docker registry, for example, the Azure Container Registry.

Having mentioned above, I will connect with you offline on this for additional info.

Answer 2

@Ting Chou , Just summarizing the comments with additional info.
With the current limitations (below) and based on your requirement, you may try something like these. If you have any further questions, kindly connect with me offline for more detailed insights.

As mentioned above (comments), based on your scenario, the Multi-containers would offer a limited support (as it is in preview currently).

  a) Multi-container support on Web Apps for Containers is in preview and *isn’t recommend* for production use.    
  b) “*build*”, *depends_on*, *ports other than 80 and 8080* are not a supported as Docker Compose option on multi-containers ([see here][1]).  
  c) Bring your own storage only supports Azure Files for read/write and is also in preview.

The docker-compose.yml provided should expose “web”, how to deal with that as outlined in this doc.

1. Build the images push them to ACR or any other repo.  
2. Expose a web container for App Service to ping for container health checks.  
3. Mount Azure Files as the volume for your CRON container so that content is added during build.

Additionally, to run cron jobs:

• For Custom Containers

Enable CRON job for custom containers:
To Enable cron, add the following line to Dockerfile:
RUN apt-get install -yqq cron
And following line in the container init script (init_container.sh)
(crontab -l && echo "* * * * * echo 'hello from cron' >> /home/site/wwwroot/cron1.txt")|crontab
In this case we will write “hello from cron” to file /home/site/wwwroot/cron1.txt every minute. You can replace this with anything else you wish to run periodically.
echo 'hello from cron' >> /home/site/wwwroot/cron1.txt

For Blessed Images
1.Create a script file in /home directory of the App Service and trigger it via the startup command under Configuration blade.
2. A test script which installs cron as well as create a CRON job:

   #!/bin/sh  
  apt-get update -qq && apt-get install cron -yqq  
  service cron start  
  mkdir /home/BackupLogs  
 (crontab -l 2>/dev/null; echo "*/5 * * * * cp /home/LogFiles/*.log /home/BackupLogs")|crontab

[1] Document : Supported and Unsupported docker composition

Apologies for any inconvenience! Hope this helps! If you have any further questions, kindly let us know.

Ting Chou 1 Reputation point

2020-11-11T17:22:55.683+00:00

hi @Anonymous ,

Thank you so much for all the information!

I will spend following days doing some experiments based on your suggestions and let you know how things go. According to you posts, I just feel that I might have some misunderstanding about how the web app service works. Let me try something first and get back to you.

Really appreciate for all the assistance. Contact to you later. :)
ajkuma 28,036 Reputation points Microsoft Employee Moderator

2020-11-12T12:32:11.323+00:00

@Ting Chou , Glad you found the information helpful.
Certainly, kindly do try the suggestions that I have shared and let us know how it goes or you need further assistance, we're here to help you!

Thanks for the follow-up and your kind co-operation :)

Share via

How to always use new docker named volume for every web app deployment?

2 answers

Your answer