Cloud AI Ecosystem Team Blog

TensorFlow on Azure: Enabling Blob Storage via Alluxio

Jing Chang - MSFT — Tue, 01 May 2018 18:06:27 GMT

Many customers Cloud AI Ecosystem in Microsoft works with, choose Azure Blob Storage as their data storage. Among those customers, if one wants to use TensorFlow to develop deep learning models, unfortunately TensorFlow does not support Azure Blob storage out of box as its custom file system plugin¹. There is no easy way to directly feed data from Azure block blobs into TensorFlow’s input pipeline². In case of setting up a Kubernetes cluster for TensorFlow workloads, Azure Blob Storage is not included in k8s’ supported Types of Volumes³. Given above, the odds are either to mount an Azure File share to Kubernetes pods and read remote Azure files via the mount path, or to manually copy data to a local SSD disk of each pod. Those approaches are summarized as part of ‘Deep Learning Toolkits with Kubernetes Clusters’ published at https://aka.ms/deeplearningk8s; however, many people would prefer Azure Blob to Azure File service because of their different performance, scale and pricing options⁵; or in the case of manual copy to local SSD, it’s not scalable to repeat that operation with big amount of training data. In this blog we introduce Alluxio’s newly released FUSE⁶ feature in Alluxio 1.7, which enables mounting Azure Blob storage to the local file system namespace, and solves integration between TensorFlow and Azure Blobs. It aims to bridge high computation workloads including TensorFlow jobs with underlying storage system, via its unified Alluxio data access layer. Alluxio-FUSE feature opens up new opportunities for Azure Blobs to be directly fed into your tensors; moreover, with current progress made on GPU computation, the input pipeline might become a bottleneck if the storage is not performant enough. The effort Alluxio has paid to optimize data access layer brings positive impact to the DL data input pipeline. For more details, please refer to https://alluxio.com/blog/flexible-and-fast-storage-for-deep-learning-with-alluxio. Follow the simple steps below to check out how to enable Azure Blobs via Alluxio-FUSE to run TensorFlow jobs on Azure.

Set up A Kubernetes Cluster

A sample k8s cluster on Azure Container Service is deployed using open source toolkit DLWorkspace⁸, the documentation can be found at https://microsoft.github.io/DLWorkspace/. The sample setup includes one master node of Standard D2 v2 Azure VM(2 vcpus, 7GB memory), and two agent nodes of Standard NC12(12 vcpus, 112 GB memory) Azure VM. To check if the GPU driver is correctly installed after deployment, run ‘nvidia-smi’ on each agent node to find the driver information.

Create Alluxio-FUSE Enabled Pods

For ease of use, Alluxio includes Docker integration⁹ and published its 1.7 Docker images¹⁰ on Docker Hub, we can pull images into the k8s cluster and create Alluxio-FUSE enabled k8s pods. Alluxio servers consist of two architectural components¹¹: master and workers, where the master is responsible for managing global metadata and workers are responsible for managing local storage resources allocated to Alluxio. We co-locate Alluxio master and TensorFlow parameter server on the same pod, also Alluxio workers and TensorFlow workers on the same pod, for better data locality. Find sample pod configuration files posted at https://github.com/jichang1/TensorFlowonAzure/tree/master/Alluxio to create your k8s pods: first tf-ps pod and then tf-worker pod. Note you will replace $yourcontainername$, $yourstorageaccountname$, and $yourstorageaccountkey$. Replace $yourpsserverip$ with the IP found from /etc/hosts of the tf-ps pod. The sample container configuration below, tells that the docker image runs /entrypoint.sh upon initialization with argument “worker”; the worker pod communicates with the master pod using port 19998; a few environment variables need to be defined upon initialization such as master host name, storage account, etc; the process runs in privileged ‘SYS_ADMIN’ mode.

containers:

      - name: tf-worker0

        image: alluxio/alluxio-tensorflow:1.7.0-1.3.0-gpu

        command: ["/entrypoint.sh"]

        args: ["worker"]

        ports:

          - containerPort: 19998

          name: alluxioport

        env:

        - name: ALLUXIO_MASTER_HOSTNAME

          value: "$yourpsserverip$"

        - name: ALLUXIO_RAM_FOLDER

          value: "/opt/ramdisk"

        - name: ALLUXIO_WORKER_MEMORY_SIZE

          value: "10GB"

        - name: ALLUXIO_UNDERFS_ADDRESS

          value: "wasb://$yourcontainername$@$yourstorageaccountname$.blob.core.windows.net/"

        - name: FS_AZURE_ACCOUNT_KEY_$yourstorageaccountname$_BLOB_CORE_WINDOWS_NET

          value: $yourstorageaccountkey$

        resources:

          limits:

            alpha.kubernetes.io/nvidia-gpu: 1

        volumeMounts:

        - mountPath: /usr/local/nvidia

          name: nvidia-driver

        - mountPath: /opt/ramdisk

          name: ramdisk

        - mountPath: /etc/resolv.conf

          name: resolv

        securityContext:

          privileged: true

          capabilities:

            add: ["SYS_ADMIN"]

            nodeSelector:

        FragmentGPUJob: active

      volumes:

      - name: nvidia-driver

        hostPath:

          path: /opt/nvidia-driver/current

      - name: ramdisk

        hostPath:

          path: /mnt/ramdisk

      - name: resolv

        hostPath:

          path: /etc/resolv.conf

After executing

sudo kubectl apply -f ./alluxio-fuse-tfgpu-psserver0.yaml

sudo kubectl apply -f ./alluxio-fuse-tfgpu-worker0.yaml

Run

kubectl get pods

to check pods are up and in healthy state. Connect to tf-ps or tf-worker pods, access your blob storage via ‘ls /alluxio-fuse’ which is already mounted to local file system.

Run TensorFlow Jobs

We take TensorFlow benchmark jobs¹² as an example. On the parameter server pod, run the command below:

python tf_cnn_benchmarks.py --local_parameter_device=gpu --num_gpus=2  --batch_size=128  --model=googlenet  --variable_update=parameter_server --num_batches=50  --cross_replica_sync=False  --data_name=imagenet --data_dir=file:///alluxio-fuse/  --job_name=ps --ps_hosts=10.244.2.2:2222  --worker_hosts=10.244.0.2:2222 --task_index=0

On the worker pod, run the command below:

python tf_cnn_benchmarks.py --local_parameter_device=gpu --num_gpus=2  --batch_size=128 --model=googlenet --variable_update=parameter_server --num_batches=50 --cross_replica_sync=False --data_name=imagenet --data_dir=file:///alluxio-fuse/  --job_name=worker  --ps_hosts=10.244.2.2:2222  --worker_hosts=10.244.0.2:2222 --task_index=0

You should observe output similar to below

name: Tesla K80

major: 3 minor: 7 memoryClockRate (GHz) 0.8235

pciBusID 0e7c:00:00.0

Total memory: 11.17GiB

Free memory: 11.09GiB

……

2018-01-11 02:22:52.188017: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1045] Creating TensorFlow device (/gpu:0) -> (device: 0, name: Tesla K80, pci bus id: 0e7c:00:00.0)

2018-01-11 02:22:52.188038: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1045] Creating TensorFlow device (/gpu:1) -> (device: 1, name: Tesla K80, pci bus id: 27e6:00:00.0)

2018-01-11 02:22:52.402467: I tensorflow/core/distributed_runtime/rpc/grpc_channel.cc:215] Initialize GrpcChannelCache for job ps -> {0 -> 10.244.2.2:2222}

2018-01-11 02:22:52.402510: I tensorflow/core/distributed_runtime/rpc/grpc_channel.cc:215] Initialize GrpcChannelCache for job worker -> {0 -> localhost:2222}

2018-01-11 02:22:52.405246: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:316] Started server with target: grpc://localhost:2222

TensorFlow:  1.3

Model:       googlenet

Mode:        training

Batch size:  256 global

             128 per device

Devices:     ['/job:worker/task:0/gpu:0', '/job:worker/task:0/gpu:1']

Data format: NCHW

Optimizer:   sgd

Variables:   parameter_server

Sync:        False

==========

Generating model

2018-01-11 02:24:26.461062: I tensorflow/core/distributed_runtime/master_session.cc:998] Start master session b26ef5a1286e9840 with config: intra_op_parallelism_threads: 1 gpu_options { force_gpu_compatible: true } allow_soft_placement: true

Running warm up

Done warm up

Waiting for other replicas to finish warm up

Starting real work at step 10 at time Thu Jan 11 02:25:57 2018

Step    Img/sec loss

1       images/sec: 200.3 +/- 0.0 (jitter = 0.0)        7.093

10      images/sec: 189.8 +/- 1.6 (jitter = 4.7)        7.093

20      images/sec: 186.3 +/- 1.3 (jitter = 6.3)        7.093

30      images/sec: 186.6 +/- 1.1 (jitter = 6.0)        7.093

40      images/sec: 186.8 +/- 0.9 (jitter = 5.5)        7.093

Finishing real work at step 59 at time Thu Jan 11 02:27:04 2018

50      images/sec: 187.4 +/- 0.8 (jitter = 5.3)        7.093

----------------------------------------------------------------

total images/sec: 186.67

----------------------------------------------------------------

We hope this blog has provided you a new way of running TensorFlow jobs on Azure with underlying Azure Blob storage.

Reference

1 https://www.tensorflow.org/extend/add_filesys 2 https://www.tensorflow.org/programmers_guide/datasets 3 https://kubernetes.io/docs/concepts/storage/volumes/#types-of-volumes 4 https://docs.microsoft.com/en-us/azure/storage/common/storage-scalability-targets 5 https://azure.microsoft.com/en-us/pricing/details/storage/ 6 https://www.alluxio.org/docs/master/en/Mounting-Alluxio-FS-with-FUSE.html 7 https://www.alluxio.org/docs/master/en/index.html 8 https://github.com/Microsoft/DLWorkspace 9 https://github.com/Alluxio/alluxio/tree/master/integration/docker 10 https://hub.docker.com/r/alluxio/alluxio-tensorflow/ 11 https://www.alluxio.org/docs/master/en/Architecture.html 12 https://github.com/tensorflow/benchmarks

How to Make Python Easier for the R User: revoscalepy

Siddarth Ramesh — Mon, 20 Nov 2017 17:50:54 GMT

Python for the R User

I’m an R programmer. To me, R has been great for data exploration, transformation, statistical modeling, and visualizations. However, there is a huge community of Data Scientists and Analysts who turn to Python for these tasks. Moreover, both R and Python experts exist in most analytics organizations, and it is important for both languages to coexist. Many times, this means that R coders will develop a workflow in R but then must redesign and recode it in Python for their production systems. If the coder is lucky, this is easy, and the R model can be exported as a serialized object and read into Python. There are packages that do this, such as pmml. Unfortunately, many times, this is more challenging because the production system might demand that the entire end to end workflow is built exclusively in Python. That’s sometimes tough because there are aspects of statistical model building in R which are more intuitive than Python. Python has many strengths, such as its robust data structures (like Dictionaries), compatibility with Deep Learning and Spark, and its ability to be a multipurpose language. However, many scenarios in enterprise analytics require people to go back to basic statistics and Machine Learning, for which the classic Data Science packages in Python are not as intuitive as R. The key difference is that many statistical methods are built into R natively. As a result, there is a gap for when R users must build workflows in Python. To try to bridge this gap, this post will discuss a relatively new package developed by Microsoft, revoscalepy.

Why revoscalepy?

Revoscalepy is the Python implementation of the R library RevoScaleR. The methods in ‘revoscalepy’ are the same, and more importantly, the way the R user can view data is the same. The reason this is so important is that for an R programmer, being able to understand the data shape and structure is one of the challenges with getting used to Python.

In Python, data types are different, preprocessing the data is different, and the criteria to feed the processed dataset into a model is different. To understand how revoscalepy eases the transition from R to Python, the following section will compare building a decision tree using revoscalepy with building a decision tree using sklearn. The Titanic dataset from Kaggle will be used for this example. To be clear, this post is written from an R user’s perspective, as many of the challenges this post will outline are standard practices for native Python users.

Revoscalepy Versus Sklearn

Dependencies of revoscalepy

Revoscalepy works on Python 3.5, and can be downloaded as a part of Microsoft Machine Learning Server. Once downloaded, set the Python environment path to the python executable in the MML directory, and then import the packages.

Data Import

The first chunk of code imports the revoscalepy, numpy, pandas, and sklearn packages. Pandas has some R roots in that it has its own implementation of DataFrames as well as methods that resemble R’s exploratory methods.

import revoscalepy as rp
import numpy as np
import pandas as pd
import sklearn as sk
titanic_data = pd.read_csv('titanic.csv')
titanic_data.head()

Preprocessing

sklearn One of the challenges as an R user with using sklearn is that the decision tree model for sklearn can only handle the numeric datatype. Pandas has a categorical type that looks like factors in R, but sklearn’s decision tree does not integrate with this. As a result, numerically encoding the categorical data becomes a mandatory step. This example will use a one-hot encoder to shape the categories in a way that sklearn’s decision tree understands. The side effect of having to one-hot encode variables is that if the dataset contains high cardinality features, it can be memory intensive and computationally expensive because each category becomes its own binary column. While implementing one-hot encoding itself is not a difficult transformation in Python and provides good results, it is still an extra step for an R programmer to have to manually implement. The following chunk of code detaches the categorical columns, label and one-hot encodes them, and then reattaches the encoded columns to the rest of the dataset.

from sklearn import tree
le = sk.preprocessing.LabelEncoder()
x = titanic_data.select_dtypes(include=[object])
x = x.drop(['Name', 'Ticket', 'Cabin'], 1)
x = pd.concat([x, titanic_data['Pclass']], axis = 1)
x['Pclass'] = x['Pclass'].astype('object')
x = pd.DataFrame(x)
x = x.fillna('Missing')
x_cats = x.apply(le.fit_transform)
enc = sk.preprocessing.OneHotEncoder()
enc.fit(x_cats)
onehotlabels = enc.transform(x_cats).toarray()
encoded_titanic_data =
     pd.concat([pd.DataFrame(titanic_data.select_dtypes(include=[np.number])),
     pd.DataFrame(onehotlabels)], axis = 1)

At this point, there are more columns than before, and the columns no longer have semantic names (they have been enumerated). This means that if a decision tree is visualized, it will be difficult to understand without going through the extra step of renaming these columns. There are techniques in Python to help with this, but it is still an extra step that must be considered. revoscalepy Unlike sklearn, revoscalepy reads pandas’ ‘category’ type like factors in R. This section of code iterates through the DataFrame, finds the string types, and converts those types to ‘category’. In pandas, there is an argument to set the order to False, to prevent ordered factors.

titanic_data_object_types = titanic_data.select_dtypes(include = ['object'])
titanic_data_object_types_columns = np.array(titanic_data_object_types.columns)
for column in titanic_data_object_types_columns:
     titanic_data[column] = titanic_data[column].astype('category', ordered = False)
titanic_data['Pclass'] = titanic_data['Pclass'].astype('category', ordered = False)

This dataset is already ready to be fed into the revoscalepy model.

Training Models

sklearn One difference between implementing a model in R and in sklearn in Python is that sklearn does not use formulas. Formulas are important and useful for modeling because they provide a consistent framework to develop models with varying degrees of complexity. With formulas, users can easily apply different types of variable cases, such as ‘+’ for separate independent variables, ‘:’ for interaction terms, and ‘*’ to include both the variable and its interaction terms, along with many other convenient calculations. Within a formula, users can implement mathematical calculations, create factors, and include more complex entities like third order interactions. In addition, with formulas, programmers can build extraordinarily complex models such as mixed effect models, which are next to impossible build without them. In Python, there are packages such as ‘statsmodels’ which have more intuitive ways to build certain statistical models. However, statsmodels has a limited selection of models, and does not include tree based models.

With sklearn, model.fit expects the independent and dependent terms to be columns from the DataFrame. Interactions must be created manually as a preprocessing step for more complex examples. The code below trains the decision tree:

model = tree.DecisionTreeClassifier(max_depth = 50)
x = encoded_titanic_data.drop(['Survived'], 1)
x = x.fillna(-1)
y =  encoded_titanic_data['Survived']
model = model.fit(x,y)

revoscalepy Revoscalepy brings back formulas. Granted, users cannot view the formula the same way as they can in R, because formulas are strings in Python. However, importing code from R to Python is an easy transition because formulas are read the same way in the revoscalepy functions as the model-fit functions in R. The code below fits the decision tree in revoscalepy:

form = 'Survived ~ Pclass + Sex + Age  + Parch  + Fare + Embarked'
titanic_data_tree = rp.rx_dtree(form, titanic_data, max_depth = 50)

The resulting object, titanic_data_tree, is the same structural object that RxDTree() would create in R. RxDTree() integrates with the rpart library in R, meaning rx_dtree() indirectly also has this integration if the users wants to export the object back into R for further exploratory analysis around this decision tree.

Conclusion

From the workflow, it should be clear how revoscalepy can help with transliteration between R and Python. Sklearn has different preprocessing considerations because the data must be fed into the model differently. The advantage to revoscalepy is that R programmers can easily convert their R code to Python without thinking too much about the ‘Pythonic way’ of implementing their R code. Categories replace factors, rx_dtree() reads the R-like formula, and the arguments of the function are similar to their R equivalent. Looking at the big picture, revoscalepy is one way to ease Python for the R user and future posts will cover more ways to transition between R and Python.