Jian Wu's Blog. https://docs.microsoft.com/archive/blogs/jianwu/feed.xml Wed, 17 Apr 2019 04:05:44 GMT en-US hourly 1 Ref: 导出HDInsight数据到MySQL（Azure） https://docs.microsoft.com/archive/blogs/jianwu/ref-hdinsightmysqlazure Fri, 09 Oct 2015 23:56:00 GMT https://blogs.msdn.microsoft.com/jianwu/2015/10/09/ref-hdinsightmysqlazure/ 在大数据计算中，经常会有一个场景：在Hadoop分布式存储DFS和关系型数据库（MySQL、Oracle、SQL Server）间导入（导出）数据。由于Azure HDInsight是基于公有云的托管式Hadoop，用户在执行数据转移时，与云上Hadoop相连的关系型数据库往往不是本地的数据库引擎，取而代之的是公有云提供的关系型数据库服务，如SQL Database和MySQL on Azure。针对如何使用HDInsight中的工具Sqoop导入（导出）数据到SQL Database，Azure官方文档给出了参考教程，本文具体实践如何使用Sqoop（HDInsight）导出数据到MySQL on Azure的场景。

关于HDInsight：https://azure.microsoft.com/en-us/documentation/articles/hdinsight-high-availability/，本实践中将会提交任务到HDInsight集群，集群中的Head Node分配任务到Worker Node，Worker Node具体实施数据的读取、处理和转移。


 

关于MySQL on Azure：https://www.windowsazure.cn/home/features/mysql/

关于Sqoop: https://sqoop.apache.org/docs/1.4.4/SqoopUserGuide.html

关于HDInsight和Azure Storage：https://azure.microsoft.com/en-us/documentation/articles/hdinsight-hadoop-use-blob-storage/，HDInsight中的数据以冗余的形式存储在Azure Storage中，HDInsight节点可以通过分布式存储路径（WASB）来读写目标数据。同时，用户也可以直接访问Azure Storage对应容器中的HDInsight数据文件。

 

注：本实践通过Azure PowerShell来实现，主要包括：

  1. 创建MySQL数据表，用于接收Sqoop导出的数据。
  2. 上传数据样本文件到HDInsight的存储（Azure Storage）
  3. 提交数据导出任务（Sqoop Job）到HDInsight集群
  4. 跟踪Sqoop任务执行情况并检查输出结果。

 

第一步：准备MySQL数据表

#MySQL variables = "mysqlservices-sha.chinacloudapp.cn"

$MySQLLogin = "login1@jianwmysql"  #注意格式： 登录用户@MySQL服务器名

$MySQLPassword = "password"

$MySQLExportedDBName = "testdb01"

$tableName = "jtable3"

 

$confirmation= Read-Host "To create a new MySQL table, input 'y'; use existing MySQL table, input anything else ..."

if ($confirmation -eq 'y') {

    $MySQLAdminUserName =$MySQLLogin;

    $MySQLAdminPassword =$MySQLPassword;

    $MySQLDatabase =$MySQLExportedDBName;

    $MySQLHost =$MySQLServerName;

    $ConnectionString ="server=" +$MySQLHost +";port=3306;uid=" + $MySQLAdminUserName+ ";pwd="+ $MySQLAdminPassword + ";database="+$MySQLDatabase

    $Query = "CREATE TABLE " + $tableName + "(id VARCHAR(40),name VARCHAR(40))"

 

    Try {

      [void][System.Reflection.Assembly]::LoadWithPartialName("MySql.Data")

      $Connection = New-Object MySql.Data.MySqlClient.MySqlConnection

      $Connection.ConnectionString = $ConnectionString

      $Connection.Open()

 

      $Command = New-Object MySql.Data.MySqlClient.MySqlCommand($Query, $Connection)

      $DataAdapter = New-Object MySql.Data.MySqlClient.MySqlDataAdapter($Command)

      $DataSet = New-Object System.Data.DataSet

      $RecordCount = $dataAdapter.Fill($dataSet, "data")

      $DataSet.Tables[0]

 

      $Query = "select count(*) from " + $tableName

      $Command = New-Object MySql.Data.MySqlClient.MySqlCommand($Query, $Connection)

      $DataAdapter = New-Object MySql.Data.MySqlClient.MySqlDataAdapter($Command)

      $DataSet = New-Object System.Data.DataSet

      $RecordCount = $dataAdapter.Fill($dataSet, "data")

      $DataSet.Tables[0].Rows[0].ItemArray[0].ToString();

      [System.Console]::WriteLine("MySQL table was created successfully.");

      }

    Catch {

      Write-Host "ERROR : Unable to run query : $query `n$Error[0]"

     }

    Finally {

      $Connection.Close()

      }

  }

 

第二步：上传样本数据

Add-AzureAccount -Environment azurechinacloud

# Define the cluter variables

$clusterName = "hadoopjianw3"

$storageAccountName = "jianwstorage"

$containerName = "hadoopjianw3"

 

#Upload a file into Azure storage (DFS of Hadoop)

$StorageAccountKey = Get-AzureStorageKey -StorageAccountName $storageAccountName

$Ctx = New-AzureStorageContext
$StorageAccountName -StorageAccountKey $StorageAccountKey.Primary

Get-ChildItem -Path d:\log2copy.txt | Set-AzureStorageBlobContent -Container $containerName -Blob tutorials2/log2copy.txt -Context $Ctx

此处用到的样本文件log2copy.txt，含有以下样本数据（分隔符为tab跳格），该样本文件可见附件。

mytest  123

world    456

 

第三步：提交Sqoop任务到HDInsight

[System.Console]::WriteLine("Start Sqoop-MySQL test ...");

 #Add-AzureAccount

Add-AzureAccount -Environment azurechinacloud

 

# Define the cluter variables

$clusterName = "hadoopjianw3"

$storageAccountName = "jianwstorage"

$containerName = "hadoopjianw3"

 

# Connection string for Azure MySQL Database.

$conStr = "jdbc:mysql://$MySQLServerName/$MySQLExportedDBName"

#Hive file location

$exportDir = "wasb://$containerName@$storageAccountName.blob.core.chinacloudapi.cn/tutorials2"

$sqoopDef = New-AzureHDInsightSqoopJobDefinition -Command "export --connect $conStr --username $MySQLLogin --password $MySQLPassword --table $tableName --export-dir $exportDir --fields-terminated-by \t -m 1"

 

$sqoopJob = Start-AzureHDInsightJob -Cluster $clusterName -JobDefinition $sqoopDef -Debug -Verbose

 

Wait-AzureHDInsightJob -WaitTimeoutInSeconds 3600 -Job $sqoopJob

Write-Host "Standard Error" -BackgroundColor Green

Get-AzureHDInsightJobOutput -Cluster $clusterName -JobId $sqoopJob.JobId -StandardError

Write-Host "Standard Output" -BackgroundColor Green

Get-AzureHDInsightJobOutput -Cluster $clusterName -JobId $sqoopJob.JobId -StandardOutput  

 

第四步：跟踪Sqoop任务并检查输出

查看HDInsight的控制台主页

 

查看Sqoop Job的执行结果：

 

查看MySQL中的结果数据

 

 

调试与问题解决：

如果建立的HDInsight集群采用的是较低版本的Hadoop，Sqoop任务执行过程中可能抛出以下错误，该错误的原因是在HDInsight集群中找不到对应的MySQL Driver，此类问题的手动解决方法是：用户手动copy相应的MySQL Driver（mysql-connector-java-5.1.36-bin.jar）到HDInsight的C:\apps\dist\sqoop-<version>\lib目录下。MySQL Driver可以在https://www.mysql.com/products/connector/下载。

15/09/21 03:35:12 INFO sqoop.Sqoop: Running Sqoop version: 1.4.5.2.2.7.1-0004

15/09/21 03:35:12 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.

15/09/21 03:35:13 INFO manager.SqlManager: Using default fetchSize of 1000

15/09/21 03:35:13 INFO tool.CodeGenTool: Beginning code generation

15/09/21 03:35:13 ERROR sqoop.Sqoop: Got exception running Sqoop: java.lang.RuntimeException: Could not load db driver class: com.mysql.jdbc.Driver

java.lang.RuntimeException: Could not load db driver class: com.mysql.jdbc.Driver

      at org.apache.sqoop.manager.SqlManager.makeConnection(SqlManager.java:848)

      at org.apache.sqoop.manager.GenericJdbcManager.getConnection(GenericJdbcManager.java:52)

      at org.apache.sqoop.manager.SqlManager.execute(SqlManager.java:736)

      at org.apache.sqoop.manager.SqlManager.execute(SqlManager.java:759)

      at org.apache.sqoop.manager.SqlManager.getColumnInfoForRawQuery(SqlManager.java:269)

      at org.apache.sqoop.manager.SqlManager.getColumnTypesForRawQuery(SqlManager.java:240)

      at org.apache.sqoop.manager.SqlManager.getColumnTypes(SqlManager.java:226)

      at org.apache.sqoop.manager.ConnManager.getColumnTypes(ConnManager.java:295)

      at org.apache.sqoop.orm.ClassWriter.getColumnTypes(ClassWriter.java:1773)

      at org.apache.sqoop.orm.ClassWriter.generate(ClassWriter.java:1578)

      at org.apache.sqoop.tool.CodeGenTool.generateORM(CodeGenTool.java:96)

      at org.apache.sqoop.tool.ExportTool.exportTable(ExportTool.java:64)

      at org.apache.sqoop.tool.ExportTool.run(ExportTool.java:100)

      at org.apache.sqoop.Sqoop.run(Sqoop.java:143)

      at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)

      at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:179)

      at org.apache.sqoop.Sqoop.runTool(Sqoop.java:218)

      at org.apache.sqoop.Sqoop.runTool(Sqoop.java:227)

      at org.apache.sqoop.Sqoop.main(Sqoop.java:236)

[DBG]: PS C:\WINDOWS\system32>> 

此问题也可以通过HDInsight中的action script来解决，使用脚本的方法安装MySQL Driver到目标目录。

https://azure.microsoft.com/en-us/documentation/articles/hdinsight-hadoop-customize-cluster/

https://azure.microsoft.com/en-us/documentation/articles/hdinsight-hadoop-script-actions/

 

本实践的完整PowerShell脚本如附件，Azure用户修改其中的（MySQL、Storage和HDInsight）参数既可应用。

 

参考文档：

https://azure.microsoft.com/en-us/documentation/articles/hdinsight-use-sqoop/

https://azure.microsoft.com/en-us/documentation/articles/storage-powershell-guide-full/

https://sqoop.apache.org/docs/1.4.4/SqoopUserGuide.html

https://azure.microsoft.com/en-us/documentation/articles/hdinsight-hadoop-introduction/

https://azure.microsoft.com/en-us/documentation/articles/hdinsight-hadoop-use-blob-storage/

 

原文：https://blogs.msdn.com/b/cciccat/archive/2015/09/21/hdinsight-mysql-azure.aspx

sqoop-mysql-summary-3.2.ps1.txt

]]> Updated: 在Windows Azure平台上运行PHP应用 https://docs.microsoft.com/archive/blogs/jianwu/updated-windows-azurephp Wed, 26 Aug 2015 23:03:00 GMT https://blogs.msdn.microsoft.com/jianwu/2015/08/26/updated-windows-azurephp/ 参考：https://blogs.msdn.com/b/cciccat/archive/2015/08/19/windows-azure-php.aspx

 

PHP技术在互联网应用中非常流行，被广泛使用在网站应用、报表管理、脚本任务等场景。Windows Azure作为企业级公有云平台，不仅仅能无缝托管传统的微软开发技术（.Net），对于PHP、Java等业内通用的语言和技术也提供了极好的支持。本文就如何在Windows Azure平台上运行PHP应用总结出一些最佳实践和参考步骤。

 

一、 选择合适的Windows Azure服务来运行PHP应用

Windows Azure提供不同层次的公有云服务，主要包括IaaS（基础设施即服务：以虚拟机、虚拟网络为代表），PaaS（平台即服务：以网站应用、云服务、数据库为代表）和SaaS（软件即服务：以Office 365，Azure活动目录为代表），其中可以用作托管PHP应用的主要有三类服务：

-  虚拟机：用户在Azure中创建虚拟机，自行部署PHP环境并发布PHP应用，实践过程和在本地发布PHP服务一致。

-  网站应用（Web App）：Azure后端存在大量已实现好PHP环境的服务器，用户只需将PHP应用代码通过Git、FTP等工具上传到目标服务器即可。

-  云服务(Cloud Service)：Azure平台根据用户的设置，实时创建和定制化（一个或多个）虚拟机，自动安装用户指定的PHP环境并加载用户的PHP应用代码，实现PHP网站服务或者后台处理应用。

关于这三种方案的不同点和相同点，可以参见：https://azure.microsoft.com/en-us/documentation/articles/choose-web-site-cloud-service-vm/，总结下来，最突出的特点是：

方案类型

适用场景

优点

缺点

虚拟机

支持LAMP（另外两种场景下网站服务器都是IIS/Windows）；PHP应用依赖于手动安装的组件或应用。

用户完全控制，和本地实践一样。

 

手动部署，效率较低。

网站应用

快速部署，用户不需要更改服务器机器环境，较少自定义配置PHP网站服务器（拓展），默认的PHP环境包（如PHP5.4）可以满足应用需求。

即时部署、迁移、更新，用户不关注服务器环境设置和后台维护。

用户不能访问后台服务器。

需要很多配置工作来使用自定义PHP运行环境或者深度修改后台网站服务器。

 

云服务

用户需要定制化的服务器环境或运行初始化任务。

平衡部署效率和服务器定制化；支持访问后台服务器；支持同时托管多个网站或应用。

部署效率不及网站应用，服务器可定制化程度不及虚拟机。

 

这三种方案的形象对比如下图：

 

二、 在Windows Azure平台上运行PHP应用

 

1. 本地开发PHP应用

这一部分同传统的PHP应用开发完全一致，若迁移本地已有的PHP应用到Azure，可以跳过这一部分内容。

l   安装PHP运行环境(Apache + PHP + MySQL 或者XAMPP)，XAMPP 提供一键式快速安装, 可参考: https://netbeans.org/kb/docs/php/configure-php-environment-windows_zh_CN.html#XAMPP

l   安装PHP开发工具，如Eclipse，Zend Studio，PHPStorm等，也可以直接使用本地记事本（NotePad）。

l   使用PHP开发工具或者记事本创建PHP网页面并写PHP代码。

l   在PHP代码中调用依赖模块连接Cache、MySQL、MangoDB等服务（Cache、MySQL、MangoDB等可以是用户部署在公有云虚拟机或本地计算机中的服务，也可以是公有云直接提供的Redis Cache、MySQL PaaS等服务）.

    - PHP连接公有云中的MySQL和连接本地的MySQL完全一致，只需更改对应的host IP和MySQL Login信息就行。

    - PHP连接公有云中的Redis Cache可参考： https://msdn.microsoft.com/en-us/library/azure/Dn690470.aspx

    - Azure Redis Cache的客户端依赖包和使用方法：https://github.com/nrk/predis

l   (可选)根据实际需要，在PHP代码中连接Azure Storage。参考：https://azure.microsoft.com/en-us/documentation/articles/storage-php-how-to-use-blobs/

 

示例项目：

创建一个文件夹PHPTest1，按照https://azure.microsoft.com/en-us/documentation/articles/storage-php-how-to-use-blobs/提示，安装Azure Storage SDK（PHP），安装界面如下：

 

安装完成后，添加两个PHP页面：test.php和mycomment.php。完整示例代码如附件（附件项目中已包含Azure Storage SDK），mycomment.php为一个简单的留言板页面，使用到了MySQL（存储留言内容）和Azure Storage（存储最后一次MySQL操作命令）。

 

mycomment.php中用到的MySQL操作和Azure Storage操作代码如下：

    <?php

    require_once 'vendor\autoload.php';  //加载安装到本地的Azure Storage客户端依赖包（for PHP）

    use WindowsAzure\Common\ServicesBuilder;

    use WindowsAzure\Common\ServiceException;

 

    $val = !empty($_POST["comment"]) ? $_POST["comment"] : null;

    if($val)

    {

        $remarker= "Anonymous";

        $retime = date('Y-m-d H:i:s');

        $link = mysql_connect('42.159.191.201', 'mysqllogin', 'password');

        if (!$link) {

            die('Could not connect: ' . mysql_error());

        }

        mysql_select_db('test', $link) or die('Could not select database.');

 

        try {

            $sqlCmd = "insert into commenttable (username, words,timestamp, ipaddress) values ('$remarker','$val','$retime','$_SERVER[REMOTE_ADDR]')";

            $res = mysql_query($sqlCmd, $link);

            saveAzureBlob($sqlCmd);

        }

        catch (Exception $e) {

            echo 'Caught exception: ',  $e->getMessage(), "\n";

        }

        mysql_close($link);

    }

 

    function saveAzureBlob($lastOperation)

    {

        $connectionString = 'BlobEndpoint=https://jianwstorage.blob.core.chinacloudapi.cn/;QueueEndpoint=https://jianwstorage.queue.core.chinacloudapi.cn/;TableEndpoint=https://jianwstorage.table.core.chinacloudapi.cn/;AccountName=jianwstorage;AccountKey=I×7+×××××××××××××××××××××××××××××J×××g==';

 

        // Create blob REST proxy.

       $blobRestProxy = ServicesBuilder::getInstance()->createBlobService($connectionString);

        $blob_name = "testblob";

        $content = $lastOperation;

        try {

            //Upload blob

            $blobRestProxy->createBlockBlob("mylog", $blob_name, $content);

        } catch (ServiceException $e) {

            // Handle exception based on error codes and messages.

            $code = $e->getCode();

           $error_message = $e->getMessage();

            echo $code . ":" .$error_message . "<br />";

        }

    }

    ?>

 

若需使用Azure Redis Cache,如上介绍，下载Predis依赖包到本地PHP项目中，通过如下示例代码可以连接Azure Redis Cache：

    <?php

    require "predis/autoload.php";  //加载下载到本地的Azure Redis Cache客户端依赖包（for PHP）
    predis\autoloader::register();
    doAzureRedisCache(false);

 

    //示例函数记录请求的客户端IP到Redis Cache缓存中，如传递参数true即读取缓存的客户端IP

    function doAzureRedisCache($isRead)
    {
        $myip = $_SERVER['REMOTE_ADDR'];
        $r = new Predis\Client('tcp://jianwRedistest.redis.cache.chinacloudapi.cn:6379');
        $r->auth("DfqB9cBTKW5BPknMMzmWly+KUDB6eUHj9sJe1eCG5Ds=");
        if($isRead)
        {
            return $r->get("myip");
        }
        else {
            $r->set("myip", $myip);
        }
        return true;
    }

    ?>

 

2. 部署PHP应用到Azure虚拟机

将PHP应用部署到Azure虚拟机和本地部署PHP应用完全一致。可以参照

https://netbeans.org/kb/docs/php/configure-php-environment-windows_zh_CN.html

https://www.xl7788.com/zt/web/platform/php_linux.html

创建Azure虚拟机可以参考：

https://azure.microsoft.com/en-us/documentation/articles/virtual-machines-windows-choices-create-vm/

 
3. 部署PHP应用到Azure网站应用

将本地开发好的PHP网站部署到Azure网站应用非常便捷，如使用FileZilla工具，将本地PHP网站的全部内容上传到目标Azure网站应用对应的wwwroot目录下即可。如图：

 

部署后的效果如下：

 

更多详细的步骤参见：https://blogs.technet.com/b/sams_blog/archive/2014/11/14/azure-websites-deploy-php-website-using-ftp.aspx

如需定制化默认的PHP环境，可以参考：

https://blogs.msdn.com/b/cciccat/archive/2015/04/08/azure-websites-php-extension.aspx

https://azure.microsoft.com/en-us/documentation/articles/web-sites-php-configure/

 

4. 部署PHP应用到Azure云服务

对于默认PHP环境够用（不需要其他拓展或定制化的配置）的PHP应用，使用网站应用比较便捷。但如果PHP应用需要使用用户定制的PHP环境和拓展，使用Azure云服务会更加灵活和高效。下面的步骤是如果将本地开发好的PHP应用及本地PHP环境（包含拓展）打包发布到Azure云服务。

l   使用Azure PowerShell命令New-AzureServiceProject创建一个新的工程，并调用命令Add-AzurePHPWebRole创建一个PHP网站角色，调用Add-AzurePHPWorkerRole创建一个PHP后台处理角色（可选，网站应用一般不需要单独的后台处理模块），这些命令会生成一个项目模板，其中包括即将创建的云服务需要使用的虚拟机大小、虚拟机数量等等，用户可以手动更改或不做任何处理。

 

 

创建远程连接用户名/密码用于将来登录云服务虚拟机（可选，发布后也可启用或更改）。

 

l   将本地开发好的PHP应用完整的复制到刚刚创建的PHP网站角色对应的文件夹中。

 

l   将本地开发机中的PHP安装环境（即本地PHP安装目录，包含php.exe, ext目录, lib目录等）完整复制到PHP网站角色对应目录下的Bin文件夹中。

 

该PHP运行环境应该包含有用户自定义的所有拓展和配置。复制完成后，找到复制过来的php.ini文件，打开后更改extension_dir="C:\xampp\php\ext"为extension_dir="ext"。（C:\xampp\php\ext为笔者本地开发机中PHP环境的拓展所在目录，改为相对地址ext，以便云服务使用）

 

 

l   找到PHP网站角色目录下bin文件夹中的文件setup_web.cmd，使用记事本打开，用以下命令覆盖原有内容，并保存。这些脚本命令的作用是在云服务虚拟机中加载前面复制过来的PHP运行环境。

@ECHO ON
cd "%~dp0"

if"%EMULATED%"=="true"exit/b 0

msiexec /i sqlncli.msi /qn IACCEPTSQLNCLILICENSETERMS=YES

SET PHP_FULL_PATH=%~dp0php\php-cgi.exe
SET NEW_PATH=%PATH%;%RoleRoot%\base\x86

%WINDIR%\system32\inetsrv\appcmd.exe set config -section:system.webServer/fastCgi /+"[fullPath='%PHP_FULL_PATH%',maxInstances='12',idleTimeout='60000',activityTimeout='3600',requestTimeout='60000',instanceMaxRequests='10000',protocol='NamedPipe',flushNamedPipe='False']"/commit:apphost
%WINDIR%\system32\inetsrv\appcmd.exe set config -section:system.webServer/fastCgi /+"[fullPath='%PHP_FULL_PATH%'].environmentVariables.[name='PATH',value='%NEW_PATH%']"/commit:apphost
%WINDIR%\system32\inetsrv\appcmd.exe set config -section:system.webServer/fastCgi /+"[fullPath='%PHP_FULL_PATH%'].environmentVariables.[name='PHP_FCGI_MAX_REQUESTS',value='10000']"/commit:apphost
%WINDIR%\system32\inetsrv\appcmd.exe set config -section:system.webServer/handlers /+"[name='PHP',path='*.php',verb='GET,HEAD,POST',modules='FastCgiModule',scriptProcessor='%PHP_FULL_PATH%',resourceType='Either',requireAccess='Script']"/commit:apphost
%WINDIR%\system32\inetsrv\appcmd.exe set config -section:system.webServer/fastCgi /"[fullPath='%PHP_FULL_PATH%'].queueLength:50000"

 

l   通过PowerShell命令Publish-AzureServiceProject -ServiceName [目标云服务名称] -StorageAccountName [使用的Azure Storage名称] -Slot [测试环境或生产环境]即可将PHP应用发布到云服务。（其中使用的云服务和存储账号需要提前准备）

 

发布完成后即可查看运行在云服务中的PHP网站应用。

 

三、 小结

Windows Azure对于PHP应用提供了与本地完全一致的运行环境，用户可以根据实际场景，选择合适的Azure服务，快速部署自定义的PHP应用。

Azure虚拟机：手动定制PHP环境和应用实践。

Azure网站应用：快速部署PHP应用到公有云平台。

Azure云服务：使用自定义的PHP环境托管运行PHP应用。

 

参考内容：

https://www.apachefriends.org/download.html

https://azure.microsoft.com/en-us/documentation/articles/storage-php-how-to-use-blobs/

https://azure.microsoft.com/en-us/documentation/articles/cloud-services-php-create-web-role/

https://netbeans.org/kb/docs/php/configure-php-environment-windows_zh_CN.html#XAMPP

PHPTest1-Share.zip

]]> Ref: 快速创建、启动Azure虚拟机(群) https://docs.microsoft.com/archive/blogs/jianwu/ref-azure Wed, 03 Jun 2015 02:13:00 GMT https://blogs.msdn.microsoft.com/jianwu/2015/06/03/ref-azure/ 在使用Azure Virtual Machine过程中，用户常常会遇到以下需求：

l  在同一个托管服务（Cloud Service）下，快速创建多个虚拟机。

l  快速启动同一个托管服务下多个已有的虚拟机。

由于这些虚拟机从属于同一个托管服务，共用托管服务所对应的网络配置和负载映射等，因此在同时创建或者同时启动多个虚拟机的操作中，用户容易遇到以下问题：

 

产生这个问题的原因是：在第一台虚拟机启动初期，Azure后端尚在分配共用资源（如局域网IP）给第一台虚拟机，需要对托管服务独占式访问操作，第二台虚拟机需要在第一台虚拟机成功获取共有资源之后才能开始启动过程。

 

对于快速创建或启动同一托管服务下的多个虚拟机，本博文介绍一些常见的实施方法和优化步骤。

查看全文： https://blogs.msdn.com/b/cciccat/archive/2015/06/02/quick-start-multi-vm-in-single-cloud-service.aspx

 

]]> Windows Azure -- 静态资源访问性能优化 https://docs.microsoft.com/archive/blogs/jianwu/windows-azure-2 Thu, 19 Mar 2015 01:54:00 GMT https://blogs.msdn.microsoft.com/jianwu/2015/03/19/windows-azure-2/ 在开发网站等可公共访问的服务时，客户端的体验至关重要，Azure网站（PaaS Web Role或者Azure Websites）同样如此。通常在Azure开发过程中，对于网站服务所有的静态资源（如图片、css、js等），推荐的做法是：

 

除此之外，对于部分存储中网站服务器中的静态资源，可以通过设置网站配置web.config来指定客户端浏览器缓存该静态内容。

<staticContent>

        <clientCache setEtag="true" cacheControlMode="UseMaxAge" cacheControlMaxAge="01:00:00" />

</staticContent>

 

尽管如此，在上述实现之后，仍然可能发现一些客户端在访问目标网站时存在一些性能问题，如下是通过Chrome浏览器抓取的网络包：

 

可以发现：访问同一个网页时，本地可以缓存的内容都是来自于网站的，而来自于CDN的内容都没有本地缓存。

 

进一步抓取Fiddler网络包，可以看到，访问来自网站的图片时，返回信息里面包括了目标内容在本地缓存的设定，这和网站配置中的设定是一致的。

#

Result

Protocol

Host

URL

Body

Caching

Content-Type

Process

Comments

Custom

8

304

HTTP

×××.cloudapp.net

/image/A1_logo-03.jpg

0

max-age=3600

  

chrome:3660

    

 

HTTP/1.1 304 Not Modified

Cache-Control: max-age=3600

Accept-Ranges: bytes

ETag: "03ca4df851d01:0"

Server: Microsoft-IIS/8.5

X-Powered-By: ASP.NET

Date: Thu, 12 Mar 2015 02:25:58 GMT

 

而访问来自CDN的内容时，返回信息里面指定本地缓存时间为0.

#

Result

Protocol

Host

URL

Body

Caching

Content-Type

Process

Comments

Custom

27

304

HTTP

az725896.vo.msecnd.net

/koudaistorage-js-thirdsource/alertify.js

0

public, max-age=0

  

chrome:3660

    

 

HTTP/1.1 304 Not Modified

Accept-Ranges: bytes

Cache-Control: public, max-age=0

Date: Thu, 12 Mar 2015 02:25:57 GMT

Etag: 0x8D21E55F68CDAC3

Last-Modified: Tue, 24 Feb 2015 14:33:17 GMT

Server: Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0

X-Cache: HIT

x-ms-blob-type: BlockBlob

x-ms-lease-status: unlocked

x-ms-request-id: c72b7ac6-0001-0043-424f-de188d000000

x-ms-version: 2009-09-19

由于网站×××.cloudapp.net中web.config的clientcache配置对于存储在Azure Storage中的内容不生效，即导致了客户端的表现。

 

解决这个问题，需要对你存储在目标Azure Storage中的目标文件都设置cache-control配置，如storage explorer中即可以设置：

 

设置cache-control后，本地访问目标文件可以看到返回的信息中含有本地缓存配置：

#

Result

Protocol

Host

URL

Body

Caching

Content-Type

Process

Comments

Custom

60

200

HTTP

perfcounter.blob.core.windows.net

/test/restart.txt

13,915

max-age=3600

  

iexplore:1836

    

 

通常开发项目中，考虑到目标blob storage中存储的用于网站的js、css文件可能比较多，如果使用工具如storage explorer去逐一设置会比较低效。因此一个比较快速的办法是采用代码的方式去完成，步骤如下：

1.       使用代码去遍历目标storage中的文件

            string ACCOUNTNAME = "storageaccountname";

            string ACCOUNTKEY = "AURnf***********==";

            // Retrieve storage account from connection string.

            CloudStorageAccount storageAccount = CloudStorageAccount.Parse("DefaultEndpointsProtocol=http;AccountName=" + ACCOUNTNAME + ";AccountKey=" + ACCOUNTKEY);

 

            // Create the blob client.

            CloudBlobClient blobClient = storageAccount.CreateCloudBlobClient();

            // Retrieve a reference to a container.

            CloudBlobContainer container = blobClient.GetContainerReference("blobcontainer");

 

            // Create the container if it doesn't already exist.

            container.CreateIfNotExists();

            //set blobs in the container to be public

            container.SetPermissions(

                new BlobContainerPermissions

                {

                    PublicAccess = BlobContainerPublicAccessType.Blob

                });

 

            // Loop over items within the container

            foreach (IListBlobItem item in container.ListBlobs(null, true))

            {

                if (item.GetType() == typeof(CloudBlockBlob))

                {

                    CloudBlockBlob blockBlob = (CloudBlockBlob)item;

                    ......

2.       对每一个文件单独设置cache-control属性

            // Loop over items within the container and save cache control property

            foreach (IListBlobItem item in container.ListBlobs(null, true))

            {

                if (item.GetType() == typeof(CloudBlockBlob))

                {

                    CloudBlockBlob blockBlob = (CloudBlockBlob)item;

 

                //set cache-control for the blob

                blockBlob.Properties.CacheControl = "max-age=3600, must-revalidate";

                blockBlob.SetProperties();

                }

            }

 

开发参考：

https://msdn.microsoft.com/en-us/library/windowsazure/hh225342.aspx

https://alexandrebrisebois.wordpress.com/2013/08/11/save-money-by-setting-cache-control-on-windows-azure-blobs/

 

更多关于Azure blob storage的缓存设置：

For blob storage, http cache is controlled by one property named cache-control of blobs, here is definition and details of the cache-control property:

https://msdn.microsoft.com/en-us/library/windowsazure/microsoft.windowsazure.storage.blob.blobproperties.cachecontrol.aspx

Gets or sets the cache-control value stored for the blob.

Regarding Cache-Control Headers, here is more about how to value it:

max-age=[seconds] — specifies the maximum amount of time that a representation will be considered fresh. Similar to Expires, this directive is relative to the time of the request, rather than absolute. [seconds] is the number of seconds from the time of the request you wish the representation to be fresh for.

s-maxage=[seconds] — similar to max-age, except that it only applies to shared (e.g., proxy) caches.

public — marks authenticated responses as cacheable; normally, if HTTP authentication is required, responses are automatically private.

private — allows caches that are specific to one user (e.g., in a browser) to store the response; shared caches (e.g., in a proxy) may not.

no-cache — forces caches to submit the request to the origin server for validation before releasing a cached copy, every time. This is useful to assure that authentication is respected (in combination with public), or to maintain rigid freshness, without sacrificing all of the benefits of caching.

no-store — instructs caches not to keep a copy of the representation under any conditions.

must-revalidate — tells caches that they must obey any freshness information you give them about a representation. HTTP allows caches to serve stale representations under special conditions; by specifying this header, you’re telling the cache that you want it to strictly follow your rules.

proxy-revalidate — similar to must-revalidate, except that it only applies to proxy caches.

 

另外，对于Windows Azure Storage中存储的网站静态资源，还有一个很好的功能可以打开：

BlobProperties.ContentEncoding

该功能的作用是指定该文件的编码方式，如较大的静态文件，可以在本地用代码的方式批量压缩成gzip格式，然后使用代码逐一上传到Azure Storage中，如示例代码：https://msdn.microsoft.com/en-us/library/azure/hh225342.aspx。

上传过程中，需要设定目标资源的ContentEncoding属性，如下代码：

// Set the CacheControl property.

blob.Properties.ContentEncoding = "gzip";

blob.SetProperties();

这样设定以后，浏览器访问该gzip格式的静态资源（如图片、css、js文件）时，会自动解压缩，然后显示到网页中。该过程中，由于目标文件被压缩了，网页加载速度自然就加快了。

 

因此，对于Azure网站中静态资源的访问性能优化，可以使用以下思路：

  • 将静态文件存储在Azure Storage中。
  • 开启CDN服务，指向目标Azure Storage。
  • 在网页中使用基于CDN的资源地址。
  • 对Storage中存储的网站静态资源，开启client cache属性。
  • 对上传到Storage中的网站静态资源，先压缩成gzip，然后上传，并指定其contentencoding属性为gzip。
  • 对于较大的图片，可以考虑生成指纹图片，网站加载时仅加载较小分辨率的指纹图片，进一步访问该图片时才返回原始图片。

 

]]> 2 known issues in Windows Azure Diagnostics https://docs.microsoft.com/archive/blogs/jianwu/2-known-issues-in-windows-azure-diagnostics Thu, 05 Feb 2015 19:56:00 GMT https://blogs.msdn.microsoft.com/jianwu/2015/02/05/2-known-issues-in-windows-azure-diagnostics/ As you know, Windows Azure Diagnostics enables you to collect diagnostic data from an application running in Windows Azure. You can use diagnostic data for debugging and troubleshooting, measuring performance, monitoring resource usage, traffic analysis and capacity planning, and auditing. After the diagnostic data is collected it can be transferred to a Windows Azure storage account for persistence. Transfers can either be scheduled or on-demand.

More reference about Windows Azure Diagnostics are here:

Overview: https://msdn.microsoft.com/en-us/library/azure/gg433048.aspx

Practice: https://blogs.msdn.com/b/jianwu/archive/2014/08/15/azure-paas-4.aspx

In this blog, 2 known issues in WAD are summarized with quick workarounds, hopes it will help get better WAD and better cloud service. Special thanks to Peter Thompson reporting this issue and assist us to fix.

Issue 1:

After upgrading Azure SDK to the latest version 2.5, the IIS logs cannot be transferred to Azure Storage account with the following settings in WAD GUI configuration page.

  • Cause:

When WAD is generating the WAD configure it gets the IIS log directory from the .NET service manager. It’s very likely that WAD generates its configure just before a new web site is created. In this scenario the configure becomes stale and uses default web site’s log settings. However the default web site’s log settings is not the actual place where the IIS logs will be put.

Defualt web site’s log settings:

<FileWatchItem dontUseNPartitionSpread="true" compressionType="none" eventName="DirectoriesTable" directoryQuotaInMB="6500" lastChangeOffsetInSeconds="300" removeEmptyDirectories="false" container="iiscontainer">

        <Directory>"D:\inetpub\logs\LogFiles"</Directory>

      </FileWatchItem>

      <FileWatchItem dontUseNPartitionSpread="true" compressionType="none" eventName="DirectoriesTable" directoryQuotaInMB="6500" lastChangeOffsetInSeconds="300" removeEmptyDirectories="false" container="failedrequestlogs-container">

        <Directory>"D:\inetpub\logs\FailedReqLogFiles"</Directory>

Actual path the web site’s logs are saved (one example:):

                C:\Resources\Directory\Test.WebRole1.DiagnosticStore\LogFiles\W3SVC1

 

  • Hotfix:

            This issue will be addressed in the next release in SDK3.0 in March or April.

 

  • Workaround:

1.       Restart WAD agent

All we need to resolve this issue is to restart the WAD agent which will redirect the monitoring folder to the correct one. You can perform either of the following to restart WAD agent.

  • ·         REBOOT the role instances to force the Diagnosticsplugin.exe restarted
  • ·         Send a WAD update via Server Explore in Visual Studio to force WAD agent being restarted

2.       Use WAD custom directory logging as a workaround

Azure WAD automatically maps wad-iis-logfiles and wad-failedrequestlogs containers to special folders which exist in web roles. In addition, WAD provides custom directory logging shown as below. Since we are intending to transfer IIS logs to azure storage account, I specify the absolute log path to “\Resources\Directory” where the IIS logs reside.  The exact directory path for IIS logs is “C:\Resources\Directory\8723df3598a241688eb39a1cee727639.WebRole1.DiagnosticStore\LogFiles\Web\W3SVC1273337584”, and I take its parent folder for simplicity as I don’t know the  deployment ID before the cloud service being deployed.

Steps to configure custom logging:

1.       Configure via WAD GUI as shown above    2.       Add the following lines to .wadcfgx file directly

<Directories scheduledTransferPeriod="PT1M">

          <IISLogs containerName="wad-iis-logfiles" />

          <FailedRequestLogs containerName="wad-failedrequestlogs" />

          <DataSources>

            <DirectoryConfiguration containerName="wad-custom1">

              <Absolute path="\Resources\Directory" expandEnvironment="false" />

            </DirectoryConfiguration>

          </DataSources>

</Directories>

 

Issue 2:

In elder version of Azure SDK like 2.2, newly deployed cloud service worked fine with WAD at beginning, but after some time, customers might observe that certain WAD logs like performance counters and IIS logs were no longer transferred to target storage, though rebooting or reimaging the role instances could resolve problem quickly, sometime later after the rebooting/reimaging, the problem would reoccur again.

  • Root cause:

In Azure elder version of SDKs, The total size of all WAD data should be at least 500MB less than the size limit of diagnostics folder C:\Resources\Directory\[deploymentid].[rolename].DiagnosticStore,

By default, the size of the diagnostics folder is 4096MB, the max size of all WAD data can be 4096MB, that means, if in your WAD settings, the sum of *QuotaInMB exceeded 4GB, then there should be less than 500MB between the gap of WAD data and folder size of C:\Resources\Directory\[deploymentid].[rolename].DiagnosticStore, the size gap is used for temporary purpose in WAD, hence if there was no enough space for this temporary use, WAD will stop transferring data to target storage after some time.

 

  • Solution & workaround:

1.      Set local resource configuration in servicedefinition.csdef file as below: (this is to set the size limit of C:\Resources\Directory\[deploymentid].[rolename].DiagnosticStore to 6GB)

….

    </Imports>

    <LocalResources>

      <LocalStorage name="DiagnosticStore" sizeInMB="6144" cleanOnRoleRecycle="false"/>

    </LocalResources>

  </WebRole>

</ServiceDefinition>

 

2.       Adjust the buffer size for each WAD item to a proper space, like 512MB, and make sure the remaining space is enough for temporary use, like no less than 1 GB. See a sample screenshot:


 3.     Debug your project first in local emulator, while there is no error, deploy the project to empty staging slot of Azure cloud, and if everything is working fine there, then swap it to production.

Temporary workaround:  if you are not able to do the setting changes as above and redeploy right now due to product schedule, you can RDP into the current deployment VMs and then delete the existing log data in “C:\Resources\Directory\[deploymentid].[rolename].DiagnosticStore”, that will make diagnostic work fine for you for some time again. But that is just a temporary workaround like VM reboot/reimage, because if size of new log comes to the limit of those 2 settings, the problem may happen again.

 

]]> One reference architecture in PaaS cloud solution. https://docs.microsoft.com/archive/blogs/jianwu/one-reference-architecture-in-paas-cloud-solution Sat, 31 Jan 2015 03:59:00 GMT https://blogs.msdn.microsoft.com/jianwu/2015/01/31/one-reference-architecture-in-paas-cloud-solution/ It is from projects to demonstrate cloud energy and relieve big headaches in traditional solutions, the architecture can be realized smartly with public cloud while keep connected with local network, and the reference may help in Windows Azure, Amazon Web Service and more cloud platforms.

Highlights to demonstrate:

  •  Apply Microsoft cloud to grant traditional solution capability to process sudden dramatic increase requests or huge incoming traffic in a very smart way
  •  Able to save more than 50% system cost
  •  On premise solution can be integrated with or migrated to cloud services, leading to a more powerful and decoupled solution
  •  Numerous great features in Microsoft Azure accelerates development cycle and make online solutions easy to go with performance guarantee
  •  Popular 3rd party technologies can be well applied as they were

Key features involved:

  • Classic programing pattern: multi tiers design & integration, security enhancement, performance optimization.
  • Auto scaling on Azure: flexible cloud technology to respond changing business traffic.
  • Virtual network: extend on premise network & machine to cloud, keep traditional software solution still working for Azure.
  • WAD: powerful monitoring and diagnostics from Windows Azure.
  • ACS: let customer benefit more from existing identity providers and Azure cloud, then focus more on business logic.
  • Service Bus: solutions for those who doesn’t have public endpoint but needs real time communication.
  • Cache Service: Improve performance both for Azure solution and on premise application.
  • SQL Azure: Powerful Relation Database Engine on cloud, provide reliable and secure structured storage for cloud apps and local apps. 
  • Others: easy management and high efficiency of Windows Azure, good support on open tech and third party plugins.

Brief Workflow:

  1. Load test generates web requests to web role.
  2. Web role handles incoming traffic initially and adds the corresponding message in target storage queue.
  3. Worker role picks up messages from storage queue, and process them one by one.
  4. When worker role process a message, it needs some important information from on premise server.
  5. During the whole workflow, end user can trace the status of his request from web role


Reference: https://www.doingcloud.com/index.php/2016/01/29/one-reference-architecture-in-azure-paas-solution/

 

]]> Expose SSL service to multi domains from the same cloud service https://docs.microsoft.com/archive/blogs/jianwu/expose-ssl-service-to-multi-domains-from-the-same-cloud-service Wed, 17 Dec 2014 01:19:00 GMT https://blogs.msdn.microsoft.com/jianwu/2014/12/17/expose-ssl-service-to-multi-domains-from-the-same-cloud-service/ When designing and deploying web services to Azure with Azure cloud service, it is very common to apply SSL to secure web access and protect the data transfer, regarding how to enable SSL in cloud service and make custom domain work instead of default cloudapp.net, here are some good tutorials:

       1. https://azure.microsoft.com/en-us/documentation/articles/cloud-services-configure-ssl-certificate/

       2. https://blogs.msdn.com/b/avkashchauhan/archive/2011/03/28/complete-solution-adding-ssl-certificate-with-windows-azure-application.aspx

in this implementation, if your certificate is not in pfx format but in others like p7b, you can follow this article https://stackoverflow.com/questions/6708223/convert-p7b-to-pfx-for-azure to get the right pfx formated certificate to move with https on Azure.

if you certificate is pfx, but after you upload it to the portal, you notice 2 or more certificates added there, you can check the solution here to fix: https://blog.michaelchi.net/2014/12/cloud-servicechained-certificatessl.html

in this post, i am summarying a resolved issue in my job, which was to enable multi SSL in the same cloud service. (a special thanks to John Ellis, whom I worked on this topic. )

for exampel, the cloud service is https://test.cloudapp.net and is listening at port 80 and 443, now the goal is to set the cloud service to be accessible at https://www.test1.com and https://www.test2.com

 

>Analysis:

In lower version of IIS, such as IIS 6 and IIS 7, it is not supported to enable multi SSL bindings for a website.

Until IIS 7.5, the major limitation of IIS is that IIS will allow you to bind only one site for one IP: Port combination using an SSL certificate.  If you try to bind a second site on the IP address to the same certificate, IIS will give you an error when starting the site up stating that there is a conflict.

To know why it doesn’t work in HTTPS lets understand the SSL handshake briefly.

1. Client - > (SSL Handshake) - > “ Hello, I support XYZ algorithm for encryption”

2. Server -> (SSL Handshake) -> “Hi there, Okay so here is my public certificate. Let’s use algorithm X”

3. Client -> (SSL Handshake)-> “Great we can use that”

4. Client -> (In Encrypted format)-> “HTTP Request”

5. Server -> (In Encrypted format)-> “HTTP Response”

<And now cycle continues>

Now let’s say hypothetically, you have set two sites on same IP-address and port and different host headers and you set two different certificates on both of them.

Look at the steps in SSL Handshake, Client sends the HTTP Request only in Step 4. That means Server doesn’t know what host header HTTP request is referring to until step 4. So at Step 2, Server has only IP-Address and Port information with it, so how can server figure out which certificate it needs to send to the Client as you have bind two certificates to same IP-Address and port.

When a request comes to HTTP.SYS layer, the HTTP.SYS reads the site configuration, including the certificate used to encrypt/decrypt the data. The host name is encrypted in SSL Blob that the client sends. However, IIS needs to know the host name in order to get the right certificate. Without the host name IIS cannot get to the correct site. As IIS is not able to get to the correct site so it cannot get the right certificate to decrypt the SSL blob to get the host name. This is the classic Chicken and Egg problem. We are turning into circles with no way out.

This is the precise reason; HTTP server can only allow one site per IP-Address: Port combination for HTTPS browsing. If you need to bind another site over HTTPS then you need to get either a different IP-Address or bind the site to a different port.

Reference: https://blogs.msdn.com/b/varunm/archive/2013/06/18/bind-multiple-sites-on-same-ip-address-and-port-in-ssl.aspx

 

However, IIS8 has a new functionality called Server Name Indication (SNI) to overcome this problem.

Problem: As more e-commerce sites come on line and more businesses are storing and sharing sensitive documents online, the ability to host and scale secure sites are increasingly more important. Prior to Windows Server 2012, there were a couple of challenges when it comes to hosting secure sites:

  • SSL Scalability: In a multi-tenanted environment, such as a shared hosting, there is a limitation as to how many secure sites can be hosted on Windows Server, resulting in a low site-density.
  • IPv4 scarcity: Because the network end-point can only be identified with IP:Port binding, where tenants request to use the standard SSL port, 443, hosting a secure site often means offering a dedicated IP address per tenant.

Solution: On Windows Server 2012, IIS supports Server Name Indication (SNI), which is a TLS extension to include a virtual domain as a part of SSL negotiation. What this effectively means is that the virtual domain name, or a hostname, can now be used to identify the network end point. In addition, a highly scalable WebHosting store has been created to complement SNI. The result is that the secure site density is much higher on Windows Server 2012 and it is achieved with just one IP address.

 

>How to expose multi SSL services from the same web role

1. Adjust your service configuration file to apply higher version of OS, which has IIS 8 in it. see below, apply osFamily="4" in your service configuration file.

<?xml version="1.0" encoding="utf-8"?>

<ServiceConfiguration serviceName="WindowsAzure1" xmlns="https://schemas.microsoft.com/ServiceHosting/2008/10/ServiceConfiguration" osFamily="4" osVersion="*" schemaVersion="2013-03.2.0">

  <Role name="WebRole1">

    <Instances count="1" />

<ConfigurationSettings>

…

More details on Azure OS family and its compatibility with SDK: https://msdn.microsoft.com/en-us/library/azure/ee924680.aspx

2. Add certificate into your project and add endpoints as below. (for the certification specified in https endpoint, that can be any one from the 2 certificates as you want).

3. Adjust service definition file with high lightened below.

<?xml version="1.0" encoding="utf-8"?>

<ServiceDefinition name="WindowsAzure1" xmlns="https://schemas.microsoft.com/ServiceHosting/2008/10/ServiceDefinition" schemaVersion="2013-03.2.0">

  <WebRole name="WebRole1" vmsize="Small">

    <Runtime executionContext="elevated" />

    <Sites>

      <Site name="Web">

        <Bindings>

          <Binding name="Endpoint1" endpointName="Endpoint1" />

          <!--<Binding name="Endpoint2" endpointName="Endpoint2"/>-->

        </Bindings>

      </Site>

    </Sites>

    <Endpoints>

      <InputEndpoint name="Endpoint1" protocol="http" port="80" />

      <InputEndpoint name="Endpoint2" protocol="https" port="443" certificate="Certificate2" />

    </Endpoints>

    <Imports>

      <Import moduleName="Diagnostics" />

      <Import moduleName="RemoteAccess" />

      <Import moduleName="RemoteForwarder" />

    </Imports>

    <LocalResources>

      <LocalStorage name="DiagnosticStore" sizeInMB="6144" cleanOnRoleRecycle="false"/>

    </LocalResources> 

    <Certificates>

      <Certificate name="Certificate1" storeLocation="LocalMachine" storeName="My" />

      <Certificate name="Certificate2" storeLocation="LocalMachine" storeName="My" />

    </Certificates>

  </WebRole>

</ServiceDefinition>

……

4. Add your target pfx SSL certificates into the web role project and set their property as “copy always”

5. Right click your web project like WebRole1 in my test and install package from Nuget as below

6. Add codes below in your WebRole.cs, and replace the key information with yours.

using System;

using System.Collections.Generic;

using System.Linq;

using Microsoft.WindowsAzure;

using Microsoft.WindowsAzure.Diagnostics;

using Microsoft.WindowsAzure.ServiceRuntime;

using System.Security.Cryptography.X509Certificates;

using Microsoft.Web.Administration;

using System.Net;

using System.Net.Sockets;

using System.Diagnostics;

 

namespace WebRole1

{

    public class WebRole : RoleEntryPoint

    {

        public override bool OnStart()

        {

            // For information on handling configuration changes

            // see the MSDN topic at https://go.microsoft.com/fwlink/?LinkId=166357

            System.Threading.Thread.Sleep(40000);

            try

            {

                var newCert = new X509Certificate2("test1.com-1.pfx", "password", X509KeyStorageFlags.MachineKeySet | X509KeyStorageFlags.PersistKeySet | X509KeyStorageFlags.Exportable);

                var readWriteMyStore = new X509Store(StoreName.My, StoreLocation.LocalMachine);

                readWriteMyStore.Open(OpenFlags.ReadWrite);

                readWriteMyStore.Add(newCert);

                readWriteMyStore.Close();

                System.IO.File.AppendAllText("log.txt", "\r\n cert 1 is installed. @" + DateTime.Now.ToString());

 

                var serverManager = new ServerManager();

                var site = serverManager.Sites[0];

                var binding = site.Bindings.Add(":443:www.test1.com", newCert.GetCertHash(), "My");

                binding.SetAttributeValue("sslFlags", 1); //enables the SNI

                serverManager.CommitChanges();

                System.IO.File.AppendAllText("log.txt", "\r\n binding 1 is added. @" + DateTime.Now.ToString());

 

                var newCert2 = new X509Certificate2("test2.com-1.pfx", "password", X509KeyStorageFlags.MachineKeySet | X509KeyStorageFlags.PersistKeySet | X509KeyStorageFlags.Exportable);

                var readWriteMyStore2 = new X509Store(StoreName.My, StoreLocation.LocalMachine);

                readWriteMyStore2.Open(OpenFlags.ReadWrite);

                readWriteMyStore2.Add(newCert2);

                readWriteMyStore2.Close();

                System.IO.File.AppendAllText("log.txt", "\r\n cert 2 is installed. @" + DateTime.Now.ToString());

 

                var serverManager2 = new ServerManager();

                var site2 = serverManager2.Sites[0];

                var binding2 = site2.Bindings.Add(":443:www.test2.com", newCert2.GetCertHash(), "My");

                binding2.SetAttributeValue("sslFlags", 1); //enables the SNI

                serverManager2.CommitChanges();

                System.IO.File.AppendAllText("log.txt", "\r\n binding 2 is added. @" + DateTime.Now.ToString());

 

                try

                {

                    RemoveDefaultIISSSL();

                }

                catch (Exception ee)

                {

                }

            }

            catch (Exception ex)

            {

                System.IO.File.AppendAllText("log.txt", "\r\n " + ex.Message + " @" + DateTime.Now.ToString());

            }

            return base.OnStart();

        }

 

        public bool RemoveDefaultIISSSL()

        {

            System.IO.File.AppendAllText("log.txt", "\r\n start to get local ip. @" + DateTime.Now.ToString());

            IPHostEntry host;

            string localIP = "127.0.0.1";

            host = Dns.GetHostEntry(Dns.GetHostName());

            foreach (IPAddress ip in host.AddressList)

            {

                if (ip.AddressFamily == AddressFamily.InterNetwork)

                {

                    localIP = ip.ToString();

                    break;

                }

            }

 

            string cmd = string.Format("netsh http delete sslcert ipport={0}:443",localIP);

            System.IO.File.AppendAllText("log.txt", "\r\n local ip is got. @" + cmd + " @"+ DateTime.Now.ToString());

            Process p = new Process();

            p.StartInfo.FileName = @"d:\Windows\System32\cmd.exe";

            p.StartInfo.UseShellExecute = false;

            p.StartInfo.RedirectStandardInput = true;

            p.StartInfo.RedirectStandardOutput = false;

            p.StartInfo.RedirectStandardError = true;

            p.StartInfo.CreateNoWindow = true;

            string Parameter = cmd;

            try

            {

                p.Start();

                p.StandardInput.WriteLine(cmd);

                p.StandardInput.WriteLine("exit");

                p.StandardInput.WriteLine("exit");

                p.WaitForExit();

                p.Close();

            }

            catch (Exception e)

            {

                System.IO.File.AppendAllText("log.txt", "\r\n " + e.Message + " @" + DateTime.Now.ToString());

                return false;

            }

            return true;

        }

    }

}

【update in step 6】:

After double checking the issue and deep dive into IIS internal, I noticed a default SSL setting for IIS that impacted multi SSL bindings, so I updated the code as above and have verified it in sample cloud service.

7. Rebuild the project in local emulator. Then deploy it to azure cloud.

8.Verification: after deploying it to cloud, by Remote access, you can confirm the website is configured as below, that worked as expected.

 

                  # localhost name resolution is handled within DNS itself.

                  #             127.0.0.1       localhost

                  #             ::1             localhost

                104.40.95.228   www.test1.com

                104.40.95.228   www.test2.com

More reference:

Appcmd command to add SSL binding: https://toastergremlin.com/?p=308

 

]]> Restrict Remote Access to Cloud VMs https://docs.microsoft.com/archive/blogs/jianwu/restrict-remote-access-to-cloud-vms Wed, 03 Dec 2014 22:22:00 GMT https://blogs.msdn.microsoft.com/jianwu/2014/12/03/restrict-remote-access-to-cloud-vms/ Recently we are working with a project intending to restrict the RDP access of Azure role (worker role/ web role) instance to some specific IP address (a special thanks to Ugo Meex). The tricky thing we should be careful is there is a port mapping for RDP access between public port (3889) and internal port (20000) for role instances. So instead of applying firewall rule restrictions to public port 3389 we need to apply it to internal port 20000. Please find more details below.

About setting up remote access in Azure, please refer to https://msdn.microsoft.com/library/azure/hh124107.aspx/

Requirement:

Restrict RDP access to some specific IP address in Azure Role instances.

Background: How Windows Azure RemoteForwarder works?

The remote desktop feature is comprised of two imports: RemoteAccess and RemoteForwarder. The RemoteForwarder is working as follows:

  • RemoteForwarder is imported on a single role and takes care of dealing with the fact that Windows Azure only provides load-balanced input endpoints.
  • The forwarder runs on every instance of the role in which it is imported and listens on port 3389 for remote desktop connections.
  • When a connection is received, it searches the first packet for a load balancing cookie which you can see if you open the Portal-supplied .rdp file in a text editor.
  • Using this cookie data it then opens a connection internally to the desired instance and forwards all traffic to that instance.

So when we set up the RDP restrict firewall rule, instead on apply the restriction on public port 3389 we need to apply it to internal port 20000. By default the endpoint mapping between public and private for RDP access is 3889 to 20000, which you can find in the service config file located in C:/Config of your cloud VM .

Here is a default firewall initialization in a cloud service VM:

 

More details about Azure Remote Access service in cloud service:

https://blogs.msdn.com/b/avkashchauhan/archive/2011/12/06/how-does-remote-desktop-works-in-windows-azure.aspx

Solution 1: apply firewall rule by startup task

1.      In your web role project(where web.config exist), Create a file with content below and name it as SetFirewallRestrictions.ps1

         Get-NetFirewallPortFilter | ? LocalPort -eq '20000' | Get-NetFirewallRule | Set-NetFirewallRule -RemoteAddress “Allowed IP Address”

         Get-NetFirewallRule –DisplayName “WaRemoteForwarderService Rule” | Set-NetFirewallRule -RemoteAddress “Allowed IP Address”

2.      Also in the same web role project, create a file with content below and name it as SetFirewallRestrictions.cmd

          Powershell set-executionpolicy remotesigned –force

         Powershell .\SetFirewallRestrictions.ps1 >> SetFirewallRestrictions.out.log 2>> SetFirewallRestrictions.err.log

3.      Set copy to output property of these 2 files above as “copy always” in Visual Studio

4.      Add content below in service definition file to start the above commands as startup

 <WebRole name="WebRole1" vmsize="Small">

   ... 

    <Startup>

      <Task commandLine="SetFirewallRestrictions.cmd" executionContext="elevated" taskType="simple" />

    </Startup>

    <Runtime executionContext="elevated" />

  </WebRole>

</ServiceDefinition>

Reference on running startup task in Azure cloud service: https://msdn.microsoft.com/en-us/library/azure/hh180155.aspx

Tips: the IP address in file SetFirewallRestrictions.ps1 should be public IP address of the target client, but not its local IP address. How to get your client public IP address: https://whatismyipaddress.com/

 

Solution 2: apply firewall rule by code part in WebRole.cs

1.       Add same file SetFirewallRestrictions.cmd and SetFirewallRestrictions.ps1 as above and set their property to “copy always”

2.       Add content below in service definition file of your web role:

  <WebRole name="WebRole1">

    <Runtime executionContext="elevated"/>

    <Sites>

3.       Add codes below in OnStart():

           // you can do some code to change the IP addresses in SetFirewallRestrictions.ps1

           string cmd = string.Format("SetFirewallRestrictions.cmd ");

            Process p = new Process();

            p.StartInfo.FileName = @"d:\Windows\System32\cmd.exe";

            p.StartInfo.UseShellExecute = false;

            p.StartInfo.RedirectStandardInput = true;

            p.StartInfo.RedirectStandardOutput = false;

            p.StartInfo.RedirectStandardError = true;

            p.StartInfo.CreateNoWindow = true;

            string Parameter = cmd;

            try

            {

                p.Start();

                p.StandardInput.WriteLine(cmd);

                p.StandardInput.WriteLine("exit");

                p.StandardInput.WriteLine("exit");

                p.WaitForExit();

                p.Close();

            }

            catch (Exception e)

            {

            }

This solution may provide more options to VM owners to grant remote access. For example, VM owners can store permitted IP addresses in a blob storage file, then before the code above, some extra code can be added to access that target blob storage file and generate a corresponding file SetFirewallRestrictions.ps1. More than that, the code above can be added into a while loop in Run() rather than OnStart(), then the while loop will run the PowerShell command for its time, then in further round, it check blob file and determine sleep or run PowerShell again, by this way, any IP change in target blog file by VM owners can be applied immediately by the code logistics. Does that fun?

 

Solution 3: applying ACL(Access Control List) in Azure deployment

Tutorial for using ACL in Azure PaaS project: https://blogs.msdn.com/b/walterm/archive/2014/04/22/windows-azure-paas-acls-are-here.aspx

Key Points: as with IaaS ACLs, PaaS ACLs allow you to do the following:

  • Selectively permit or deny incoming traffic based on remote subnet IPv4 address range to a virtual machine input endpoint
  • Blacklist IP addresses
  • Create multiple rules per virtual machine endpoint
  • Specify up to 50 ACL rules per virtual machine endpoint
  • Use rule ordering to ensure the correct set of rules are applied on a given virtual machine endpoint (lowest to highest)
  • Specify an ACL for a specific remote subnet IPv4 address

 

Note: the sample was done in Azure Cloud Service VMs, for Azure IaaS VMs, similar command should also work, feel free to have a try and make your cloud VMs only accessible to your limited IP clients.

]]> Azure Auto Scaling -- 设计强伸缩性的云方案 https://docs.microsoft.com/archive/blogs/jianwu/azure-auto-scaling Tue, 25 Nov 2014 04:51:00 GMT https://blogs.msdn.microsoft.com/jianwu/2014/11/25/azure-auto-scaling/ 云计算解决方案中，最大的一个特色就是强伸缩性。如果托管服务的工作流量骤增、骤减或者只需服役一段时间，Microsoft Azure可以提供自动化伸缩（Auto Scaling）的功能，Auto Scaling可以依据开发者定义的扩容（缩容）规则来监视运行在云端的托管服务，一旦托管服务的运行达到规则生效条件，Microsoft Azure即启动自动化扩容（缩容）动作，完成扩容（缩容），使其在保证在线上服务的高质量时，使用的云资源最少。

本博文以Azure Cloud Service为例，就常用的基于Auto Scaling的设计和实施进行小结和分析，供Azure开发（架构）者参考。主要包含以下几个部分：

1. 使用Azure管理主页（portal）进行Auto Scaling设计和实现。

2. 高级使用Auto Scaling。

3. Auto Scaling性能调优及关键参数。

 

使用Azure管理主页实现Auto Scaling

此方法非常简便，无需任何代码或编程操作，开发者登录Microsoft Azure管理门户https://manage.windowsazure.com，选择目标托管服务之后，在Scale页，开发者可以设置Auto Scaling。在默认情况下，Auto Scaling是关闭的（“SCALE BY METRIC”项是“NONE”），但开发者可以手动修改“INSTANCE COUNT”项，临时增加或者减少角色实例的数量。

若开发者选择使用CPU或者Queue Storage来实现Auto Scaling，可以将Scale By Metric置为CPU或者Queue，然后配置具体的Scaling细节，如下参考配置：

更多关于此页面中Scaling设置的参数可以参考配置旁边的？中的提示信息，也可以参考以下文章和实践手册：https://azure.microsoft.com/en-us/documentation/articles/cloud-services-how-to-scale/

 

 高级使用Auto Scaling

采用了上述方法实施Auto Scaling设计以后，开发者往往会发现这样一个问题，当外部流量很大，导致Cloud Service中的每个虚机内CPU使用率高于80%时，Scaling Up（自动化扩容）大约1小时以后才发生，这与预期的效果可能存在差异。对于此问题，需要详细了解一下Auto Scaling工作的机制和主要要点。

 

主要要点：

1. Role状态正常即指Role中的每一个虚机都正常工作

2. 默认情况下，CPU统计值来自于之前45分钟内该Role下所有虚机的CPU平均。

3. Azure平台中，虚机内部的性能指标，传输到Scaling模块有一定的时差（约15分钟）

鉴于此，可以预期：若0：00分设置基于CPU的Scaling方案，然后迅速将所有的虚机CPU使用率消耗到90%，需要持续将近45+15=60分钟才会有Scaling Up事件发生。

深入了解这个问题，还需要仔细看一下Auto Scaling工作所依赖的配置。使用上述方法在portal上实施Auto Scaling后开发者可以在management services里面找到添加的auto scaling内容。（操作类型为PutAutoscalingSetting）

AutoscaleSettings:
[
  {
    "Profiles": [
      {
        "Name": "Scale.WebApi.Cpu",
        "Capacity": {
          "Minimum": "1",
          "Maximum": "2",
          "Default": "1"
        },
        "Rules": [
          {
            "MetricTrigger": {
              "Name": "Percentage CPU",
              "Namespace": "CloudService",
              "Resource": "CloudService:test1205:Staging:WebRole1",
              "TimeGrain": "PT15M",
              "Statistic": "Average",
              "TimeWindow": "PT45M",
              "TimeAggregation": "Average",
              "Operator": "GreaterThan",
              "Threshold": 80.0
            },
            "ScaleAction": {
              "Direction": "Increase",
              "Type": "ChangeCount",
              "Value": "1",
              "Cooldown": "PT10M"
            }
          },
          {
            "MetricTrigger": {
              "Name": "Percentage CPU",
              "Namespace": "CloudService",
              "Resource": "CloudService:test1205:Staging:WebRole1",
              "TimeGrain": "PT15M",
              "Statistic": "Average",
              "TimeWindow": "PT45M",
              "TimeAggregation": "Average",
              "Operator": "LessThan",
              "Threshold": 60.0
            },
            "ScaleAction": {
              "Direction": "Decrease",
              "Type": "ChangeCount",
              "Value": "1",
              "Cooldown": "PT10M"
            }
          }
        ]
      }
    ],
    "SubscriptionId": "959bf13d-17fc-4cc2-9669-594×××××2b",
    "Source": "CloudService:test1205:Staging:WebRole1",
    "Version": "1.0",
    "Enabled": true
  }
]

Resource Name: CloudService:test1205:Staging

Resource Type: Autoscale

大部分的配置内容都来自于portal上的设置，都比较容易理解，其中有两个重要的参数不是来自于portal上的设置：TimeGrain和TimeWindow，TimeGrain表示Azure Auto Scaling计算平均值的样本时间，15分钟即表示CPU平均值按每15分钟做一次平均统计，如2：00～2：15得到一个CPU平均值，2：15～2：30得到一个CPU平均值。TimeWindow表示Azure Auto Scaling用于检验目标role的过去表现的时长。如TimeGrain=15和TimeWindow=45，即表示，过去45分钟内，若所得的CPU平均值均高于80%，即scaling up。

因此，在需要更高灵敏的自动化伸缩方案中，开发者需要对默认的Scaling规则进行优化，开发者可以通过代码的方式来配置TimeGrain和TimeWindow。注：以下代码需要在项目中使用Nuget添加Azure Management Service Library。

        static void Main(string[] args)
        {
            // Cloud Service and Role to be auto scaled.
            var cloudServiceName = "test1205";
            var isProduction = false;
            var roleName = "WebRole1";

            // Azure Subscription ID and Management Certificate Thumbprint.
            string subscriptionId = "959bf13d-17fc-4cc2-9669-5&&××××b";

            // Get the certificate from the local store.
            X509Certificate2 cert = GetCertificate();

            // Genereate a resource Id for the cloud service and role.
            var resourceId = AutoscaleResourceIdBuilder.BuildCloudServiceResourceId(cloudServiceName, roleName, isProduction);

            // Create the autoscale client.
            AutoscaleClient autoscaleClient = new AutoscaleClient(new CertificateCloudCredentials(subscriptionId, cert));

            //Create an Autoscale Profile
            AutoscaleSettingCreateOrUpdateParameters createParams = new AutoscaleSettingCreateOrUpdateParameters()
            {
                Setting = new AutoscaleSetting()
                {
                    Enabled = true,
                    Profiles = new List<AutoscaleProfile>()
                    {
                        new AutoscaleProfile()
                        {
                            Name = "Scale.WebApi.Cpu",
                            Capacity = new ScaleCapacity()
                            {
                                Default = "1",
                                Minimum = "1",
                                Maximum = "2"
                            },
                    Rules = new List<ScaleRule>()
                        }
                    }
                }
            };

            //Create a ScaleRule to Scale Up
            var cpuScaleUpRule = new ScaleRule()
            {
                // Define the MetricTrigger Properties
                MetricTrigger = new MetricTrigger()
                {
                    MetricName = "Percentage CPU",
                    MetricNamespace = "",
                    MetricSource = AutoscaleMetricSourceBuilder.BuildCloudServiceMetricSource(cloudServiceName, roleName, isProduction),
                    TimeGrain = TimeSpan.FromMinutes(5),
                    TimeWindow = TimeSpan.FromMinutes(15),
                    TimeAggregation = TimeAggregationType.Average,
                    Statistic = MetricStatisticType.Average,
                    Operator = ComparisonOperationType.GreaterThan,
                    Threshold = 60.0
                },
                // Define the ScaleAction Properties
                ScaleAction = new ScaleAction()
                {
                    Direction = ScaleDirection.Increase,
                    Type = ScaleType.ChangeCount,
                    Value = "1",
                    Cooldown = TimeSpan.FromMinutes(10)
                }
            };

            //Create a ScaleRule to Scale Down
            var cpuScaleDownRule = new ScaleRule()
            {
                // Define the MetricTrigger Properties
                MetricTrigger = new MetricTrigger()
                {
                    MetricName = "Percentage CPU",
                    MetricNamespace = "",
                    MetricSource = AutoscaleMetricSourceBuilder.BuildCloudServiceMetricSource(cloudServiceName, roleName, isProduction),
                    TimeGrain = TimeSpan.FromMinutes(5),
                    TimeWindow = TimeSpan.FromMinutes(15),
                    TimeAggregation = TimeAggregationType.Average,
                    Statistic = MetricStatisticType.Average,
                    Operator = ComparisonOperationType.LessThan,
                    Threshold = 25.0
                },
                // Define the ScaleAction Properties
                ScaleAction = new ScaleAction()
                {
                    Direction = ScaleDirection.Decrease,
                    Type = ScaleType.ChangeCount,
                    Value = "1",
                    Cooldown = TimeSpan.FromMinutes(10)
                }
            };

            // Add the rules to the profile
            createParams.Setting.Profiles[0].Rules.Add(cpuScaleUpRule);
            createParams.Setting.Profiles[0].Rules.Add(cpuScaleDownRule);

            // Apply the settings in Azure to this resource
            autoscaleClient.Settings.CreateOrUpdate(resourceId, createParams);

            //Get Auto Scale Settings if you need to check what is the current scaling settings
            AutoscaleSettingGetResponse setting = autoscaleClient.Settings.Get(resourceId);
            Console.WriteLine("existing settins: " + setting.ToString());

        }

        public static X509Certificate2 GetCertificate()
        {
            var certByteData = File.ReadAllBytes(@"E:\cie-jianw-managecert-pw1.pfx");
            X509Certificate2 certificate = new X509Certificate2(certByteData, "password");
            return certificate;
        }

运行完上述代码之后，开发者可以在portal上的management service中，确认自定义配置已生效：

AutoscaleSettings:
[
  {
    "Profiles": [
      {
        "Name": "Scale.WebApi.Cpu",
        "Capacity": {
          "Minimum": "1",
          "Maximum": "2",
          "Default": "1"
        },
        "Rules": [
          {
            "MetricTrigger": {
              "Name": "Percentage CPU",
              "Namespace": "CloudService",
              "Resource": "CloudService:test1205:Staging:WebRole1",
              "TimeGrain": "PT5M",
              "Statistic": "Average",
              "TimeWindow": "PT15M",
              "TimeAggregation": "Average",
              "Operator": "GreaterThan",
              "Threshold": 60.0
            },
            "ScaleAction": {
              "Direction": "Increase",
              "Type": "ChangeCount",
              "Value": "1",
              "Cooldown": "PT10M"
            }
          },
          {
            "MetricTrigger": {
              "Name": "Percentage CPU",
              "Namespace": "CloudService",
              "Resource": "CloudService:test1205:Staging:WebRole1",
              "TimeGrain": "PT5M",
              "Statistic": "Average",
              "TimeWindow": "PT15M",
              "TimeAggregation": "Average",
              "Operator": "LessThan",
              "Threshold": 25.0
            },
            "ScaleAction": {
              "Direction": "Decrease",
              "Type": "ChangeCount",
              "Value": "1",
              "Cooldown": "PT10M"
            }
          }
        ]
      }
    ],
    "SubscriptionId": "959bf13d-17fc-4cc2-9669-59××××ac2b",
    "Source": "CloudService:test1205:Staging:WebRole1",
    "Version": "1.0",
    "Enabled": true
  }
]

Resource Name: CloudService:test1205:Staging

Resource Type: Autoscale

在此配置下，scaling up将会在约 15分钟（time window） + 15分钟（后端性能数据延时） = 30分钟后发生。

 

Auto Scaling性能调优及关键参数

针对上述使用代码进行的自定义Scaling配置，其运行过程如以下截图：

Azure中的Monitor数据较实际时间落后了将近15分钟。

鉴于Azure后端对目标虚机的性能采用传输到Scaling模块需要一定的时间（经反复测试，约15～25分钟），一个较保险的TimeWindow的推荐值是25分钟。若TimeWindow=15但数据延时是20分钟时，Auto Scaling会无法得到最新一次的数据（去平均值失败），即影响Auto Scaling的表现。TimeGrain的最小单位是5分钟，对于CPU类型的Scaling，可以取最小值，从而得到最快的表现。

基于上述Auto Scaling的讨论以及实际工作的场景，一个常见的伸缩性方案设计如下：

另外，若开发项目中，对于Scaling要求实时响应，如CPU超过80%，立即Scaling Up的需求，开发者可以考虑结合前面推荐的WAD自定义分析https://blogs.msdn.com/b/jianwu/archive/2014/09/23/windows-azure-storage-wad.aspx 和 REST操作https://msdn.microsoft.com/en-us/library/azure/ee460809.aspx来实现更高级的Scaling方案。

]]> Windows Azure数据存储与比较 https://docs.microsoft.com/archive/blogs/jianwu/windows-azure Tue, 18 Nov 2014 06:48:00 GMT https://blogs.msdn.microsoft.com/jianwu/2014/11/18/windows-azure/ 对于绝大多数解决方案而言，数据都是至关重要的一部分，云计算中同样如此。在云计算里面，绝大多数传统存储的建议都可以直接拿来用，如数据冗余、数据加密等。但是云计算中的数据与传统的数据有所不同，主要体现在三个方面。

（1）数据可能存储在本地，也可能存储在云端

对于存储在本地的数据，如果云端服务需要连接，则可以通过建立不同的渠道来实现数据访问或者同步，如图所示Windows Azure中的数据同步技术。

  1. „ 网络与网络互联：IaaS中的Virtual Network实现本地网络和云端网络的无缝整合，云端数据和本地数据都将成为局域网内部的数据，通过传统的局域网通信方式即可实现数据访问和管理。
  2. „ 计算机间互联：IaaS中的Point-to-Site实现单台计算机与云端计算机的安全连接，支持基于IPv4的通信。
  3. „ 应用程序互联：Service Bus实现应用程序间的互联，数据可以通过Service Bus暴露或传递于不同平台上的应用程序之间。
  4. „ 数据同步：基于互联网和软件产品本身的数据同步，如SQL Server和SQL Azure的同步、本地Active Directory和云端Windows Azure Active Directory的同步。

（2）数据访问主要依赖于Internet，但数据访问和安全性要求极高

访问云端的数据或者通过建立云端连接来共享数据均需要保证数据安全性，Microsoft Azure中的数据服务大多数支持SSL通信加密，且提供多种不同安全级别的验证支持，对于SQL Azure和托管服务，开发者都可以自定义访问IP的“白名单”，实现更高级别的安全性保障。归纳起来，Microsoft Azure中的数据服务主要基于以下几个特性来实现数据安全：

  • „ SSL加密通信。
  • „ 使用用户名、密钥管理数据。
  • „ 使用SAS签名授权方式控制访问权限。
  • „ 基于ACS或者WAAD的登录授权。
  • „ IP白名单、黑名单。
  • 证书加密数据访问密钥等关键信息

（3）数据访问客户端运行于世界各地，访问速度很重要

云计算的数据可以存放在许多地方，而且WindowsAzure数据存储也是非常多样化的。为了实现高性能的数据服务，开发者可以选用最佳的数据存储/缓存来实践解决方案，同时对于公开的云端静态数据（如文件、视频等），选用较近的数据中心，并开启CDN加快访问速度。

选择正确的数据类型时，应充分考虑项目需求和数据上下游的交互压力，除了常用的文件存储和缓存技术之外，对于队列存储和表存储，开发者应仔细分析Queue Storage和Service Bus Queue、Table Storage和SQL Azure的对比，选择最佳的数据类型来实践。

如下表所示是Queue Storage和Service Bus Queue的对比。

服务特性

Queue Storage

Service Bus Queue

先进先出

不保证

保证

数据传输

至少传输一次

至少传输一次，至多传输一次

队列大小

最大可至200TB

最大不超过5GB

访问方法

读取 --> 独享 --> 删除

锁定 --> 读取 --> 删除

锁定时长

（lease）默认30s，最多为7天

（lock）默认60s，RenewLock API支持不断更新

锁定粒度

消息

整个队列

批量读取

支持，单次最多读取32条消息

支持，pre-fetch属性实现批量读取

批量写入

不支持

支持

读写效率

最优2000条消息/秒，平均延时10ms

最优2000条消息/秒，平均延时100ms

存储成本

经济

一般

  

如下表所示是Table Storage和SQL Azure的对比。

服务特性

Table Storage

SQL Azure

关系型数据存储引擎

不是

是

服务端处理

不支持

支持存储过程、联合、触发器等

批量读取

同一个索引区支持

全库支持

数据表结构

动态定义

固定定义

数据形式

固定有限量数据类型

多种数据类型，且支持自定义类型

存储成本

经济

一般

 

结合上述数据特性和前面讨论的Microsoft Azure开发技术，对于常用的数据共享和数据服务，作以下分析和对比（如下表）

数据形式

数据使用方

解决方案

数据公开
（互联网访问）

备注性能

数据通信

云端服务、本地服务、跨平台服务等

Service Bus Queue、Service Bus Topic

公开

实时传输，先进先出

数据通信

云端服务、本地服务、跨平台服务等

队列存储（Queue Storage）

公开

高并发、不保证先进先出

数据通信

云端服务、本地服务、跨平台服务等

托管式缓存（Managed Cache Service）

公开

数据缓存

数据通信

托管服务

In-Role Cache

不公开

托管服务内部使用

 

数据形式

数据使用方

解决方案

数据公开
（互联网访问）

备注性能

数据通信

托管服务

Internal Endpoint、WCF服务

不公开

托管服务内部通信

数据通信

虚拟网络

虚拟网络、局域网
通信

不公开

虚拟网络内部的托管服务、虚拟机、本地网络间的局域网通信

数据存储

云端服务、本地服务、跨平台服务等

Blob Storage

公开

文件存储

数据存储

托管服务、IaaS虚拟机

Blob Storage （页式）

不公开

托管服务虚拟机或者IaaS虚拟机加载成本地磁盘

数据存储

云端服务、本地服务、跨平台服务等

Table Storage

公开

非关系型数据存储、非固定数据结构类型的数据存储

数据存储

云端服务、本地服务、跨平台服务等

SQL Azure

公开

云端关系型数据库引擎

数据访问

云端服务、本地服务、跨平台服务等

账户名+密钥

公开

应用于WAS存储、SQL Azure存储、Managed Cache

数据访问

托管服务

指定Cache Role名称

不公开

In-Role Cache

数据访问

云端服务、本地服务、跨平台服务等

签名授权访问（Shared Access Signature）

公开

应用于WAS存储、Service Bus消息

数据访问

云端服务、本地服务、跨平台服务等

直接访问

公开

应用于CDN，设定为Public的WAS存储内容

数据访问

云端服务、本地服务、跨平台服务等

ACS、WAAD

公开

集成ACS、WAAD实现项目中的登录验证

数据可用性

托管服务

多个实例、Auto-Scaling

公开

负载均衡器实现高可用性

数据可用性

IaaS虚拟机服务

多台虚拟机、Load Balanced Endpoint

公开

负载均衡

数据可用性

云端服务、本地服务、跨平台服务等

CDN（数据传输网络）

公开

缓存静态文件

数据可用性

托管服务

Traffic Manager

公开

实现多个托管服务的对外统一服务

数据可靠性

托管服务

多个实例、Upgrade Domain

公共

提高托管服务可靠性

数据可靠性

IaaS虚拟机服务

Availability Set

公共

提高虚拟机服务对外可靠性

数据可靠性

WAS存储、SQL Azure存储、WAAD等

异域备份、数据同步、虚拟网络

不公开

实现数据备份、数据同步
 
 
 

数据形式

数据使用方

解决方案

数据公开
（互联网访问）

备注性能

数据通信

托管服务

Internal Endpoint、WCF服务

不公开

托管服务内部通信

数据通信

虚拟网络

虚拟网络、局域网
通信

不公开

虚拟网络内部的托管服务、虚拟机、本地网络间的局域网通信

数据存储

云端服务、本地服务、跨平台服务等

Blob Storage

公开

文件存储

数据存储

托管服务、IaaS虚拟机

Blob Storage （页式）

不公开

托管服务虚拟机或者IaaS虚拟机加载成本地磁盘

数据存储

云端服务、本地服务、跨平台服务等

Table Storage

公开

非关系型数据存储、非固定数据结构类型的数据存储

数据存储

云端服务、本地服务、跨平台服务等

SQL Azure

公开

云端关系型数据库引擎

数据访问

云端服务、本地服务、跨平台服务等

账户名+密钥

公开

应用于WAS存储、SQL Azure存储、Managed Cache

数据访问

托管服务

指定Cache Role名称

不公开

In-Role Cache

数据访问

云端服务、本地服务、跨平台服务等

签名授权访问（Shared Access Signature）

公开

应用于WAS存储、Service Bus消息

数据访问

云端服务、本地服务、跨平台服务等

直接访问

公开

应用于CDN，设定为Public的WAS存储内容

数据访问

云端服务、本地服务、跨平台服务等

ACS、WAAD

公开

集成ACS、WAAD实现项目中的登录验证

数据可用性

托管服务

多个实例、Auto-Scaling

公开

负载均衡器实现高可用性

数据可用性

IaaS虚拟机服务

多台虚拟机、Load Balanced Endpoint

公开

负载均衡

数据可用性

云端服务、本地服务、跨平台服务等

CDN（数据传输网络）

公开

缓存静态文件

数据可用性

托管服务

Traffic Manager

公开

实现多个托管服务的对外统一服务

数据可靠性

托管服务

多个实例、Upgrade Domain

公共

提高托管服务可靠性

数据可靠性

IaaS虚拟机服务

Availability Set

公共

提高虚拟机服务对外可靠性

数据可靠性

WAS存储、SQL Azure存储、WAAD等

异域备份、数据同步、虚拟网络

不公开

实现数据备份、数据同步

 

 

]]>