1. How To - Create an Azure Data Lake Store Account: https://mix.office.com/watch/1k1cycy4l4gen
2. How To - Use the Data Explorer to Manage Data in Azure Data Lake Store: https://mix.office.com/watch/icletrxrh6pc
3. How To - Connect Azure Data Lake Analytics to Azure Data Lake Store: https://mix.office.com/watch/qwji0dc9rx9k
4. How To - Access Azure Data Lake Store via Data Lake Analytics: https://mix.office.com/watch/1n0s45up381a8
5. How To - Connect Azure HDInsight to Azure Data Lake Store: https://mix.office.com/watch/l93xri2yhtp2
6. How To - Access Azure Data Lake Store via Hive and Pig: https://mix.office.com/watch/1n9g5w0fiqv1q
7. How To - Use DistCp (Hadoop Distributed Copy) to copy data to and from Azure Data Lake Store: https://mix.office.com/watch/1liuojvdx6sie
8. How To - Use Apache Sqoop to move data between relational sources and Azure Data Lake Store: https://mix.office.com/watch/1butcdjxmu114
9. How To - Data Orchestration using Azure Data Factory for Azure Data Lake Store: https://mix.office.com/watch/1oa7le7t2u4ka
10. Core Concepts - Securing Data in the Azure Data Lake Store: https://mix.office.com/watch/1q2mgzh9nn5lx

Step 2: Create Storage Account and Upload the VHD to Azure

Step 3: Create a virtual machine based out of the custom VHD

Step 4: Test your Virtual Machine

Step 1: Download RHEL ISO and prepare VHD to be upload to Azure

You can download the ISO from RedHat developer portal. Once you download the ISO, our next step is creating a VHD that is ready to be use in Azure. In order to prepare RHEL to be used in Azure you have to follow some specific steps which explains here. Note some important points before you start.

RHEL installation notes

• The newer VHDX format is not supported in Azure. You can convert the disk to VHD format by using Hyper-V Manager or the convert-vhd PowerShell cmdlet.
• VHDs must be created as "fixed"--dynamic VHDs are not supported.
• When you're installing the Linux operating system, we recommend that you use standard partitions rather than Logical Volume Manager (LVM) (which is often the default for many installations). This will help you to avoid LVM name conflicts with cloned VMs, particularly if an OS disk ever needs to be attached to another VM for troubleshooting. You can use LVM or RAID on data disks if you prefer.
• Do not configure a swap partition on the OS disk. You can configure the Linux agent to create a swap file on the temporary resource disk. More information about this is available in the steps below.
• All of the VHDs must have sizes that are multiples of 1 MB.
• When you use qemu-img to convert disk images to VHD format, note that there is a known bug in qemu-img versions 2.2.1 or later. This bug results in an improperly formatted VHD. The issue is intended to be fixed in an upcoming release of qemu-img. For now, we recommend that you use qemu-img version 2.2.0 or earlier.

In my case, I have downloaded the ISO and used Hyper-V to spin-up a virtual machine. There are detailed steps mentioned in the RedHat developer portal.

After creating the Virtual machine based out of the instructions from RedHat portal, next step was to prepare RHEL to be used in Azure. I had used RHEL 7.2 and I had to follow the instructions listed here.

Once you are done till here, you can shut down the virtual machine. In my case I was using the VHDX format, I had to convert it to VHD first to upload the OS disk to Azure.

Convert-VHD –Path E:\Hyper-V\rhlinux-dev.vazure loginhdx –DestinationPath E:\Hyper-V\rhlinux-dev.vhd –VHDType Fixed 

Above will create a VHD disk with Fixed size which is what we need in Azure. Next steps are working with Azure

Step 2: Create Storage Account and Upload the VHD to Azure

I used Azure CLI for most part of the remaining tasks.

//login to Azure, below CLI will give you a code and you can login in the web portal using that code which will authentication you in the CLI

azure login

//List out all of your subscription to set the one that you need to test these steps

azure account list

// In my case I have used my MSDN Azure Subscription

azure account set "Visual Studio Ultimate with MSDN"

// Its recommend to use the ARM model, below CLI will switch to ARM Mode

azure config mode arm

// Create a Resource Group by specifying the name and location

azure group create -n "dev-redhat-rg" -l "East US"

// Create storage account, I'm not providing all parameters in line instead I let the command to prompt

azure storage account create "devredhatstg01"

//You have to grab the storage account key for further steps, so list the keys, used the same resource group

azure storage account keys list devredhatstg01

//create a container

azure storage container create "rhlxvhdtemplate" -k "SEmner/2424jljadfasdfasdl+4VDplBJhw6LCS0rDYstM/chdYf3VyHkzRA==" -a "devredhatstg01"

//Upload the VHD, it took 2 hours for me to complete, for info below.

azure storage blob upload -f "E:\Hyper-V\rhlinux-dev.vhd" -b "rhlx-dev.vhd" -t page --container "rhlxvhdtemplate" -a "devredhatstg01" -k "SEmner/ZP7HIMukWI46WBC+82usasdfhl23kddlaYP34ouklj345l34jljl34k5j3lk5j+4VDplBJhw6LCS0rDYstM/chdYf3VyHkzRA=="

It took two hours to upload 25 GB VHD to Azure storage account. I think this experience will vary depends on your network bandwidth speed to Azure data center. I'm in Woodbridge, US and chosen East US as my Azure Storage location.

Step 3: Create a virtual machine based out of the custom VHD

Now, we are ready to create a virtual machine based out of the VHD we have it in Azure. I used ARM template and CLI to provision the VM using the custom image. You can get it from here. Once you are done you can see that VM is up and running. I had to create a NSG and create a SSH input rule for port 22 to enable SSH remotely.

azure group deployment create --template-uri "https://raw.githubusercontent.com/Azure/azure-quickstart-templates/master/101-vm-from-user-image/azuredeploy.json" dev-rhel-vm-rg customVhdDeployment

Step 4: Test your Virtual Machine

I used Putty on my windows 10 machine to SSH and test the machine. You might need to register to your RHEL subscription if you have already unsubscribed part of RHEL preparation.

Good news is – RHEL provides no-cost RHEL developer subscription and one can sign up for it from this link. https://developers.redhat.com/blog/2016/03/31/no-cost-rhel-developer-subscription-now-available/

I would strongly recommend to read the Terms & Conditions before you start https://developers.redhat.com/terms-and-conditions/

You can create a RHEL VHD based out of the developer subscription and you can bring your VHD to azure. You won't be getting SSO experience like the premium images in Azure though, but I believe it doesn't matter for developers J

In my next blog post, I'm going to give go deep into this process. I have already tested and it works great in my Azure subscription. Stay tuned!

https://powerapps.microsoft.com/en-us/blog/introducing-microsoft-powerapps/

With the help of Azure App Service its now very easy to deploy and manage APIs for PowerApps. Have a look at the below resources to get started.

https://channel9.msdn.com/Events/Microsoft-Azure/PowerApps/PowerApps-and-Azure-App-Service-for-Developers

https://azure.microsoft.com/en-us/documentation/suites/powerapps/

"<View><Query><Where><Geq><FieldRef Name='EmployeeNumber'/>
      <Value Type='Number'>10</Value></Geq></Where></Query><RowLimit>100</RowLimit></View>";

When we started looking at this issue, it was little confusing as they were filtering and pulling only 100 items from a list which has 7000 items. This leads us to look at other throttling contributors, we looked at the lookup columns in the custom list and there were only 2 of them. There was nothing logged in the ULS except the below throttling error message with call stack.

“The attempted operation is prohibited because it exceeds the list view threshold enforce by the administrator. 0x80070024”

Finally we were able to figure out that the “Employee Number” column was not added as an indexed column in the SharePoint List. It is mandatory to add an indexed column(s) for the column(s) that we are using for filtering from SharePoint list views. It is clearly documented in in the UI settings of any list views.

Here the take away is we must need to create an indexed column(s) for the filtering column(s) even from any custom code. Note: maximum indices on a list is 20, so plan your list schema in advance.

"Solutions no longer support deploying Code Access Security (CAS) policy so any CAS policy specified in solution "{0}" was skipped and not deployed. Deploy this solution without the -CASPolicies flag to suppress this warning."

The existing alert that logs to ULS that CAS policy is detected and getting skipped is still in place so if  you removes the -CASPolicies (for PowerShell) or -allowCasPolicies (for stsadm) flag to suppress the above warning, you will continue to get an entry in the ULS log explaining their CAS policy has been skipped until you actually remove the CAS policy from the solution (You can verify it in the manifest.xml of WSP).

Except Usage Logging & State Service application all other service applications were able to use the pre-created database while provisioning the new service application using New-SP* commandlets. Usage Logging & State Service applications were complaining that the there is already a database exists with the same name. To work-around this, we have to approach the provision in a different way. I’m giving the provisioning script samples for both Usage Logging & State Service Applications.

Usage Logging Service Application :

Steps are explaining below.

1. Create new Usage Service Application with a temporary ( or default) database

2. After that, modify the Usage Service Application to use the pre-created database name, this will dismount the usage logging database created in step #1

3. Delete the database created in step #1 manually from the SQL server

Automating the Steps 1-2 given below.

add-pssnapin microsoft.sharepoint.powershell

Write-Host -ForegroundColor White "Provisioning WSS Usage Application..."

$SPUsageApplicationName = "WSS Usage Service Application"

$precreatedUsageDB = "WSS_Usage_Database"

$sqlserver = "spsql"

$tempUsageDB = "WSS_Usage_ToDelete"

New-SPUsageApplication -Name $SPUsageApplicationName -DatabaseName $tempUsageDB               

$UsageServiceApp = Get-SPUsageApplication $SPUsageApplicationName

Set-SPUsageApplication -Identity $UsageServiceApp.ID -DatabaseName $precreatedUsageDB -DatabaseServer $sqlserver

Write-Host -ForegroundColor White "Re-provisioning Health Data Collection Proxy as by default it will be in stopped state"

$SPUsageApplicationProxy = Get-SPServiceApplicationProxy | where {$_.DisplayName -eq $SPUsageApplicationName }

                                                                    $SPUsageApplicationProxy.Provision()

Write-Host -ForegroundColor Blue "Done provisioning SP Usage Application."

State Service Application :

This is little different than Usage Logging Service Application provisioning.

Steps are explaining below.

1. First we have to mount the pre-created database to SharePoint and have to initialize it.

2. Use the same method as other service application provisioning, using New-SPStateServiceApplication and provide the pre-created database name.

3. Create state service application proxy.

add-pssnapin microsoft.sharepoint.powershell

$StateServiceDB = "State_Service_DB"

$StateServiceName = "State Service Application"

$StateServiceProxyName = "State Service Application Proxy"

Write-Host -ForegroundColor White "Provisioning State Service Application..."

Mount-SPStateServiceDatabase -Name $StateServiceDB | Initialize-SPStateServiceDatabase

New-SPStateServiceApplication -Name $StateServiceName -Database $StateServiceDB | Out-Null

Write-Host -ForegroundColor White "Creating State Service Application Proxy..."

$StateServiceApplication = Get-SPStateServiceApplication -identity  $StateServiceName

$StateServiceApplication | New-SPStateServiceApplicationProxy -Name $StateServiceProxyName -DefaultProxyGroup 

Write-Host -ForegroundColor Blue " - Done creating State Service Application."

Other service applications can be pointed in this scenario are

1. ASP.NET Session state service (Enable-SPSessionStateService). With that there is no way to specify a pre-created database , 

    ASP.NET session state database schema is provisioned by ASP.NET by calling SQLServices class. once you use Enable-SPSessionStateService.

From my deployment experience this was the main issue and there was no any other deployment issues faced while going with the DBA pre-created deployment method. Hope this hint will help someone.

Here is my first blog after a long vacation back from my hometown in India. Tomorrow is a special day as Hurricane Sandy is going to hit my area ( Edison, NJ) , so writing this post tonight itself :)

There are many articles out in web provides the details of Office Web Apps 2013 , its planning, deployment etc. What I’m trying to give in this post is my experience with the installation and configuration of Office Web Apps 2013 for SharePoint Server 2013.

Environment details are given below :-

Total 4 Servers in my Windows Azure Virtual Machine Environment.

litdc – AD & DC (Windows Server 2008 R2)

litsp1 – WFE – SPS 2013 RTM (Windows Server 2012 Datacenter Edition)

litsp2 – App Server – SPS 2013 RTM (Windows Server 2012 Datacenter Edition)

litsp3 – OWA 2013 RTM Server (Windows Server 2012 Datacenter Edition)

After installing and configuration of my SharePoint 2013 environment I have started the setup for Office Web Apps 2013 in litsp3.

Please refer the TechNet articles for details about planning & deployment of Office Web Apps 2013. In this post I’m providing details of deployment of Office Web Apps 2013 on a Windows 2012 Server. Also, this will be a single server OWA farm only.

Step #1

Enable the prerequisites for Office Web Apps 2013 in Windows Server 2012. You can do this using PowerShell. Open Windows PowerShell Console as Administration and run the below.

# if script execution policy is “Restricted”, enable it using Set-ExecutionPolicy "UnRestricted"

Add-WindowsFeature Web-Server,Web-Mgmt-Tools,Web-Mgmt-Console,Web-WebServer,Web-Common-Http,Web-Default-Doc,Web-Static-Content,Web-Performance,Web-Stat-Compression,Web-Dyn-Compression,Web-Security,Web-Filtering,Web-Windows-Auth,Web-App-Dev,Web-Net-Ext45,Web-Asp-Net45,Web-ISAPI-Ext,Web-ISAPI-Filter,Web-Includes,InkandHandwritingServices

System restart required after provisioning the above prerequisites features.

Step #2

After system reboot, go ahead and run the OWA 2013 installation setup as administrator.

Step #3

Before creating a new OWA farm, you have to do couple of things based upon the planning you have done. Make sure that there won’t be any application running in default ports in the server that you are setting up the OWA farm. As a best practice, always use HTTPS for the communication, so in my test environment, I have imported a domain certificate in the IIS. Since this was a domain certificate that got from the CA (certificate authority) in my DC it was already added to my trusted root authority in the litsp3 server, if not , you have to import it manually.

Step #3

Once the server certificate is installed, its ready to provision the new OWA single server farm. You have to run New-OfficeWebAppsFarm PS commandlet , if you need more details of the Commandlet and also to verify whether the PowerShell snap is correctly added,  you can use

Get-Command New-OfficeWebAppsFarm

As you see it in the screenshot below, also notice the default web site in IIS running in port 80, because after creating a new web app farm it will remove that site.

Step #4

Execute the below PowerShell Commandlet

New-OfficeWebAppsFarm –InternalUrl “https://litsp3” -ExternalUrl https://litsp3.litware.local –CertificateName "OfficeWebApps Certificate" –EditingEnabled

Since I have used the –EditingEnabled parameter it will prompt the confirmation for the licensing. Remember that editing is only possible in SharePoint 2013 and not in Exchange & Lync 2013. After executing the commandlet successfully , we can see the details of the configurations. It will list out the URLs, Cache location, ULS log location and other specific configuration for excel and stuff.

Now, if you look at the IIS you can see that default web site has been replaced by OWA web sites. To confirm a successful configuration you can browse to the web service discovery location like this : https://litsp2.litware.local/hosting/discovery

In the below screenshot you can see the IIS websites for OWA and a discovery details for the above mentioned URL. Please note the port numbers used for the binding. Make sure that all those ports are not blocked by the firewall to avoid any communication issues with OWA.

Step #5

Alright, once above steps are done then next step is to configure SharePoint Server 2013 to make the WOPI binding with Office Web Apps 2013.

You have to use New-SPWOPIBinding commandlet to do that.  In my environment I run it like below.

New-SPWOPIBinding –ServerName “litsp3.litware.local”

Office Web Apps Server uses the concept of zones to determine which URL (internal or external) and which protocol (HTTP or HTTPS) to use when it communicates with the host, which in this case is SharePoint 2013. By default, SharePoint Server 2013 uses the internal-https zone. Verify that this is the current zone by running the following command:

Get-SPWOPIZone

After you run this command, you should see the WOPI zone displayed. It should be internal-https.

Finally to test everything working fine or not, create or upload different types of office documents. Below screenshot is a test with a new PowerPoint document. You can see the URL that since SharePoint 2013 is already has the capability as a host using WOPI, it will detect the document and render it in the WOPI interface (/_layouts/15/WopiFrame.aspx)

Troubleshooting & Monitoring

For troubleshooting, we have our friend :) ULS log available for Office Web Apps 2013 as well. Location is different than default location of SharePoint. You might have noticed it in the Step #4 that after provisioning a new farm it will provide the details of the ULS log as well.

Since by default ULS log location configured under “C:\ProgramData\Microsoft\OfficeWebApps\Data\Logs\ULS”, you have to enable the hidden files  in C:\ drive as “ProgramData” folder is hidden by default.

Location of the ULS log can be changed using Set-OfficeWebAppsFarm, but only one caveat is that to get the change to effect , we have to restart the OWA server, if there are more servers in the farm all servers has to be started to get the change updated. Not only this setting there are other configuration changes as well which needs the system restart, you can refer the documentation of above mentioned PS commandlet for more information about those all. I would recommend you to go through the that PS commandlet and plan all your required configurations before the deployment so that way you can avoid any system reboot after the deployment.

Like in SharePoint, ULS log will be very helpful to get to know what is happening behind the scene and hence troubleshoot issues easily. In my case I have faced an issue while rending the document in SharePoint web interface after the configuration. Then I have checked the ULS logs in OWA server then found that there was a communication issue due to firewall in my environment. In windows Azure virtual machines ping is disabled by default I have had to enable the below firewall rule (Virtual Machine Monitoring (Echo Request - ICMPV4-In)) to get it working.

ULS log exception is given below.

Health report by PowerPointViewingServicesWatchdog_Web: Agent: WebPPTViewer, eventId: 1188, eventType: Error, categoryId: 0, eventMessage: PowerPointViewingServicesWatchdog_Web reported status for WebPPTViewer in category 'WfeRedirect'. Reported status: Exception when checking /default.aspx: System.Net.WebException: Unable to connect to the remote server ---> System.Net.Sockets.SocketException: No connection could be made because the target machine actively refused it 10.10.1.10:80   
at System.Net.Sockets.Socket.DoConnect(EndPoint endPointSnapshot, SocketAddress socketAddress)   
at System.Net.ServicePoint.ConnectSocketInternal(Boolean connectFailure, Socket s4, Socket s6, Socket& socket, IPAddress& address, ConnectSocketState state, IAsyncResult asyncResult, Exception& exception)     -
-- End of inner exception stack trace ---   
at System.Net.HttpWebRequest.GetResponse()   
at Microsoft.Office.Web.Common.WatchdogHelper.CheckWfeRedirect(ServiceInstance instance, Int16 category, String categoryName)

For monitoring the OWA server there are many performance counters available

Perfmon view to monitor the performance by using the counters.

Task Manager view of OWA related processes

Hope this post will be helpful for someone !

I would recommend to read the below blog posts as well

Spencer’s article on Editing License with OWA  2013

Ali’s article on Machine Translation Services with OWA 2013

Steve’s article on Configuring OWA 2013

Here is the scenario. Customer has two Windows 2008 R2 Std machines (lets call it as srvsp1 & srvsp2) installed and configured with the following products.

1. SharePoint Server 2010 Enterprise Edition
2. Office Web Apps

Recently we have patched those two servers with June 2012 CU , for some troubleshooting one of the servers (srvsp2) was disconnected from the farm. After completing the installation while trying to add the server back to the farm by running the PSConfig we got the below error,

The license state for the current server doesn't match the farm's license state.
System.InvalidOperationException: The current server cannot be joined to this farm because the set of installed products does not match the products installed in the farm.

The license state for the current server doesn't match the farm's license state.
   at Microsoft.SharePoint.Administration.SPFarm.Join()
   at Microsoft.SharePoint.PostSetupConfiguration.ConfigurationDatabaseTask.CreateOrConnectConfigDb()
   at Microsoft.SharePoint.PostSetupConfiguration.ConfigurationDatabaseTask.Run()
   at Microsoft.SharePoint.PostSetupConfiguration.TaskThread.ExecuteTask()

Here are the troubleshooting steps.

# Step 1

Verify the configuration database has correct entries of all installed products. For that , we ran the following query against the “Objects” table of SharePoint configuration database.

Select * from Objects with (nolock) where properties like '%icrosoft.SharePoint.Administration.SPInstallState%'

From the result copy the “Properties” column value to notepad or any xml editor tool, in my case I have used Visual Studio.

Values that we are interested the ones highlighted in the below screenshot. Same is given below.

<sFld type="Guid">beed1f75-c398-4447-aef1-e66e1f0df91e</sFld>
<sFld type="Guid">d5595f62-449b-4061-b0b2-0cbad410bb51</sFld>

# Step 2

Once we get those GUIDs, next step is make sure that any difference in the actual Physical SharePoint server’s installed products and configuration database.

To do that, check the below registry location in both the SharePoint Servers (srvsp1 & srvsp2).

HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Shared Tools\Web Server Extensions\14.0\WSS\InstalledProducts

In our case it looked like this:

{90140000-1014-0000-1000-0000000FF1CE} REG_SZ BEED1F75-C398-4447-AEF1-E66E1F0DF91E
{90140000-110D-0000-1000-0000000FF1CE} REG_SZ D5595F62-449B-4061-B0B2-0CBAD410BB51
{90140000-112D-0000-1000-0000000FF1CE} REG_SZ 926E4E17-087B-47D1-8BD7-91A394BC6196

Here we can clearly see that the physical box has 3 entries whereas the configuration database has only 2 of them.

More details of the GUIDs are given below.

BEED1F75-C398-4447-AEF1-E66E1F0DF91E is for "SharePoint Foundation 2010",

D5595F62-449B-4061-B0B2-0CBAD410BB51 is for "SharePoint Server 2010 Enterprise"

926E4E17-087B-47D1-8BD7-91A394BC6196 is for “Office Web Companions 2010" (Office Web Apps)

(Just FYI, for more information all different product SKUs and it’s GUIDs please refer this article : https://msdn.microsoft.com/en-us/library/ff721969.aspx )

In our case, the Configuration database should have had 3 entries to match the registry in a normal farm. For some reason it was not updated with the office web apps production installation , it may caused by an installation issue.

Resolution:
=========

Run the following Power Shell commandlet on any working server in the farm. In our case it was srvsp1, the reason we would need to run from this server is because the problem server – srvsp2 is still not in the farm.

Set-SPFarmConfig –InstalledProductsRefresh

This will pick the entries from the server and reset the entries in the Config DB. To double-check, you can run the same SQL query mentioned above and confirm the <sFId /> values.

Hope this will helpful while troubleshooting this specific issue related to SharePoint. There are different scenarios as well, eg: configuration database may have the values but it was not there in the physical server’s registry. To fix that you may end up with uninstall and install the product to update it correctly.