Kenneth's Journal https://docs.microsoft.com/archive/blogs/kennethteo/feed.xml A blog column to share my two cents, learning and sometimes, rant. Tue, 16 Apr 2019 05:17:15 GMT en-US hourly 1 TfsMigrator ISVError:300005 Mapping MemberSid > MemberId https://docs.microsoft.com/archive/blogs/kennethteo/isverror300005-mapping-membersid-memberid Thu, 09 Aug 2018 16:40:42 GMT https://blogs.msdn.microsoft.com/kennethteo/?p=575 Note and warning: This is a workaround and by no mean an official fix, use with care and make sure you backup your config database. If you are using tfsmigrator you may have encountered this error ISVError:300005 Unexpected non group identity was found to have direct membership to everyone group, and according to official documentation https://docs.microsoft.com/en-us/vsts/articles/migration-troubleshooting you need to run DECLARE @p6 dbo.typ_GroupMembershipTable INSERT into @p6 values('{GroupSid}','Microsoft.TeamFoundation.Identity','{MemberId}',0) EXEC prc_UpdateGroupMembership @partitionId=1,@scopeId='{ScopeId}',@idempotent=1,@incremental=1,@insertInactiveUpdates=0,@updates=@p6,@eventAuthor='9EE20697-5343-43FC-8FC5-3D5D455D21C5',@updateGroupAudit=0 You are supposed to populate the above t-sql with the parameters you get from the error tfsmigrator threw out: ISVError:300005 Unexpected non group identity was found to have direct membership to everyone group. GroupSid:S-1-9-1551374245-3746625149-2333054533-2458719197-2313548623-0-0-0-0-3, MemberId:76050ddf-4fd8-48c4-a1ff-859e44364519, ScopeId:7df650df-0f8b-4596-928d-13dd89e5f34f But instead of getting MemberId like all thing should be, you are getting, MemberSid:System.Security.Principal.WindowsIdentity;S-x-x-xx-xxxxxxxxx-xxxxxxxxx-xxxxxxxxxx-xxxxxx What you do? Note down the ScopeId and GroupSid from the original tfsmigrator error and perform the following t-sql #1 SELECT DisplayName from tbl_Group where Sid = '{GroupSid}' This should get you the offending tfs group #2 SELECT Name from tbl_GroupScope where ScopeId = '{ScopeId}' This should give you the project name (I think) at least according to this doc https://docs.microsoft.com/en-us/previous-versions/visualstudio/visual-studio-2013/ff731995(v=vs.120) Go to project tfs web portal (from #2) tfs>admin>security find tfs group from #1 and look for any suspicious member (hopefully not a lot, anything that is non TFS group identity, such as an AD group or user identity<from here) and find the accountname #3 SELECT Id from tbl_Identity where AccountName = '{AccountName}' You should get the membereid, I HOPE. Powered with this memberid from #3 you can proceed to re-run the command as suggested https://docs.microsoft.com/en-us/vsts/articles/migration-troubleshooting DECLARE @p6 dbo.typ_GroupMembershipTable INSERT into @p6 values('S-1-9-1551374245-3746625149-2333054533-2458719197-2313548623-0-0-0-0-3','Microsoft.TeamFoundation.Identity','76050ddf-4fd8-48c4-a1ff-859e44364519',0) EXEC prc_UpdateGroupMembership @partitionId=1,@scopeId='7df650df-0f8b-4596-928d-13dd89e5f34f',@idempotent=1,@incremental=1,@insertInactiveUpdates=0,@updates=@p6,@eventAuthor='9EE20697-5343-43FC-8FC5-3D5D455D21C5',@updateGroupAudit=0]]> Azure Kubernetes Service (AKS) with Azure AD https://docs.microsoft.com/archive/blogs/kennethteo/aks-with-aad Thu, 26 Jul 2018 14:04:32 GMT https://blogs.msdn.microsoft.com/kennethteo/?p=545 here, and do check them out from time to time because things change very freuqently. After reading the article above, I thought I could (maybe) everyone a favor by explaining a little bit more in details (and in layman terms ala Explain like I'm Five eli5) on what is going on. Here's my attempt: In Azure, you get to add users and groups to resources and resource groups, delegating permission and grant them access as needed. Wouldn't it be great if you can just add your colleague within your Azure AD tenant to AKS so you get to control what access they have and they get to sign-in with their usual corporate credentials? This is exactly what AKS allows you to do today. Here's what you need:
  1. An Azure Subscription (you can get it for free for a year thorugh my.visualstudio.com)
  2. Azure Kubernetes Services (AKS) - You will find a lot of references on how to create this but I suggest you to follow this guide
  3. Azure CLI, with AZ AKS installed (I like to do everything locally, but you can work with cloud shell as well)
Once you have you AKS ready run the following command:
  1. az login and follow on screen instruction
  2. az aks get-credentials --resource-group k8s --name k8s --admin Basically, what the above does is to get credential from your Kubernetes cluster and store them in C:\Users\[username]\.kube\config so kubectl can use them later.
  3. az aks browse --resource-group k8s --name k8s You can then open up the Kubernetes Dashboard by running above command. This will create a tunnel, and open up your browser to Kubernetes cluster's dashboard. (not very imporatant, but good to know)
you are now seeing the dashboard using the credential stored in .kube\config (for me it it was clusterAdmin_k8s_k8s) Depends on what version of Kubernetes you are running, if you see the following error, that's because AKS (the latest one as of the time of this blog's entry) has RBAC enabled by default and there is an extra step you need to do. [caption id="attachment_555" align="alignnone" width="1024"] basically the first error is telling you the service account (kubernetes-dashboard) you are using to view the k8s dashboard at namespace (kube-system) has no permission to do what is needed (to view configmaps) - configmaps is forbidden: User "system:serviceaccount:kube-system:kubernetes-dashboard" cannot list configmaps in the namespace "default"[/caption] Run this kubectl create clusterrolebinding kubernetes-dashboard --clusterrole=cluster-admin --serviceaccount=kube-system:kubernetes-dashboard We are creating a "clusterrolebinding" name it as "kubernetes-dashboard" with "cluster-admin" as the permission level and grant this to the service account "kubernetes-dashboard" at "kube-system" namespace. Now refresh.

RBAC Binding

To add a Azure AD user, create a .yaml file with following contents apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: contoso-cluster-admins roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: cluster-admin subjects: - apiGroup: rbac.authorization.k8s.io kind: User name: "[username]@contoso.onmicrosoft.com" run this kubectl apply -f "[path to .yaml]" you can then run kubectl get namespaces --as=[username]@contoso.onmicrosoft.com to verify We were doing all the above through "local user account --admin", if you may. What we are going to do next is to login and run kubectl using Azure AD user. Remove all the context in .kube\config by running kubectl config unset contexts.k8s kubectl config unset clusters.k8s kubectl config unset users.clusterUser_k8s_k8s or you can delete .kube\config just to be safe az aks get-credentials --resource-group k8s--name k8s kubectl get nodes You will be asked to sign-in again, and you can use the azure ad account you just added to sign-in. I haven't test the following scenario yet, but if I am correct you now can add any developers as reader to the Azure Resource Group as reader (so they can run az aks get-credentials) and then grant them permission separately within Kubernetes cluster using RBAC-Binding I hope my attempt in eli5 was good enough.]]> Analyzing W3C IIS Log with PowerBI https://docs.microsoft.com/archive/blogs/kennethteo/analyzing-w3ciis-log-with-powerbi Fri, 27 Apr 2018 17:19:22 GMT https://blogs.msdn.microsoft.com/kennethteo/?p=425
  • Log Parser (free!)
  • PowerBI Desktop (free!)

    • Transforming W3C IIS Logs to CSV

    You most likely have over hundreds of log to be analyzed, I wrote a script to help in converting them into CSV. In my most recent customer, they have 500 logs, over period of just one year.
    1. Make sure you have logparser installed, and note down the install location. I wrote a small batch file to convert W3C IIS logs to CSV using logparser. You can get the .bat here, make sure to modify the script to include your logs location, and logparser.exe location.
    2. Run the script, and the converted .CSV will be placed in the same folder as your logs. [caption id="attachment_435" align="alignleft" width="1024"] Converting W3C IIS Logs into CSVs[/caption]

    Import Converted .CSVs into PowerBI

    1. Open Power BI
    2. Choose Get File and select the folder where your converted CSV Logs reside, click OK
    3. You will notice that it is importing the file's binary (column 1) and its metadata (rest of columns). We don't want the metadata, we want the binary within these files. Click Edit.
    4. In the Query Editor, remove all the other columns except the one contains the binary data.
    5. Once you remove the rest of the columns, click on the button (next to Content, with two downward arrows) as shown below to expand the binary data
    6. Once expanded, in the Query Editor, click "Close and Apply"
    7. Wait for a few minutes, and you should see the "report" windows.
    8. You can then build any report or charts you like based on the CSV you have imported. The example I have shows table with highest average return time, sc-status, etc.
    A sample of the PowerBI Model (.pbix) used in this blog can be downloaded from my GitHub repository Happy charting.]]>         Azure App Service - Options Method (and other HTTPS Verb) disabled / not working in Azure Web App https://docs.microsoft.com/archive/blogs/kennethteo/azure-app-service-options-method-and-other-https-verb-disabled-not-working-in-azure-web-app Sat, 17 Mar 2018 15:57:43 GMT https://blogs.msdn.microsoft.com/kennethteo/?p=375 What success should look like. This was hosted on a Windows 2012 R2 VM's IIS[/caption] Note that in the example above we are calling the Http verb - Options, basically exposing to the clients what options are available from the service; in this case "WebSockets", "ServerSentEvents", and "LongPolling". As the application was uploaded to Azure App Service as an API, calling it from Postman, gave us this error message: "the resource you are looking for has been removed, had its name changed, or is temporarily unavailable - HTTP 401". Since this is not 404 error, we are quite sure the resource we are looking for is there, but not responding to our HTTP request. After some digging around, it turns out that on Azure Web App, Options method is being disabled by default. We eventually added the following lines in one of our Web.config files to allow it. 
    <configuration>
 <system.webServer
  <security>
   <requestFiltering>
    <verbs allowUnlisted="true">
     <add verb="OPTIONS" allowed="true" />
    </verbs>
   </requestFiltering>
  </security>
 </system.webServer>
</configuration>
    Unfortunately, I wasn't able to find any official documentation on this behavior, and if you do please let me know at the comment section!]]>         Finding QnAKnowledgebaseId and QnASubscriptionKey to configure the new Azure Bot Service https://docs.microsoft.com/archive/blogs/kennethteo/finding-qnaknowledgebaseid-and-qnasubscriptionkey-to-configure-the-new-azure-bot-service Wed, 24 Jan 2018 20:12:11 GMT https://blogs.msdn.microsoft.com/kennethteo/?p=335 Here's where you can find them: Go to https://qnamaker.ai/Home/MyServices click on your bot's "View Code" The first string right after "knowledgebases" is your QnAKnowledgebaseId, whereas the one right after Ocp-Apim-Subscription-Key is your QnASubscriptionKey   Hope that helps.]]> Quick Guide on Importing Data from SQL Server (or SQL Azure) to Cosmos DB https://docs.microsoft.com/archive/blogs/kennethteo/quick-guide-on-importing-data-from-sql-server-or-sql-azure-to-cosmos-db Tue, 16 Jan 2018 21:52:19 GMT https://blogs.msdn.microsoft.com/kennethteo/?p=315
  • Create SQL DB Table.txt That will create the initial database in SQL Server or SQL Azure
  • The main Program will read the database, parse the data and store them in Cosmos DB
    • Before you start the program, you will have to do step 1, and edit the app.config:
    • endpoint: Your cosmosDB endpoint
    • primaryKey: Your cosmosDB primary key, you can get this from your Azure Portal's CosmosDB blade
    • connectionString: The connection string to the SQL Database where your original data resides. In the sample App.config, this is referenced to SQL Azure, Microsoft Azure SQL DB
    • database: database name in your CosmosDB
    • collection: Collection name of the CosmosDB database
    ]]>         Enable "Analyze" tab in Visual Studio https://docs.microsoft.com/archive/blogs/kennethteo/enable-analyze-tab-in-visual-studio Sat, 25 Nov 2017 13:48:41 GMT https://blogs.msdn.microsoft.com/kennethteo/?p=276 Fortunately, Microsoft made it easier to add (or remove) extra features retroactively with ease. If you are missing Analyze tab in your Visual Studio, just follow the following steps:
    1. Open Visual Studio (I am on Visual Studio 2017), you can download VS2017 here
    2. Open up Visual Studio Installer, the easiest way to open the installer is to go File>New>Project, in the New Project window, click Open Visual Studio Installer.
    3. Click on "Individual components" tab, check on "Code Clone", "Code Map", "Developer Analytics Tool", "Live Dependency Validation" and on the right summary panel, check "Architecture and analysis tools"
    That's it, you can now do code clone and other type of analysis, including Static Code Analysis using some of the built-in rule sets.  ]]>         Deleting Azure AD Tenant - Unable to delete all Enterprise Applications https://docs.microsoft.com/archive/blogs/kennethteo/deleting-azure-ad-tenant Tue, 19 Sep 2017 19:27:08 GMT https://blogs.msdn.microsoft.com/kennethteo/?p=195 Hi All,
    This is a quick guide on deleting an Azure AD Tenant, especially when there is some stubborn enterprise applications that refuse to go away. Few weeks ago, I was trying to delete an Azure AD that I created as a test account for my customer. I happily go to portal.azure.com, changed to the relevant directory, in this case it was ronelx.onmicrosoft.com, and try to delete it using the "delete" button. Unfortunately, it came to a pause when Azure AD complaints that I have an enterprise application that needed to be removed. Normally you can click on the button "Delete all enterprise applications" as shown in the screenshot and you can delete them manually:
    However, as you can see from the following screenshot, when I try to delete, it refuses to go away with the "delete" button greyed out.
    WHAT DO WE DO?! First of all, you will need to create a new Global Admin OrgID (Work or School account) in the directory you are trying to delete (assuming you have removed all your users, as this is the first thing Azure AD checks for as shown above when you try to delete Azure AD tenant). Make sure you copy the temporary password: Go to https://www.powershellgallery.com/packages/AzureAD/ to get the latest powershell Azure AD module
    Start Powershell and run:
    Install-Module -Name AzureAD
    Once done run
    Connect-AzureAD
    You will be prompted to login, login with the user you created and will be asked to change your password:
    You should now be able to list all the current application SPs in a directory:
    Get-AzureADServicePrincipal
    Now, delete all non-required apps in the tenant directory, for me it was the Microsoft Visual Studio Team Services. Remove-AzureADServicePrincipal -objectid [enter object id here]Now that you are done, you will have to delete your user in AAD first, before deleting the Directory, or you will get a message saying you cannot delete the directory, because you have one or more users.
    Try deleting the Azure AD again, and this time it should pass all the checks:

    Hope this helps.
    Reference: https://blogs.msdn.microsoft.com/ericgolpe/2015/04/30/walkthrough-of-deleting-an-azure-ad-tenant/ (this blog was written in 2015, I have updated some powershell scripts, as you can now install powershell modules through PowerShell Gallery)
    ]]>         [Event] Distributor and Partner Azure Enablement - January 2017 https://docs.microsoft.com/archive/blogs/kennethteo/event-distributor-azure-enablement-january-2017 Tue, 17 Jan 2017 17:45:49 GMT https://blogs.msdn.microsoft.com/kennethteo/?p=185 TCO Calculator

    https://tco.microsoft.com

    Azure AWS Service Mapping

    https://docs.microsoft.com/en-us/azure/guidance/guidance-azure-for-aws-professionals-service-map

    [slides coming soon]

    ]]>         Separating Tags in Azure Usage Details using *free* Power BI Desktop for Better Data Visualization https://docs.microsoft.com/archive/blogs/kennethteo/separating-tags-in-azure-using-free-power-bi-desktop-for-better-data-visualization Thu, 15 Sep 2016 11:04:33 GMT https://blogs.msdn.microsoft.com/kennethteo/?p=25 The Problem Azure supports tagging of resources for the longest time; However, one of the complaints I have with this feature is: there is no easy way to separate them into their own column when exporting the usage details (see exhibit 1). Tagging can be done at resource group or individual resource. The purpose of tags is loosely defined. Most often than not, an organization uses it to tag for cross-charge purposes. [caption id="" align="alignnone" width="505"] Notice that the Tags column is very messy/unorganized when you have multiple tags.[/caption] The Solution If you have worked with JSON on a project or dueled with it extensively, you may have noticed the Tags column in the screenshot above is actually in JSON format. Enter the Powerful Power BI Desktop... I am not going to write a long-a** essay to tell you what's Power BI, you will be able to find online articles written by experts with better ability to explain it than I am. I'll leave you with this though: Power BI is an online service that transforms your data into rich visuals [blah blah blah]. Power BI also has a free for download desktop client, go ahead download and install it, I'll wait.
        1. Launch Power BI Desktop
        2. Import the usage data, either download from account.windowsazure.com or ea.azure.com and click “edit” in the import wizard (and since I am such a good person, I have attached a sample usage data at the end of this guide for your convenience) [caption id="" align="alignnone" width="634"] Select the Tags column to separate them into their own column[/caption]
          1. Select Tags column
          2. Click on Transform
          3. Click on Parse
        3. Once you have parsed it, click on the small expand icon at the corner in the top cell, like below [caption id="" align="alignnone" width="465"] Expand it[/caption]
          1. Click on the expand button
          [caption id="" align="alignnone" width="936"] Tags separated into their own respective columns[/caption]
        4. That's it!
    If you have any questions, feel free to comment below. [Appendix]
    • The sample usage details file: https://1drv.ms/x/s!AthVHX51HSt1n9UXaFIiywn2SDtsvw
    [Disclaimer] The information in this weblog is provided “AS IS” with no warranties, and confers no rights. This weblog does not represent the thoughts, intentions, plans or strategies of my employer. It is solely my opinion. Inappropriate comments will be deleted at the author’s discretion. All code samples are provided “AS IS” without warranty of any kind, either express or implied, including but not limited to the implied warranties of merchantability and/or fitness for a particular purpose. The inclusion of any link does not imply endorsement by Microsoft of the linked site.]]>