One of the great new features the App Insights team has delivered is the ability to export your telemetry data.  Enabling and configuring this is described here:

https://azure.microsoft.com/en-us/documentation/articles/app-insights-export-telemetry/

Once you can get the raw data out of App Insights, you can import into other analysis or visualization tools.  Today I'll talk a little bit about how you might want to do that with the ELK stack.  ELK refers to "Elastic Search, Logstash, Kibana", and is a suite of open source technologies designed to build rapid data visualizations of semi-structured text data.  ElasticSearch is the indexing system, logstash is an agent for pumping data into elasticsearch, and kibana is a dashboard building tool that works well with ElasticSearch.

For this posting, I'll be talking about the "custom event" event type that AI exposes.  A custom event allows you to manually insert telemetry points into your code base.  You choose the event name, and you can attach properties or measures to the event as necessary.  Your mileage may vary if you work with other event types.

AppInsights custom events are exported as Azure storage blobs into the storage account you configure when you enable exporting.  The blob files themselves are organized by application, date, and hour.  Within each per-hour directory,  one or more blob files with semi-random names will be present.  Each blob file will have 1 or more events.  Each event will be a separate JSON object, one event per line of text.

If you'd like to do a one-time download of all of your blobs, doing a recursive copy with AzCopy.exe is a great way to get the data onto a local machine.  A more elegant solution might be to have a worker role that monitors the storage accounts for new blobs, and retrieves them as needed.

Once you've got a blob, you need to insert its contents into ElasticSearch.  You could adapt the code in the article above and have it use ElasticSearch.NET to insert each line of JSON into your ES instance.

Consider the following code:

IElasticsearchClient client = new ElasticsearchClient();
 
 string indexName = "myevents";
 string typeName = "aievent";
 
 foreach(var blob in directoryFullOfBlobs)
 {
 string s = blob.ReadAllText();
 string lowerCaseEventText = "";
 foreach (string eventString in s.Split(new[] { '\r', '\n' }))
 {
 // eventString is a JSON object representing 1 event..
 if (eventString != "")
 {
 lowerCaseEventText = eventString.ToLower();
 client.Index(indexName, typeName, lowerCaseEventText);
 }
 }
 }
 

(Warning: Don't run this code and insert your events into ES just yet.  Read the whole post to see why)

Assuming that you have some enumerable of blob files, this code would read them, split each file up into separate events, transform the entire event into lower case, and then insert the event body into the default ES server, using index "myevents" and object type "aievent".

(The case transformation is optional - in our environment, we've seen some case inconsistencies which break certain reports, so I normalize the JSON to all lower case)

Note that we're not using the "L" part of the ELK stack here - you could probably create a Logstash plugin that does the event insertion instead of writing custom code.  If you come up with a good one, please let me know.

At this point, your AI events should start showing up in kibana.  However, if you start looking at the data, you may notice some things aren't showing up quite the way you like.  For one thing, the locale field of your users is normally in the form of "languagecode-countrycode", with examples being en-US, fr-FR, or en-GB.  The default string tokenizer in ES splits up this field on the dash character.  Kibana draws you a pie chart with values like "en", "GB", and "fr".  This is not what you want.

Similarly, if your event names have any sort of separator characters in them, they'll be tokenized into segments.

Finally, when we inserted these events, we didn't specify an ID for them, so ES auto-generated an id for each event we inserted.  This is nice, but it means that if we re-insert the same event more than once, it will be duplicated in our ES index and subsequently in any reports we build with kibana.

app insights events already have a durable unique ID, so it would be nice if we could tell ES to simply honor the id that's already present.   

It turns out, we can fix all three of these problems by adding an ES mapping to our index before we insert any data into it.  The mapping I present here is one that I've been happy with; you can use it as a starting point for your own ES indexes.  The mapping does 3 things. 

1) it maps the ai internal event ID to be the ES document ID. 

2) it adds a 2nd property for the event name that won't be tokenized.  This lets you report on the full event name

3) it adds a 2nd property for the locale name that won't be tokenized.  This lets you report on the full locale value

To create this mapping, first create the index

curl -XPUT https://localhost:9200/indexname/

the inject the mapping:

curl -XPOST https://localhost:9200/indexname/aievent/_mapping 

{
 "aievent": {
 "_id": {
 "path": "internal.data.id"
 },
 "properties": {
 "context": {
 "properties": {
 "device": {
 "properties": {
 "locale": {
 "type": "multi_field",
 "fields": {
 "locale": { "type": "string" },
 "originallocale": { "type": "string", "index": "not_analyzed" }
 }
 }
 }
 }
 }
 },
 "event": {
 "properties": {
 "name": {
 "type": "multi_field",
 "fields": {
 "name": { "type": "string" },
 "originalname": { "type": "string", "index": "not_analyzed" }
 }
 }
 }
 }
 }
 }
 }
 

This mapping adds "event.originalname" and "locale.originallocale" to the list of properties you can visualize in Kibana.

Note that you can add this mapping after you've inserted your data, but the ID assignment and additional fields will only show up for data you insert after the mapping is created.  It is best to apply the mapping to an empty index and then insert your AI data afterwards.

If you're interested in seeing your AI telemetry data in kibana, and building custom visualizations above and beyond what the AI portal supports today, hopefully this posting gives you a headstart. 

The underlying issue is how Chrome asks for data and how the OData libraries that LightSwitch uses respond to those requests.

Chrome appends the "Accept-Charset" HTTP header onto outgoing requests -- even those that come from the Silverlight plugin that hosts the Ligthswitch client.  Specifically, for en-US and English language users, Chrome specifies the iso-8859-1 charset encoding on all outbound requests.

Other browsers have done this in the past; older versions of IE and Firefox also asked for iso-8859-1.  Chrome still does it, and there is an active bug on Chrome to ask them to stop doing it.

(Modern browsers either do not set this header at all, or set it to the value "utf-8", which has emerged as the de-facto internet standard)

When a request asking for Accept-Charset: iso-8859-1 comes into the LightSwitch server, it dutifully responds by sending a response which has an iso-8859-1 charset encoding.

The problem is that the Silverlight platform, upon which the LightSwitch Silverlight client is based, doesn't support this content encoding.  When the client tries to de-serialize the XML payload, it sees the content-encoding is one that it doesn't support, and it throws a client-side exception.  Thus, the red-x.

Chrome is perfectly capable of dealing with utf-8 formatted data -- in fact, the Accept-Charset header allows you to specify a list of acceptable character sets and the browsers preferences for each.  Chrome actually sends over an Accept-Charset header that specifies iso-8859-1 as 1st choice, and utf-8 as second choice.  So if we can make the server somehow return a utf-8 response, everyone will be happy.

Until the proper fix is available, here is a workaround that can be used to do just that.  We'll use the "URL Rewrite" module in IIS to accomplish this.  Specifically, we'll create an in-bound rule that re-writes all requests so that the "Accept-Charset" header is replaced with "utf-8".  This will make it seem like Chrome asked for utf-8 to begin with.

First, open the IIS administration console and use Web Platform Installer to get URL Rewrite 2.0

After Web PI launches, type "URL Rewrite" in the search box and install it.

Next, navigate to the LightSwitch app you'd like to apply the fix to.  I think you can also apply the URL Rewrite rule at the top level of your IIS server, but that's riskier so I don't recommend it. 

Double click on the "URL Rewrite" icon in the IIS management pane.

The first thing we need to do is enable changing of the Accept-Charset header by URL Rewrite.  To do that, we choose "View Server Variables" and then choose "Add".  The name of the server variable is "HTTP_ACCEPT_CHARSET".  The name must be upper case and must have underscores where I've specified them.

Now that we've allowed rules to manipulate the Accept-Charset header, we can create a rule that does the manipulation.  From the main URL Rewrite screen, choose "Add Rules" and then choose "Blank Rule"

First, give the rule a name like "rewrite all requests as utf-8"

For the Match URL, choose "Regular Expressions" and specify ".*" for the pattern. 

Under the Server-Variables tab, choose "Add" and then specify "HTTP_ACCEPT_CHARSET" as the name of the variable, and set its value to "utf-8".  Make sure "Replace" is checked.

Finally, under the "Action" section, choose "Rewrite".  Specify "{UNENCODED_URL}" as the rewrite target, and MAKE SURE TO UNCHECK "Append Query String".

Apply/Save the rule and test your app with Chrome.  If things aren't working, look at the traces with fiddler.

Once you get the basic re-write rule working, you can experiment with the “Conditions” tab to try and narrow the applicability of the rule.  For instance, you may want to target only requests that specify iso-8859-1 encoding on the request.

private void Customer_Created()

{

  this.LastName = "Smith";

}

To make this happen, as you are working in the Lightswitch design experience, Lightswitch is generating code on your behalf that defines the Customer class and all of its properties.  These objects aren't simple; there are separate concrete implementations of them for client and server, and the magic to make a Customer remote back and forth between Client and Server is all handled for you via code generation. 

Lightswitch gives you a programming experience that is suited towards your specific project.  Your first app will have "Customer", "Person", Address", "Order", whereas your second app may have objects like "Appointment", "Office", and "Holiday". 

But what happens if you want to write some code that works on any entity?  What if you won't know its name until runtime? 

In addition to the strongly typed, code generated programming experience we expect customers to use, Lightswitch also offers a second programming experience, one that is late bound and weakly typed.

Suppose that like above, you have a table named Customer, with a string property named "LastName".  The following code is the weakly typed equivalent to the above code:

private void Customer_Created() {

  this.Details.Properties["LastName"].Value = "Smith";

}

Why might you want to use this?

Consider the case above: suppose I want to write an extension method that works on any entity.  Say that I am writing a logging or auditing framework to include in an LS app.  I want to make any entity in the system "describe" itself. 

        /// <summary>

        /// Given any possible Lightswitch Entity, return a string of key:value pairs that describe its primary key columns and values

        /// </summary>

        /// <param name="entity">The entity instance</param>

        /// <returns>an Id string</returns>

        public static string GetKeyString(Microsoft.LightSwitch.IEntityObject entity)

        {

            string s = "";

            object propVal = null;



            foreach (Microsoft.LightSwitch.Model.IKeyPropertyDefinition kp in entity.Details.GetModel().KeyProperties)

            {

                if (!string.IsNullOrEmpty(s))

                    s += ", ";



                s += kp.Name; s += ":";

                propVal = entity.Details.Properties[kp.Name].Value;

                if (null != propVal)

                    s += propVal.ToString();

                else

                    s += "(null)";



                IEntityPropertyDefinition nameProp = entity.Details.GetModel().Properties.SingleOrDefault(x => x.Name == "Name");

                if (null != nameProp)

                {

                    s += ", Name:"; s += entity.Details.Properties["Name"].Value;

                }

            }

            return s;



        }

There are a few things going on in this code.

1) we're using "IEntityObject".  An IEntityObject can hold any LightSwitch entity.  If a function asks for an IEntityObject, you can pass it a Customer, an Order -- whatever you like.

2) We're using the Model API off of the Details property.  A lot of the tricky "goodies" in the Lightswitch programming experience are available off of the "Details" member of any object.  The Model retreives the metadata of the object instance.  This is how the object describes itself to the Lightswitch runtime.

3) We're using the weakly-typed property access, just like we did above in the "Customer" example.  The difference is that we don't know the name of the property we want to access; we're asking the Lightswitch Model what the names of the properties are.

4) Note that the Model API knows the difference between a property and a Key property.  This is how we can ask any entity what its primary key columns are.  Once we know the appropriate property names, we can use the weakly typed property access to get their values as normal.

Try pasting this code somewhere in your own project.  Send it different types of entities and convince yourself that it will work.

Challenge: Modify this code so that it returns the name of the type of entity, in addition to its key info.  For instance, if I pass this routine a "Customer" instance, and Customer's Id column is named "Id", and the customer I pass in is customer id #7, I would expect the output to be:

"Customer, Id:7"

 If you figure it out, wait a day or two before responding with an answer.

Are there other uses you can think of for this?  Suppose that you have lots phone number properties on a few different entities in your LS app.  Suppose that you want to do phone-number defaulting, for any phone number on any entity in the system.  Imagine a routine called "SetPhoneNumbers", and that you called this routine from the _Created event for each of the entities in your system.

Here's what the logic might look like:

private static void SetPhoneNumber(IEntityObject e) {

 foreach (var v in e.Details.GetModel().Properties.Where(x => x.PropertyType.Name == "PhoneNumber"))

 {

   e.Details.Properties[v.Name].Value = "(800)555-1212";

 }

}

Some quick notes:

1) we're doing a LINQ query against the Properties collection in the Model API. We're asking LS to give us any property object who's type is "PhoneNumber".

2) we're using the C# "var" syntax to avoid caring about what the Lightswitch Property object actually is. You will get full intellisense if you type this code in manually. You won't have to worry about Ligthtswitch's Model.IEntityPropertyDefinition

I think it's pretty cool that in 3 lines of code, given any entity of any type from any LS project, we can set any discovered phone number fields on that entity to some default value.  Or run any other logic we like.

I hope you've enjoyed this introduction to the weakly typed programming experience.  So far, we've only discussed entities and entity properties.  We also used the Model API to get the metadata about a runtime entity instance, like it's properties, key properties, and the data types of its properties.  These simple techniques let developers write re-usable Lightswitch functionality. 

The WCF Binary Inspector for fiddler is a great help when you are working with WCF servies using the SOAP messages that have been binary encoded/compressed according to the MS-NBFS standard. 

As near as I can tell, MS-NBFS is essentially an implementation of MS-NBFX, which covers the actual binary encoding protocol, with a specified shared dictionary.  The NBFX protocol specifies "dictionary lookup" op-codes, both for elements and attributes, but says that both sides of a connection must agree on what the dictionary contents are.  MS-NBFS specifies that dictionary, with SOAP messages in mind, so high-probability strings like "mustUnderstand" are entries in the dictionary and messages get a good effective compression ratio.

The Binary inspector plugin uses the WCF message facility for doing its decoding, however, the message must be a valid SOAP message in WCF Binary format.  I was recently looking at a system where messages were in MS-NBFX, but without SOAP envelopes.  As such, the WCF Binary plugin was blowing up as it was setup to deal with SOAP messages.

It is possible to decode the MS-NBFX directly, by using XmlDictionaryReader.CreateBinaryReader. 

This post has sample code and some other good links.

I've sent over a copy of this code to the author of the fiddler WCF Binary inspector so that Fiddler's MS-NBFS decoder can handle more than just SOAP binary formatted messages.

https://www.nynaeve.net/?p=100

https://www.blackhat.com/presentations/bh-dc-07/Whitehouse/Paper/bh-dc-07-Whitehouse-WP.pdf

The first link talks about how to turn on ASLR for your dlls.  The second is an in depth discussion of the Vista RTM ASLR implementation and some deficiencies from a security perspective.

I'm actually interested in ASLR for an entirely different reason -- performance.

An important scenario for Axapta is hosting multiple instance of the client on a Terminal Server.  Obviously the less memory each instance of Ax32.exe uses, the more clients we'll be able to successfully host on a single machine.  This means customers save money on hardware and licensing.

Most modern operating systems look for ways to eliminate redundancy w.r.t. memory consumption.  One mechanism for doing this is attempting to load a DLL into the same spot in the virtual address space of each process on the system.  The short version goes something like this: notepad.exe and explorer.exe both need "shell32.dll" loaded into their own "private" address space. 

Windows doesn't use fully relocatable code like the linux ELF format is (that is, in a linux module, virtual addresses in machine instructions reference the program counter or some other well-known base address (like the GOT), so you can load that segment of code essentially anywhere in the flat 32bit address space).

Linux: https://en.wikipedia.org/wiki/Position-independent_code

Windows: https://en.wikipedia.org/wiki/Portable_Executable

Windows DLLs specify a "preferred address" in their PE image, telling the loader where, ideally, that code will be loaded.  This is a bit weird - at link time, you specify in the DLL where you'd ideally load your self into a processes address space.

In the event that notepad.exe and explorer.exe can both load shell32.dll into shell32's desired address space, windows can be pretty clever about the backing memory for shell32.  Even though each process gets a distinct and seperate logical virtual address space, the pages backing shell32 in each of them can be backed by the same physical memory.  Essentially, the VM system is only charged "once" for shell32. 

In the event that a DLL cannot load into its preferred spot in memory, the loader must "rebase" the DLL to a new address that is available in the target process.  This rebasing disqualifies the VM pages from being re-used across multiple processes.

Obviously in the case of Ax32.exe on terminal servers; we want to avoid rebasing as much as possible, as each instance of the client with a rebased module will consume more memory than is strictly needed, thus limiting capacity.

This is where ASLR comes in.  ASLR stands for "Address Space Layout Randomization", and its a security feature in Vista.  Read the earlier links on how and why ASLR is used from a security perspective.

The practical upshot of ASLR for us is that the rebasing mechanism has now been moved in-kernel, and is system wide per DLL instead of process-wide.  This means that if a rebase does occur in Ax32, it should be shared across all subsequent copies of AX32, leading to less private memory consumption per instance of the client.

Additionally, because the kernel rebaser doesn't leave gaps between loaded modules, there is less memory  page fragmentation and fewer pages are used in general. 

We're still experimenting with the per-client memory footprint advantages of ASLR, but hopefully they'll be there.  We're hoping to see some per-client memory savings due to ASLR on ASLR enabled operating systems like Windows Server 2008.

 There are two sides of document management... the end-user aspect and the administrator or setup side of it.  Ultimately what you want is to give your end users a better experience; I'll mostly be talking about the sorts of documents you can setup using the administration side of things.

User Overview of Attaching Documents 

To get a quick look at doc handling in Axapta, open up AR->Customers.  Next to the big "bell" icon (the Alerts button, if you're using 4.x) there will be an icon that looks like a document.  If you mouse-hover (in 4.x, at least) over the button it will say "Document handling"

If you click this button you get the Document Handling form (DocuView). 

DocuView is an app-singleton form.  That means that you can only have one instance of it open at a time.  Since you open DocuView from some other form, you are operating on a linked data source (that is, the DocuView form knows who called it, and restricts its data to the context-appropriate records).  In our case, you'll see the documents attached to whatever customer was selected when you hit the doc handling button.

If you're new to doc handling, you'll probably have an empty grid.    Off on the right you'll see a "New" button.  Axapta has some in-built document types; the most commonly used one is "Note".  Select New->Note.  You'll be able to enter a description, and then set the flag "Internal" or "External".  Finally, in the window at the bottom of the DocuView form you'll be able to enter free form text.

hit ctrl-S to save the note.  You've now attached a document to this customer.

Administrator view of Setting up Document Types 

Now, you may have noticed other options available to you under New->.  On my configuration, one of them is "Letter". 

Axapta has the ability to create Word documents for you with data pre-populated.  Here we begin to talk about the Setup side of things.

In Basic->Setup->Document Management->Document Types, you'll find the configuration of the document types available in Doc handling. 

You can create new records here.  Give your new document type a name like "CustLetter".  The "Job description" property is most interesting.  Here we'll choose from the drop down "Create Word document via COM".

Switch over to the general tab and make sure to set the "Archive directory".  Then click "Options" on the right.

Mapping Axapta Table Fields to Document Template Bookmarks

The new form that opens is how you bind Axapta data to the document type.  Here you'll define what data to export into the Word document, and how it will be mapped into the document.

Hit Ctrl-N to setup a new table mapping.  Choose the table you'd like to pull data from (CustTable is a fine choice). 

Here we're going to use a Word Template. 

Suppose I've got a word document that is a nice form letter I'd like to send customers.  It has my company logo on it, some standard legal text, my return address, and so on.  I'd like to auto-populate regions of this letter with data like the customers name and address, and then store the resultant letter in the doc management system.

Here's how we can use Doc handling to automate this (to a degree).

Create a word document that looks like you'd like the letter to appear.  For places where you'd like the customers address or name, you'll insert word Bookmarks.  In Word 2007, you can insert a Bookmark right from the ribbon bar... Insert->Links->Bookmark.  Move the cursor in the document to the position you'd like to insert the customer name, and then insert a bookmark.  Name the bookmark "CustName". 

You'll want to save this file as a word template, by going to File->Save As, and changing the type.  The template lets you control content and formatting, and is used as a starting point from which to generate new word documents that have the bookmarks filled in with real customer data.

Back in AX on the Setup form, edit the "Template File" column to point at your newly saved template.

Now flip over to the "fields" tab on this form.  Here you can hit Ctrl-N to add new Field->Bookmark mappings.  Create a mapping from data field "name" on table "CustTable", to the bookmark "CustName".  Save your configuration and then save/click OK to all of the setup forms you've got open. 

Now we're ready to try it out!

If you open up a customer, click doc handling, and then click "New->CustLetter".  Word should fire up, show you a new document named according to the configured number sequence, and the document should have the content of the template you created earlier, with the customer's name filled in where you defined the bookmark.  Save the document and you're done!

Wrap Up

Hopefully you were able to setup a new document type that exported table data to a word doc based on a template without too much trouble.  Templates can be reasonably complex, and data can come from multiple tables, but there are some limitations. 

Naturally, this doesn't have the "media" keys of the recently awful MS keyboards.  So the common scenario is that a person comes over to talk to me and i have to find media player, restore it, hit pause, then take off my headphones, then turn to talk to the person.

What i wanted was a global "pause" button.

It turns out that somebody has made a tiny (12kb) program that lets you bind arbitrary key sequences (context-free, it seems) to arbitrary actions.  This thing fits the bill exactly, and is available from https://dana.ucc.nau.edu/~tsr22/apps/hoekey.htm.  It has stuff built-in to send messages to whatever app, so i can bind windowskey-P to send the play/pause command to WMP.  Life is good.

As an aside, the default config includes "windowskey-4" as a command which..turns the currently selected window to half alpha-transparency.

In any case, the thing is tiny.  The display is beautiful, the interface works how you'd expect.  It has the ability to store vCards and other cool stuff.

So my co-worker asked "do you need iTunes to sync to it?".  I didn't know.  My friend figured you didn't, and that it just showed up as a mass storage device.  Sweet.  But before I told my coworker about this, i figured i had better look into it myself.

So later this weekend i hooked the thing into my XP SP2 machine via USB cable.  It showed up as a disk, and when I went to explore that disk, it had folders for eTexts, vCards, and calender appointments.  Nothing about music.  There is a hidden folder called something like "itunes data" and if you explore it, you find a few index files and album art collections and other stuff, and eventually you get to a folder that contains what appear to be the song files - they have .mp3 and .aac (or whatever they are) extensions on them.

The problem is that these files all have 4 letter names.. there's no effective way to look through this folder and figure out what the songs are.  The containing folder heirarchy is also broken down into arbitrary folders.. something like M1 through M6 on this particular unit. 

The ipod apparently has a media database that it uses to let you navigate its menus, and as part of putting songs on it, you're updating this database, and during that time, turning the proper filenames into hash names and then recording those hashnames in the database.

While i can understand the design decision to have an offboard index calculator (itunes), it stinks that getting media into or out of this thing is more difficult than it needs to be.

I can think of two or three people that would be more interested in the iPod Nano if they didn't need to use iTunes to deal with it.  Not that Apple should worry about what 2 or 3 people want, but I suspect there are more pro-Ipod, anti-iTunes people out there.

This can leave you with hundreds of databases that you'll never use again, and there's no reason for them to continue taking up space and cluttering things which you do not wish to be cluttered.

Now, you could use a GUI tool like enterprise manager, and right click on every database and choose "Delete"

Obviously, that stinks.

Alternatively, you could use isqlw (SQL Query Analyzer) and enter the appropriate T-SQL statement to delete a database:

drop database Foo 

go 

This also stinks, because thats a lot of typing, once you do it for 100 databases

Fortuneately, you can do better. What you've got, fundamentally, is a problem that computers are good at. You've got a lot of somethings (databases) that you want to do something to. You know what you want to do - drop a database. Now you just need to leverage that across a big list of them.

If one wanted to get a list of data in SQL, one would write a a SELECT statement. It turns out that if we're willing to look at the system tables, sql server will tell us about the databases installed in this instance.

select name 

from sysdatabases 

where sid <> 0x01 

and name like 'HW_%' 

Now, this will give you the name of any databases from this instance which match "HW_%" and which have an sid other than 0x01. The sid <> 0x01 clause is interesting. On the SQL instances I'm looking at, the "system" databases always have sid=0x01. But on one other instance, Northwind also has sid=0x01. User databases always have a long gooofy, non-0x01 value for sid. So you can use this as a hack to do a "reasonable" job of excluding "system" databases from your select list. If you can make a determination based on the dbname, as I have above, you can do even better.

Great, we can get SQL server to tell us the list of user databases installed from this instance. What does that buy me? Well, suppose we change the select list:

select 'drop database ', name 

from sysdatabases 

where sid <> 0x01 

and name like 'HW_%' 

If you run this select statement (and you have databases that match the select list) you'll get the following output (at least on my machine - you probably won't get this output as your dbnames will be different

drop database HW_North 

drop database HW_South 

drop database HW_Enterprise 

Congratulations, you've just told SQL server to generate SQL for you. Obviously, if you had 100 databases in your select list, you'd get 100 statements generated. Here's where the hack comes in, i don't know of a way to actually execute this in one step, but I bet it is possible. Instead, I copy and paste the results (results to grid works fine in ISQLW, results as text will give you a giant column width for the name column), back into a command window, and then execute the batch.

This technique is handy for dropping lots of databases, but its also good in other places. For instance, you can generate INSERT or UPDATE statements this way, which may be desirable if you've got just one or two peices that make doing the job with a single insert/update too difficult.

It is not elegant, not efficient, and probably upsets somebody somewhere, but for a quick and dirty management task, using select literals to generate statements for you is a great technique to keep in mind.

Let's say, for the sake of argument, that I'm going to use System.DirectoryServices.DirectoryEntry to represent links to external directory objects. 

I need to persist the relationship between a DE and my concept of a User. 

The way I've chosen to do this is by taking the Path Property  of the DirectoryEntry and saving that, as a string, into the database.

When I need to re-hydrate my user Account object, I'll back the DirectoryEntry property by using the DirectoryEntry constructor which takes the Path as its argument.

Now, this is all well and good, except what it means is that anytime anyone loads an instance of my User Account, and looks at the DirectoryEntry property (say, a user management application), they're going to be running code that I supplied data to.  Namely, the constructor code for the DE class.

We're now in a situation where we're running code, and passing it an argument from an untrusted source. 

DE is currently a shim on top of the COM based ADSI architecture.  ADSI is a scheme with multiple, pluggable providers.  The DE Path specifies the provider to use.

This is nice from a programming perspective - if you want to get an Active Directory object corresponding to a user, you might try:

de = new DirectoryEntry(”LDAP://CN=mattev,DC=microsoft,DC=com”);

alternatively, if you had an NT4 provider:

de = new DirectoryEntry(”WinNT://localhost/administrator”);

You've got the same .NET DirectoryEntry object, and the same ADSI COM interface, but underneath you've got an LDAP provider handling one LDAP object, and a WinNT provider handling the other.

Now, it turns out that there's a 3rd “standard” provider -the IIS:// provider.  This provider lets you use ADSI to script commands against the IIS metabase.  The IIS ADSI interface is extensive - you can create virtual directories, delete them, delete metabase backup copies, and so on.  Essentially any IIS admin task can be performed via the IIS ADSI interface.

Note that any of these IIS:// type paths are perfectly valid DirectoryEntry paths - Infact, only the IIS provider can say if an IIS:// path isn't valid. 

So I think you can see where this is going.  Suppose hypothetically, user “Evil” is running our user management system.  Evil, for whatever reason, can update the DirectoryEntry property of one of our user objects.  Say that Evil does the following:

user.DirectoryEntry = new DirectoryEntry(”IIS://importantserver/W3SVC/1/ROOT/Important/deleteApplication”);

Note that I'm not familiar enough with the IIS provider to actually know what the IIS path would be to perform this action, so play along :)

Now, because Evil doesn't have any sort of permissions to administer https://importantserver , this property assignment may fail or it may not - it depends on the ADSI provider, frankly.

If the property assignment silently fails, then we persist the above ADSI path into our database.

Now user “Admin” comes along and runs the user management application.  If the user admin app displays the directoryentries associated with a given user, the DirectoryEntry constructor will be run on the path saved in the database.

Admin does have permission to administrate https://importantserver.  And just the process of the application running the DirectoryEntry constructor is enough to to destroy the IIS application that Evil was targeting. 

In reality, there was at least one thing that kept this attack from working.  When Evil tried to set the DirectoryEntryPath property, we checked the Schema type of the object indicated by the path.  There's nothing in the IIS ADSI space that looks anything like a user or group schema.  This marginal validation was just enough to keep the IIS:// path from being a valid DirectoryEntry  in the eyes of the application, so it was not persisted. 

Secondly, it seems likely that the IIS provider, given a path with a verb, would return a failure code if the verb failed, and thus the DirectoryEntry constructor would throw an exception if its underlying COM call returned the error. 

Finally, it's not clear that you can actually construct IIS:// paths that are verbs.