So how has the journey to Android-land been for this dotNet developer?

I've been using the Android Eclipse IDE, which you can download from here. I am also considering using Android Studio: I will likely use Eclipse for my current coding, and then re-do it in Android Studio so I can compare the two. I have heard negative things about Eclipse, and perhaps the fact that there's an on-going replacement project lends credence, but for now I'm being charitable and assuming that most of the problems derive from my own lack of familiarity with Eclipse.

Here's my initial list of good and bad, coming from someone on a steep learning curve, and who "lives" inside Microsoft Visual Studio. This is based on Building Your First App, plus my experience writing several WAMS tutorials.

First the good:

• It works! This was my first stint at writing apps for any kind of phone, and I was amazed at how easy it was to get a "Hello World" app loaded onto my Android phone, as well as to run it in emulator mode. There was no hassle with any kind of App Store, I just coded it and loaded it onto my phone.
• Google's docs are generally pretty good.
• In Eclipse, I like being able to comment out code blocks automagically (and re-commenting them). I'd like to believe this feature is lurking somewhere inside Visual Studio too, but I haven't found it by looking around the VS menus. I probably need to do an internet search to verify this or not.
• I like Eclipse's Package Manager and the ability to click on methods and go to the definition code.
• From my limited standpoint, Android appears on the surface to be well architected and organized.

Now for the "bad stuff" (relatively: I was able to figure this out and nothing permanently blocked me):

• Downloading a USB driver for my Android phone was NOT a seamless process.
• Eclipse has a number of issues for me:
• The relationship between Workspaces and Projects is not clear. Is it analogous to Solutions and Projects in Visual Studio? When I see multiple projects in a Workspace, are they independent of each other, or considered to be pieces of a larger whole? When I am in one Project, I see errors that were generated in a different Project.
• Eclipse doesn't seem to update real time: I had an error in a layout, fixed it, and had to save the file for the error to go away.
• Perspectives are confusing: I used the Debug perspective, wasn't aware that I needed to switch back to the Java perspective (or even that by debugging I had changed perspectives). Somewhere in all of this the Emulator icon vanished off my tool bar. I was able to run the Emulator from the command line, and eventually figured out how to get the icon back.
• The Run command is kind of confusing: when you click it, it looks like nothing happens. I would like a message to tell me either to start up an emulator, or plug in my phone with a USB cable. Or at least some kind of error message. Having nothing happen is mysterious, and leaves it unclear whether it recompiled my latest changes, or what it did... Eventually, the Emulator started coming up consistently, but it was unclear what I did to cause this.
• Having to separately download Java is annoying, and it is confusing where the best path to put it is.
• Emulator issues:
• The best advice when it is starting up is to just WAIT, eventually things work out.
• But initially I had trouble getting the Emulator to update. I did the basic Hello World, then followed up with Building a Simple User Interface, and initially the Hello World app kept appearing in the emulator. But waiting worked, once I quit moving on to something else out of impatience...
• Getting Started with Data throws exceptions when run with my Emulator, but works fine when you connect an actual phone. This probably has something to do with the device that the Emulator is emulating, but the exceptions are pretty non-transparent. It would help if Google would recommend a "vanilla emulator" to use for tutorials.

I'm big on writing really clear tutorials for beginners, and know that this is much harder than it looks, largely because it is hard for an experienced person to put themselves back into the beginner's mind. Here, I'm lucky: I AM that beginner with Android. I find that approaching a new area with a blank mind is a great way to find shortcomings in documentation and UI, and uncover pitfalls that other beginners may fall into. Often when you know a subject deeply, you lose sight of all the assumptions, context, and implicit knowledge that a beginner has to learn. This is as true of Microsoft products as it is of Google's.

Overall, it appears to me that the Android docs don't always succeed in getting down to that beginner level. But this is perhaps my own perfectionism: they were good enough for me to get up and running. And I find my problems also reflected (and solved) on sites like StackOverflow.

I guess the bottom line is, since Eclipse is open source, if something bothers me enough, I should quit whining about it, and jump in and contribute code. Not sure I have the spare cycles to do that though...

If you have read prior posts from me, you may wonder what an enterprise-oriented back end database guy like me is doing messing around with non-Microsoft phones. I have to confess to having always had kind of a technical split personality. I wound up writing the original Accounts Services pages at MSN because I was the only one in our group that knew bits of HTML. And I was very interested in Java, prior to the creation of C#, at a time when there wasn't much scope for that skill inside Microsoft.

I'm also very interested in understanding issues that arise for our open-source oriented customers who want to connect with Windows Azure. Microsoft has put an increasing focus on addressing open source issues in the last 2-3 years or so, it's been an amazing transformation to watch (I will simply confine myself to saying that way back in 1995 when I started contracting here, the attitude towards open source was not a warm one...). Basically I like to get outside the Microsoft culture I've been living in, and try to see things from other perspectives.

In particular, those of us embedded in the Microsoft ecosystem have our ways of doing things with Visual Studio and other tools and languages that don't map automatically to what happens in the open source world. In the Android tutorials, we use open source tools (Eclipse, Google api's, etc), but it is possible that I occasionally do so with a "Visual Studio flavor". We'd like this stuff to be useful to open source developers, so if you are one, and see something that makes you say "well no one I know would do it that way, that's just not how you do things in Eclipse", please let us know! We think our technology is sound, and that it successfully integrates multiple software worlds together, but if something jars you as being "too Microsoft style" or just not the best way to accomplish a task in Eclipse, let me know!

In that overview, I briefly mentioned the bewildering array of choices facing developers today in the data architecture area. But the array of choices is actually much longer than I was able to describe in a brief overview.

A Bewildering Array of Choices

There is not only a wide variety of possible data stores, but there are also a number of choices for physical architecture.

Here is a short list of the possible data stores that could be used in an application:

• A relational database such as Microsoft SQL Server, MySQL, or another relational database product
• Data Warehouses such as SQL Server Analysis Services

• Different kinds of “NoSQL” databases:

• Key/Value stores such as Windows Azure Tables;
• Document Databases;
• Graph Databases;
• Column Stores;

• “Big Data” solutions such as HDInsight (aka Hadoop on Windows), with its many components built on top of Hadoop

• Messaging Solutions:

• Windows Azure Storage Queues
• Windows Azure Service Bus
• SQL Server 2012 StreamInsight for real time “push” event processing
• Windows Azure Blobs

And here is a list of possible choices for physical architecture:

• Cloud services, such as Windows Azure, which include several modes of operation:
• Platform as a Service (“PaaS”)
• Infrastructure as a Service (“IaaS”)
• On premise: in your own data center, which may or may not be a “private cloud”
• Hybrid combinations of the above

The above is only a partial listing. PaaS is somewhat circumscribed by the services offered by the cloud service provider.  So for example, with Windows Azure you have choices like Windows Azure Table Storage, or SQL Azure Database. On the other hand, the IAAS architecture is virtually unlimited in that you are free to install whatever you want on a VM.

In the interests of full disclosure, while this paper addresses architecture in general, the cloud service provider  focus is mainly on Windows Azure.

Another dimension to consider is that for some products, cloud versions and on-premise versions both exist, but the feature sets of the products might not be identical across the two platforms. Two examples of this are Microsoft SQL Server, and HDInsight. For both of these products there is a Windows Azure version, and an on-premise version. Over the long run you can expect the feature sets to converge, but at present you must take feature differences into account.

Why “Everything is Different” with the Cloud and Internet

Two major differences have emerged:

• Data is distributed across many independent nodes that are independent structurally and dispersed geographically; this has led to the “death” of the classical Two Phase Commit method of maintaining database consistency across transactions, and has also led to the emergence of a whole category of non-relational “NoSQL” databases.
• The Internet has made large amounts of data available to applications, in the petabyte range on up, thus the emergence of “Big Data”, and “Internet Scale” applications like Facebook . The data used by these applications are of such size that it cannot be realistically processed with a relational database, which has led to the emergence of “Big Data” solutions, notably Hadoop.

Several consequences flow from the above:

• Geographic dispersion of nodes implies the need to deal with latency. One way this is done is horizontal partitioning of relational databases.
• The large number of nodes means that any “internet scale” application is faced with the inevitability of random node failure at any time, thus leading to the need for redundancy and resistance to the effects of node failure.
• The need to scale out horizontally, the effects of latency, and the requirement for node redundancy all combine to make the use of commodity hardware more cost effective than using the highest performance premium hardware.

Two-Phase Commit and the “CAP” Theorem

In a distributed system, the basic mechanism that enables complex transactions in relational databases, two-phase commit, no longer works well at extreme scaling. This has been summed up by the “CAP Theorem”, which says that in a large distributed system, you can provide only 2 of the following items, and not all 3:

• Consistency
• Availability
• Partition Tolerance

Consistency refers to the internal consistency of a database’s data. Relational database transactions are expected to satisfy the ACID properties:

• Atomicity, meaning that transactions are either “all or nothing": if part of a transaction fails, the entire transaction fails and the database state is rolled back to its initial state.
• Consistency requires that the result of a successful transaction is that the database transitions from one valid state to another.
• Isolation requires that multiple transactions execute in isolation, with no “side effects” dependent on their order.
• Durability requires that once a transaction is committed, it remains so, even in the event of system failures.

With the two-phase commit protocol, you can ensure that your data is always consistent and satisfies the ACID properties. The stereotypical example is moving $10 from your bank’s checking account to the savings account. You first subtract $10 from checking, and next add $10 to savings. If the system goes down, you don’t want $10 to be subtracted from checking only. The “two-phase commit” process ensures that a transaction either completely succeeds, or is discarded.

Availability means that every transaction or request receives a response indicating whether it succeeded or failed.

Partition tolerance means that the overall system is unaffected when any piece of hardware fails, such as a disk, a computer, or an entire rack in the data center. This is ensured by means of massive hardware redundancy. Windows Azure provides at least 3 copies of every resource, and transparently recovers from hardware failures.

The item that has the highest performance cost is “Consistency”, and particularly the two-phase commit process that relational databases use to enforce it. Consequently it is the commonest item to be dropped in large distributed systems, in favor of availability and partition tolerance, since you can only get two out of three. Typically ACID consistency is relaxed to require “eventual consistency”: sooner or later the database is guaranteed to be in a consistent state.

For more information on Distributed Two-Phase Commit and the CAP theorem see:

• Distributed Transactions and Virtualization, which focusses on Windows Azure Service Bus, but is applicable more generally;
• CAP Theorem, in Wikipedia
• Consistency, part of a work on CouchDB, a “NoSQL” database vendor
• Perspectives on the CAP Theorem, for more theoretical depth.

I am writing a White Paper, tentatively titled "Data Architecture in a Cloudy World", (feel free to critique that title) and I'm going to experiment with posting parts of it, and collecting feedback before it is published.

Here's the intro to the paper:

"This paper provides best practice advice for two questions:

• Logical data architecture: which kind of data stores should I use? A relational database and a data warehouse? A NoSQL data store? Big Data? Windows Azure Service Bus? Etc. etc.
• Physical architecture: where to distribute the data stores: on premise? In the cloud? In a hybrid scenario?

Until the last decade, this used to be a straightforward problem: the data store would most likely be a relational database (with some content going into a file system), and the physical architecture would depend on your scaling and availability requirements, involving clusters, replication, hot backups, etc. While these were complex and deep subjects, they were at least well understood, and the best practices were somewhat widely known and uncontroversial.

Today all this has changed. There are many new storage options, driven by the emergence of Internet-scale applications: not only relational databases and data warehouses, but a host of “NoSQL” data stores, “Big Data”, and more. And the rise of the “Cloud” has led to several new, evolving physical architectures.

Choosing the right architecture is a critical decision for your application. In some cases the choice may be self-evident and easy; in other cases you may have to trade off conflicting factors in making your decision.

This paper provides current best practices for choosing both data and physical architectures. Once you have made your decision, there are additional architecture decisions to be made that are specific to a platform, which we will describe, and where available, provide best practices. And we provide links to other resources that go into more detail about each platform than we can do here."

The paper goes through each of the major architectures, and why you might pick one over the other, and for each architecture, goes into the secondary architectural decisions you may need to make. For example, in the case of Windows Azure SQL Database, if you need to scale out horizontally, then you will need to use the Federation feature, and the paper describes some of the design decisions involved in using Federation.

Some of your architectural decisions are well understood: for example, how to build a well-normalized relational database on premise. In such cases, I do not go into detail, and just link to some resources on the subject.

This was a pretty abstract overview of the paper. We've had some indication that the wide variety of data architecture choices available today has caused some customer confusion. I'd love to hear your feedback about whether you've found this with your own customers.

My next post will be the draft of a section tentatively titled Why Everything is Different with the Cloud and the Internet. It's main focus is the "CAP Theorem", how that affects the ability to do 2-phase commits of database transactions, and the architectural implications of that.

Normally we circulate papers inside Microsoft before they are published, but the subject I'm writing about has many experts on the outside as well, so I decided to experiment with soliciting feedback pre-publication. I'm interested in technical accuracy, but also, whether it addresses your questions as a customer, so feel free to wade in even if you don't consider yourself an expert.

An English/American folk proverb says, if you like to eat sausage, you probably don't want to watch how it's made. Well, this will be an experiment to see if that's true of writing also, hopefully not! I've been getting about 1700 hits per blog posting (a few less than Balmer or Sinovsky!), so if only 1% of you make comments, that will be a nice contribution!!

A lot of my topic deals with Windows Azure SQL Database (the database previously known as "SQL Azure"), and especially Federations. In that light, I want to call out a great video of a presentation on Federations by Cihan Biyikoglu at the 2012 US Tech Ed conference in Orlando. Cihan's blog is also a great source of information about Federations.

Federations illustrates an interesting paradox concerning Windows Azure performance. Individual operations in Windows Azure are often slower than the corresponding operations on an in-premises server, for a number of reasons, including latency, and automated fail-over (nodes, including persistent data, are generally replicated a number of times: this takes time, and if a node goes down, and fails over to a secondary node, this also takes time). But this can be more than compensated for by the scaling out of resources, and Federations is an outstanding example of this. The result can be a massive parallelization of work, which can greatly increase performance.

While database scale-out is the commonest example, the same techniques can be applied to other Windows Azure resources.

Another important aspect of Windows Azure performance is the need to test your architectural design by early-on creating a proof of concept application. Windows Azure contains a lot of "moving parts", and it is in a state of rapid evolution and development. Your exact combination of parts has not necessarily been tested by Microsoft (a Computer Science class in Combinatorics will clarify why this is (hint: look at the factorial function...)), and you should validate your architecture early in the development cycle, rather than trying to fix perf problems when about to deploy your application, or even worse, after it's in production (this has of course never happened to me :-)).

Finally, Eric Lippert has a video where he talks briefly about performance as an engineering discipline. Actually most of the video is about C#'s new features and also about Roslyn... But his remarks about performance are worth hearing.

I want to give a shout-out to another topic in the Best Practices series, that was published along with the Performance and Trouble-Shooting ones: Windows Azure Security Guidance.

Finally, as always I welcome feedback! Windows Azure performance is a vast area, and there are undoubtedly areas that could use more detail. And it is a continuously evolving platform, so new issues are likely to appear.

The bottom-line idea of the topic is that because SQL Azure Federation members can have different database schemas, therefore you can use this in order to upgrade your schema incrementally. This can be handy if you have an application with multiple customers, or a multi-tenant application: it enables you to roll out new features selectively, instead of all at once. You might want to do this if your customers vary in how quickly they wish to adopt new features. And it could be a useful way of validating your testing: just go live with a small subset of customers, and the impact of any bugs will be more contained.

I'd love to hear about your experiences using this scenario: is it easy to use? Are there glitches in the process that Microsoft could improve? Our software development gets driven by customer requests and experiences, so let us know what you think.

This topic describes one scenario that comes from being able to have different database schemas in federation members. But perhaps you have discovered a new scenario where you can use this feature. If so, we'd love to hear about it.

As an experiment, this was first released on the Microsoft Technet Wiki, at https://social.technet.microsoft.com/wiki/contents/articles/7894.versioning-sql-azure-schemas-by-using-federations.aspx. As a wiki article, you can easily edit it and make changes to it, unlike with MSDN articles. I'll be monitoring the wiki article for updates and comments, so that I can incorporate them into the MSDN version.

The first step is to use the Translation Widget on the wiki English page to create an automated translation. Next, you hand-correct it, since the translation quality varies widely. I had some initial unclarity over the exact mechanics of making this work: but once I figured it out (I may edit the wiki article on how to use the widget), it went smoothly: really, no different than learning any other new technology.

One bit of advice I was given was to write simple, uncomplicated sentence structures. Since I was translating an existing article, this wasn't an option. I could have rewritten the English version, but you don't know in advance what is going to cause problems, so are likely to do more work than is needed.

A problem in translation is technical terms. For example, a common term in the Entity Framework area is "Plain Old CLR Objects"", and you can just imagine what the Widget did with this: colloquial terms like "plain old" are hard to translate. Here at Microsoft we have a centralized glossary of technical terms and their translations into a variety of languages. Having this available both speeded up my manual translating significantly, and it also ensured consistency. If you don't work at Microsoft, and need to do a significant amount of translating, I advise compiling such a glossary.

Another problem is slang. For example, "good to go"' gets translated literally, word for word. As you likely know, slang is specific to a country or region. As a high school student in Chile, my friends and I produced some hilarious results in both languages because we didn't realize this at first.

The bottom line is that the Widget saves you a whole lot of time. Despite the occasional poor quality of translation, it is much faster than doing it entirely by hand. And if you have a technical glossary available, things go even faster.

 Examples of Translation Problems (Warning: non-English appearing soon)

How good is the automated translation? Here are a few examples of problematic results.

A general problem seems to be the amount of context that the Widget takes into account. It appears to have a narrow window of just a few words surrounding any given word, and so sometimes loses the overall syntax. At times it looks like the Widget has done a word for word translation. But often it does a good, or adequate, job.

Another problem is ambiguity. Here is a sentence from one of the FAQs:

For more details, and a full example of reading entities from a stored procedure, seë:

The word ""reading"" is clearly a verb, however the translator interprets it as an adjective modfiying ""entities"". So it evidently thinks that there are entities, and then there are special ""reading entities"". Here's the translation it produces:

Para obtener más detalles y un ejemplo completo de entidades de lectura desde un procedimiento almacenado, consulte:

The sentence also sounds slightly off, so I corrected the translation to this:

Para obtener información más detallada y un ejemplo completo que explica como leer entidades usando un procedimiento almacenado, consulte:

And then, every so often the widget seems to get very random. Here's a sentence where I can't really deduce what it did.

This is something we considered but in the end rejected

which turns into

Esto es algo que considera pero al final rechazada

For some reason the widget forgot how to conjugate verbs properly, and since the sentence isn't that complex, I can't really guess why.

For some more examples, use the widget on this page to translate this posting into a language you know. That should provide you with a good feel for the quality of what gets produced.

My contribution to this, largely co-authored with James Podgorski from the Windows Azure Customer Advisory Team (who did the really hard work of discovering customer pain points, and coming up with code to solve them), is inside Developing Windows Azure Applications: the section titled SQL Azure and On-Premises SQL Server: Development Differences. The section contains topics about things that are different between SQL Server and SQL Azure: things that you know about developing with SQL Server (including Entity Framework), that work differently in SQL Azure. Or to be blunt about it: some code that you write for SQL Server will not work in SQL Azure without some modifications.

The section includes these topics:

• Handling Transient Connections in Windows Azure: my last two blog posts were about this. Please note that this topic currently has a broken link for the Microsoft Enterprise Library 5.0 Integration Pack for Windows Azure.
• Entity Framework Connections and Federations
• Issues with Transactions and Federated Databases
• Transactions with Entity Framework in Federated Databases

These topics show you the code changes needed to resolve specific problems. We are also researching more general solutions that might require less code on your part.

The section overview, SQL Azure and On-Premises SQL Server: Development Differences, lists two topics as being in the section that are not linked to:

• Unsupported Features in SQL Azure: SQL Server commands that are not implemented in SQL Azure
• Schema Definition Differences: good database schema design principles are slightly different for SQL Azure

These topics will be coming out "real soon now", and the broken link mentioned above will get fixed ASAP.

Are there other obstacles you run into when you take your existing SQL Server knowlege and apply it to SQL Azure? Things that you "know" from developing with SQL Server that "don't work" in SQL Azure? Let me know in the comments to this post, and if they are of general enough interest, I'll get them written up and added. And if you've already solved similar problems in the past, again, let me know.

This framework has 3 components to it:

• a classification scheme for connection exceptions that tells you which ones are worth trying to reconnect, since not all of them are
• a set of strategies for actually doing the re-connect: things like "retry immediately", or, "retry every 5 seconds", etc. The client application has to choose one of these strategies.
• execution: when an exception is encountered, carry out the chosen reconnect strategy.

Two points that might not have been clear from my post:

• this framework can be used for all forms of Windows Azure storage, not just SQL Azure, but also Azure tables, and Azure queues
• it looks to me like this framework could be used for non-Azure SQL Server connections as well: I haven't tried this, but may be verifying it in the next month or two.

To me it looks like this framework, or something similar, ought to be pulled into the .Net Framework. I have no knowledge of any plans in this regards (and even if I knew, I likely couldn't tell you), but it seems like a pretty logical thing to do.

I should probably add the usual disclaimer about these blog posts representing my own opinions and not those of my employer, unless otherwise explicitly stated, so consider it done!

"Transient connections" is a polite way of saying "dropped connections". These are of course a fact of life when developing distributed appilications: a wide variety of network errors can occur. However there are additional causes of dropped connections in Windows Azure, and these additional cases are not all bad things.

Here are the main causes of dropped connections:

• the usual network errors that are inherent in distributed applications
• SQL Azure throttling exceptions: because SQL Azure is a multi-tenant database service based on shared resources, it will throw exceptions on any applications that use "too many" resources. SQL applications that hog resources are generally a bad thing, and can often be a symptom of a badly written query. Dealing with SQL Azure throttling exceptions is pretty similar to applying best practices to any distributed SQL application.
• Windows Azure has extensive capabilities for reliability, load balancing, and scaling: these are in fact some of the major benefits of using Windows Azure, and were high priority design goals for the service. One implication of this architecture is that there are replicated instances of your application, and that sooner or later, one of these instances will get taken down (either by hardware failure, or database sharding, or resource re-balancing). Ideally these actions would be invisible to your application, but at this point in time your application must anticipate connections dropping, and be able to handle reconnecting.
• SQL Azure "sharding" aka the Federations feature: contemplate what happens when your app is executing a query while a partition is being "re-sharded"; or if you are running a multi-partition query while re-sharding happens, and some of the partitions are in the old state, and some in the new state. I will be researching this scenario more in the near future: I am just calling it to your attention to think about, as a source of interesting query behavior.

Here are some resources that I have found useful.

SQL Azure Connection Management: provides a great overview, as well as detailed listings and explanations of the individual exceptions you may encounter.

SQL Azure and Entity Framework Connection Fault Handling: very detailed explanation with code samples of how to handle the exceptions encountered when using the ObjectContext class in Entity Framework. Uses a retry framework described in the next resource.

Best Practices for Handling Transient Conditions in SQL Azure Client Applications: describes a framework that can be used to implement retry logic in SQL Azure; the preceding resource uses this framework.

Transient Fault Handling Framework: the code used by the retry framework in the preceding resource

Minimizing Connection Pool errors in SQL Azure: illustrates how to handle dropped connections

I've particularly found the postings from members of the Windows Azure Customer Advisory Team ("CAT") valuable, since they are based on creating solutions to actual customer problems. Other source of information include Bing, the Windows Azure Platform forums, etc.