To those of you who are frequent readers, listeners and commenters, THANKS YOU! I had a chance to work with many of you over the years and learned a great deal about your experiences with these products. Thanks to everyone who read and reached out through twitter, this blog or walked into one of my session at conferences around the world, for your passion for technology, friendship, patience and your hard work. I truly enjoyed our interactions and I hope our roads cross again!

Twitter: @cihangirb
Linkedin: https://www.linkedin.com/in/cihangirb

Thanks

-cihan biyikoglu

Want to read more… Here is the link.

https://blogs.msdn.com/b/windowsazure/archive/2012/05/24/windows-azure-isv-blog-series-full-scale-180-tackles-database-scaling-with-windows-azure.aspx

Enjoy.

-cihan

https://technet.microsoft.com/en-us/edge/inside-sql-azure-self-governing-self-healing-massively-scalable-database-service-in-the-cloud

Follow @cihangirb

Open field; fire away and leave a comment or simply tweet #sqlfederations and tell us what you'd like us to work on next...

Thanks

-cihan

https://odatafederation.codeplex.com/

Not every tool in the world natively understand federations today. OData simplifies life for a lot of the tools that can’t speak ‘sqlfederations’ yet and the list of OData consumers is full with great names like Excel! be able to query your federated data using Excel now that should appeal to many!

You can find the full list here; https://www.odata.org/consumers

Browsers IE, Chrome, Safari etc; Most modern browsers allow you to browse Atom based feeds. Simply point your browser at one of the OData Producers.

Excel 2010: PowerPivot for Excel 2010 is a plugin to Excel 2010 that has OData support built-in.

Client Libraries: Client libraries are programming libraries that make it easy to consume OData services. We already have libraries that target: Javascript, PHP, Java, Windows Phone 7 Series, iPhone (Objective C) and .NET. For a complete list visit the OData SDK.

OData Helper for WebMatrix: The OData Helper for WebMatrix and ASP.NET Web Pages allows you to easily retrieve and update data from any service that exposes its data using the OData Protocol.

Tableau: Tableau - an excellent client-side analytics tool - can now consume OData feeds

Telerik RadGrid for ASP.NET Ajax: RadGrid for ASP.NET Ajax supports automatic client-side databinding for OData services, even at remote URLs (through JSONP), where you get automatic binding, paging, filtering and sorting of the data with Telerik Ajax Grid.

Telerik RadControls for Silverlight and WPF: Being built on a naturally rich UI technology, the Telerik Silverlight and WPF controls will display the data in nifty styles and custom-tailored filters. Hierarchy, sorting, filtering, grouping, etc. are performed directly on the service with no extra development effort.

Telerik Reporting: Telerik Reporting can connect and consume an existing OData feed with the help of WCF Data Services.

Database .NET v3: Database .NET v3 - A free, easy-to-use and intuitive database management tool, supports OData

Pebble Reports: Pebble Reports lets you create reports consisting of tables, charts and crosstabs. Reports can be exported to Word, Excel and PDF formats, or emailed via Outlook.

This operation would move the data back into a single member that would cover all federated data in 0..100. MERGE AT would also allow transactions to still continue to happen on the 2 source members: 0..50 and 50..100, while the system rearranges the data back into a single member and would not require any downtime. Some do ask why we did not implement MERGE? Sure, we could have waited until MERGE is implemented to light up federations but many customers told us that they won’t need MERGE right away and they do need SPLIT AT badly. Given the original pricing model in SQL Azure MERGE also didn’t make sense. Not so any more… The new pricing model explained here, gives you benefits when you combine members into larger sizes. So what can you do today, until we get MERGE done? Big thanks to George Huey (@gihuey), here is a very timely article that talks about how to implement MERGE yourself using SQL Azure Migration Wizard to take advantage of the new pricing model and pay less. There are a few limitations; it isn’t a single simple high level command like ALTER FEDERATION and it does require downtime for your app so not online but it can be done! Take a look under the section titled “scaling down” for details.

https://msdn.microsoft.com/en-us/magazine/hh848258.aspx

#1 - New 100MB Option: Smaller databases has been a big ask. If you do not have a multi-tenant database tier, this may be the way to go given you can have many tenants. The story is told better with this picture. This is just going up to database capacity going up to 10GB. Look at that bottom left corner. that is the 0.1GB database.

#2 - Granular Increments: The new model introduces 1GB Increments as opposed to the original jump points : 1-5-10-20-30-40-50-150…GB. The picture looks so different that I think it is easier to begin telling the story with the only that didn’t change: 1GB database capacity. That is still $9.99 but everything else simply improved for customers. Not only do we have granular increments but we no longer have business edition starting at 10GB. You can have a 100MB business edition database and still pay 4.99 a month instead of 99.99. All these sound fantastic to all of you but the engineering team is in tears – just like that VWJetta commercial. Just to help you appreciate the price difference, here is the full picture to show you savings all the way up to 150GB databases. We have saving over %78 for some of the segments.

One important point to remember is; for this price, you are not only getting database storage for this price! Beyond just storage in database, the simplified price measurement on database-GBs in SQL Azure also include rich TSQL language with stored procedures for high performance query processing as well as transactional processing, the built in high availability and worry free physical administration you get with SQL Azure

For more information here are a few useful links;

Pricing Federations in SQL Azure: https://blogs.msdn.com/b/cbiyikoglu/archive/2011/12/12/billing-model-for-federations-in-sql-azure-explained.aspx

Price Calculator: https://www.windowsazure.com/en-us/pricing/calculator/

Pricing Details: https://www.windowsazure.com/en-us/pricing/details/

Offers: https://www.windowsazure.com/en-us/pricing/purchase-options/

Pay by Invoice option: https://www.windowsazure.com/en-us/pricing/invoicing/

Thanks!

We looked at queries examples in the previous post with member queries with no summary queries, that is the member queries were simply UNIONed. In this post, we’ll take a look at Summary queries. when are they needed? what are some common patterns and examples?

Summary queries are required for post processing the member queries. Simply put, summary queries can help reshape the unioned member results into the desired final share. Summary queries refer to object generated by the member queries and depending on the implementation can be executed on the client side or the server side. That said, today Federations do not provide a built in server side processing option for summary queries. Here are a few options for processing summary queries:

- LINQ To DataSets offers a great option for querying datasets. Some examples here. LINQ is best suited for the job in my opinion with flexible language constructs, dynamic execution and parallelism options.

- ADO.Net Expressions in DataSets offers a number of options for summary query processing as well. For example, DataColumn.Expressions allow you to add aggregate expressions to your Dataset for this type of processing. Or you can use DataTable.Compute for processing a rollup value.

- Obviously server-side full fan-out processing is also an option. This option refers to server side running member and summary query in a single round-trip from the client to SQL Azure. However as of Jan 2012, this is not built into SQL Azure Federations. We’ll take a look at a simulated version of this in the sample tool here; You can use the deployment closer to your database for efficiency;

Americas Deployment: https://federationsutility-scus.cloudapp.net/
European Deployment: https://federationsutility-weu.cloudapp.net/
Asian Deployment: https://federationsutility-seasia.cloudapp.net/ 

The tool provides a basic and full ‘fanout’ page. Basic page contains only member queries through a simplified interface. The results are simply unioned (or ‘union all’-ed to be precise) together. Full page provides additional capabilities including member and summary queries. Both member and summary queries is expressed in TSQL. The full page also allows for parallelism and allows specifying a federation key range other than all members. Tool has a help page with detailed notes on each of these capabilities.

Lets dive in and take a look at where summary queries can be useful. We’ll start with Ordering, TOP, aggregations and finally DISTINCT processing. By the way, for the examples in this post, I’ll continue to use the BlogsRUs_DB schema posted at the bottom of this article.

GROUP BY and HAVING with Fan-out Queries

With simple group-by items the rule is simple; if the grouping is aligned to the federation key in fan-out queries in federations, processing of group-by and having needs no special consideration. Simply union the results (or union-all to be precise) and we are done. However processing unaligned groupings (any grouping that does not include the federation key) requires a summary query.

When grouping isn’t aligned to federation key, grouping isn’t completely done with member queries. that means processing HAVING will generate incorrect results in member queries. Lets take the example form the previous post; here is the query simply reporting the months and counts of posts with more than a million posts;

SELECT DATEPART(mm, be.created_date) mon, COUNT(be.blog_entry_title) cnt
FROM blog_entries_tbl be
GROUP BY DATEPART(mm, be.created_date)
HAVING COUNT(be.blog_entry_title) > 100000

The grouping on month in each member yields the results from each member but grouping all members isn’t fully done yet! We need to GROUP_BY the same columns and expressions to finish grouping before we can apply the HAVING predicate.

So to correctly process HAVING predicate, we need to push t to the summary query;

SELECT DATEPART(mm, be.created_date) mon, COUNT(be.blog_entry_title) cnt
FROM blog_entries_tbl be
GROUP BY DATEPART(mm, be.created_date)

and the summary query should process the HAVING.

SELECT mon, sum(cnt) FROM #Table
GROUP BY mon
HAVING sum(cnt) > 100000

‘ORDER BY’ and ‘TOP’ with Fan-out Queries:

Lets start with ordering and TOP functionality in TSQL. The member queries can include ORDER BY and TOP but you will need the summary query to reprocess and finalize the ordering and top filtering of the member query results. Take the following example; The query is calculating the top 10 blogs created across the entire BlogsRUs_DB. Remember the blog_entries_tbl is federated on blog_id.

SELECT TOP 10 blog_entry_text FROM blog_entries_tbl
ORDER BY created_date DESC

Here is how to break this into a fan-out query with a member and a summary query; The #Table in the summary query refer to the resultset generated from the member query.

Member Query:
SELECT TOP 10 blog_entry_text, created_date FROM blog_entries_tbl
ORDER BY created_date DESC
Summary Query:
SELECT TOP 10 blog_entry_text FROM #Table
ORDER BY created_date DESC

Here is the output for the member query:

And the output with the summary query. Now the output only contains the true TOP 10 rows.

Additive Aggregates: MIN, MAX & SUM with Fan-out Queries:

When the processing of the MIN, MAX and SUM align with a grouping on the federation key, a UNION ALL of the results can simply yield the result. For example the count of blog entries per blog could be proceed with the following query;

SELECT blog_id, COUNT(blog_entry_id)
FROM blog_entries_tbl GROUP BY blog_id

Same applies to the OVER clause used for defining windows when processing aggregates. that is, as long as the PARTITION BY includes the federation key, a UNION ALL of the results is sufficient for the summary query.

However when the grouping does not align to the federation key, you will need summary queries. Here is the example that gets us the latest date that a blog entry was created. The grouping in the absence of a GROUP BY clause is a single bucket.

SELECT MAX(created_date) FROM blog_entries_tbl

Here is the way to break this down into a fan-out query;

Member Query:
SELECT MAX(created_date) FROM blog_entries_tbl
Summary Query:
SELECT MAX(Column1) FROM #Table

Fan-out Queries Processing AVG and COUNT

Average is a none additive aggregate thus it takes a rewrite to take an average when grouping does not align to the federation key. Here is an example of an average aggregate that aligns with a federation key: the next query calculates the average days between the blog entry and the last comment for the post for bloggers. the inner query gets the MAX date for the last comment on the blog entry. AVG is calculated per blogger for all the blog entries with the days between the create date of the blog and the last comment date on all blog entry.

There is no need for a summary query other than a UNION ALL of the results when fanning this query out given it is aligned to the federation key: blog_id.

However when the grouping is not aligned, there is work to do! Average and Count are not additive. That is, one cannot take 2 averages and average those to get the correct average or take 2 counts and use another count to calculate the output of 2 counts; we need SUM in the case of 2 counts to correctly get the final correct count and we need to use sum of values we want to average and the item count to calculate avg correctly from multiple member queries. Here is an example: Lets take the average length of blog entries across all each month. This time the grouping is on the month of the blog posts.

SELECT DATEPART(mm,be.created_date) month_of_entry,
       AVG(LEN(blog_entry_text))
FROM blog_entries_tbl
GROUP BY DATEPART(mm,be.created_date)

Lets use SUM and COUNT to get to the correct average.

Member Query:
SELECT DATEPART(mm,created_date) month_of_entry, 
       SUM(LEN(blog_entry_text)) sum_len_blog_entry, 
       COUNT(blog_entry_text) count_blog_entry
FROM blog_entries_tbl
GROUP BY DATEPART(mm,created_date)

Summary Query:
SELECT month_of_entry,
       SUM(sum_len_blog_entry)/SUM(count_blog_entry) avg_len_blog_entry
FROM #Table
GROUP BY month_of_entry

‘DISTINCT’ with Fan-out Queries:

DISTINCT is fairly easy to calculate much like the other additive aggregates. As long as the grouping is on the federation key. When grouping isn’t aligned to the federation key, a summary query reapplying the distinct for de-duplication is needed. Here is the query for DISTINCT count of languages used for blog comments across our entire dataset;

SELECT DISTINCT bec.language_id
FROM blog_entry_comments_tbl bec JOIN language_code_tbl lc
     ON bec.language_id=lc.language_id

To break this apart, you have to run distinct in each member and then rerun the distinct on the summary query much like MIN and MAX;

Member Query:
SELECT DISTINCT bec.language_id
FROM blog_entry_comments_tbl bec JOIN language_code_tbl lc
     ON bec.language_id=lc.language_id
Summary Query:
SELECT DISTINCT language_id FROM #Table

Now lets take a look at some more challenging none additive aggregates.

Fan-out Queries with TON N PERCENT and DISTINCT COUNT

TOP N PERCENT is not as popular as TOP N but for those who use it, lets dissect what you need to do with fanout queries. TOP N PERCENT simply require you collect all results without TOP from members and only apply the TOP N PERCENT in the summary query. Imagine the same query we used for TOP with TOP 10 PERCENT. Query gets latest 10% blog entries.

Here are the member and summary queries.

Member Query:
SELECT blog_entry_text FROM blog_entries_tbl
ORDER BY created_date DESC
Summary Query:
SELECT TOP 10 PERCENT blog_entry_text FROM #Table
ORDER BY create_date DESC

Alarm bell should go off whenever we are not able to push the filtration predicate (TOP clause) down to the member queries. So processing TOP N PERCENT is costlier that TOP N and additive keywords.

DISTINCT COUNT calculation is trivial if grouping is on the federation key. However when grouping isn’t aligned, it takes more work to calculate distinct count given that it isn’t additive. That means; you cannot simply add distinct-counts from each member given you may not know if you are counting certain things multiple times without full de-duplication of all the items. Here is an example; this query get the distinct count of languages per month across the whole resultset;

We need to centralized processing so all comments language ids across days first need to be grouped across all members and then distinct can be calculated on that resultset. Here is the member and summary query for calculating distinct-count;

Member Query:
SELECT DISTINCT DATEPART(mm,created_date), bec.language_id
FROM blog_entry_comments_tbl bec JOIN language_code_tbl lc
ON bec.language_id=lc.language_id

Summary Query:
SELECT Column1, COUNT(DISTINCT language_id)
FROM #Table
GROUP BY Column1

Here is what the member query output looks like;

Here is the output with the summary query;

In conclusion, additive operations like MIN, MAX, COUNT or SUM and predicates like TOP or order by can easily be processed with fan-outs using simple modifications to original queries to break them into member and summary queries. With additive queries big advantage is that you can push these operation to the member queries for efficiencies. In cases where you need to deal with none-additive operations like average, there is some rewrite to help push filtrations and aggregate predicates to member queries. However there are a few cases like TOP N PERCENT or DISTINCT COUNT, where fan-out queries may require larger dataset shoveling thus will be more expensive to calculate because you cannot push these predicates to member queries and can only process them in the summary queries.

Happy fan-out querying.

Follow @cihangirb

*Sample Schema

Here is the schema I used for the sample queries above.

-- Connect to BlogsRUs_DB
CREATE FEDERATION Blogs_Federation(id bigint RANGE)
GO
USE FEDERATION blogs_federation (id=-1) WITH RESET, FILTERING=OFF
GO
CREATE TABLE blogs_tbl(
blog_id bigint not null,
user_id bigint not null,
blog_title nchar(256) not null,
created_date datetimeoffset not null DEFAULT getdate(),
updated_date datetimeoffset not null DEFAULT getdate(),
language_id bigint not null default 1,
primary key (blog_id)
)
FEDERATED ON (id=blog_id)
GO
CREATE TABLE blog_entries_tbl(
blog_id bigint not null,
blog_entry_id bigint not null,
blog_entry_title nchar(256) not null,
blog_entry_text nchar(2000) not null,
created_date datetimeoffset not null DEFAULT getdate(),
updated_date datetimeoffset not null DEFAULT getdate(),
language_id bigint not null default 1,
blog_style bigint null,
primary key (blog_entry_id,blog_id)
)
FEDERATED ON (id=blog_id)
GO
CREATE TABLE blog_entry_comments_tbl(
blog_id bigint not null,
blog_entry_id bigint not null,
blog_comment_id bigint not null,
blog_comment_title nchar(256) not null,
blog_comment_text nchar(2000) not null,
user_id bigint not null,
created_date datetimeoffset not null DEFAULT getdate(),
updated_date datetimeoffset not null DEFAULT getdate(),
language_id bigint not null default 1
primary key (blog_comment_id,blog_entry_id,blog_id)
)
FEDERATED ON (id=blog_id)
GO
CREATE TABLE language_code_tbl(
language_id bigint primary key,
name nchar(256) not null,
code nchar(256) not null
)
GO

• Member query is the piece that is sent over to all members involved in the query and
• Summary query is the query that is the post processing piece to allow condensing the results from the member query to desired final result-set.

Americas Deployment: https://federationsutility-scus.cloudapp.net/
European Deployment: https://federationsutility-weu.cloudapp.net/
Asian Deployment: https://federationsutility-seasia.cloudapp.net/

The basic fan-out query utility page shows you an example of a tool that can do member query. We will cover other parts of the tool in future posts but member queries are dead simple so wanted to show this for those of you who are looking for ways to deploy schema, maintain ref data etc. Simply go under the Home Menu and select the Fan-out Query Utility to access the page; The tool simply take a connection string, federation name a query and executes the query (or batch) in all federation members. You can also get to the source code for the page through the link. the member query is executed with the button click event for the “Submit Fanout Query”. Lets quickly walk through the code first;

• First the connection is opened and three federation properties are initialized for constructing the USE FEDERATION statement: federation name, federation key name and the minimum value to take us to the first member.

• In the loop, the app constructs and executes the USE FEDERATION routing statement using the above three properties and through each iteration connect to the next member in line.

• Once the connection to the member is established to the member, query is executed through the DataAdapter.Fill method. The great thing about the DataAdapter.Fill method is that it automatically appends the rows or merges the rows into the DataSet.DataTables so as we iterate over the members, there is no additional work to do in the DataSet.

• Once the execution of the query in the current member is complete, app grabs the range_high of the current federation member. Using this value, in the next iteration the app navigates to the next federation member. The value is discovered through “select cast(range_high as nvarchar) from sys.federation_member_distributions”

• The condition for the loop is defined at line#71. Simply expresses looping until the range_high value returns NULL.

*Sample Schema

Here is Dan’s question:

Dan - Tue, Dec 13 2011 11:57 AM

Cihan, thanks for keeping us all posted all the way until the D-day the 12th.

Having followed your posts and some Azure documentation (so far limited), there is 1 simple but key question that keeps bugging me, my team and quite likely many other developers - so far I have not found any reasonable explanation and I fail to understand on HOW and WHY having ALL(!) federation member requests go ALWAYS(!) through ONE(!) Federation Root DB would eliminate the SQL performance bottleneck problem that we are trying to get away with???

…

This is a common confusion I think we create with the root database but root, besides holding onto some metadata, really does not do the processing of transactions when it comes to federations. In fact, unless you have specific processing requests that happen in the root, root stays pretty idle. This is the magic of the USE FEDERATION statement.

Before I go into the detail of USE FEDERATION, let me dive into some architectural details of SQL Azure. SQL Azure physically has a separate layer in front of the database nodes called the services layer (a.k.a the gateway). Services layer owns managing connectivity, translation of logical to physical server names, some TSQL processing and a few other services. Services layer does some complicated TSQL processing; In fact processing of USE FEDERATION statement and all other CREATE/ALTER/DROP FEDERATION statements are done by the services layer not the platform layer (or the database nodes) in SQL Azure. Services layer is a scaled out tier and SQL Azure configuration controls the number of gateway nodes per cluster in SQL Azure. You can see the detailed picture of the layers below. You can find more details on SQL Azure architecture here.

Connections from your applications are distributed over to the services layer nodes through a load balancer. When a connection from a client application comes in, one of the services layer nodes handle the connection, translate which physical server now contains the logical server and database name in the connection string and establishes the connection to the database node in the platform layer.

Ok, so lets go back to federations and how federation do their processing: With federations, applications use the name of the root database in their connection string. At this point, a connection from the app to the services layer node is established (white arrows) and a connection from the services layer node to the root database node is established or if a pooled connection already exists to the root database it is reused (blue arrow).

In federation model, before you do transactional processing we require that you issue a routing statement called USE FEDERATION to work with your scaled-out schema. USE FEDERATION takes a federation name, a federation key value and some attributes. When USE FEDERATION is received by the services layer, the connection from the app to the services layer stay intact but the connection from the services layer to the database node is switched over to the federation member that contain the federation key value provided in the USE FEDERATION statement. So if you execute;

“USE FEDERATION orders_federation(tenant_id=155) …”

Connection from the application to Node#1 in services layer stays intact but the the services layer switches the connection from the root database to the federation member that contain tenant_id value 155 (white arrow from gateway node#1 to orders_federation member database node)

After this point all transaction processing, aside from a few TSQL statement gateway nodes process, happen directly on the federation member node. All your stored procedure executions to queries directly work with the node that contain the federation member which contains tenant_id 155. Each federation member and root database is spread over to various nodes in the platform layer so as more processing hit other federation members that contain tenant_id 655 or 1055, other nodes in the platform layer is engaged in processing those transactions.

At this point you may say, fine the transactional processing is decentralized but how about processing of USE FEDERATION itself? The information in the root database is the only way to figure out which federation member has tenant_id=155 or 655 or 1055? SQL Azure services layer is smart about that as well: For processing this connection routing, gateway nodes do caching of federation metadata and pooling of connections to process the USE FEDERATION command itself. That means while the cache and pool is getting built the root database receive queries asking for the federation map but after things get warmed up, services layer nodes do the heavy lifting.

So with the magic of USE FEDERATION statement both the processing of connection routing and the transaction processing in federation is decentralized! that gives you the ability to do processing much beyond capacity limits of a single node!

Hope this helps explain your question Dan!

-cihan biyikoglu