Lots has happened since I last blogged.  For almost two years, I've worked on Cosmos, some of the plumbing for Bing.  It stores hundreds of petabytes of data on tens of thousands of computers.  Large scale batch processing using Dryad with a high-level language called SCOPE on top of it.  Working on this team (with some amazing colleagues and friends) has been one of the highlights of my career.

Last June, I was honored to be able to write a paper for the Communications of the ACM.  It is called "If You Have Too Much Data, then 'Good Enough' Is Good Enough" and can be found here: http://cacm.acm.org/magazines/2011/6/108666-if-you-have-too-much-data-then-good-enough-is-good-enough/abstract.   If was quite an honor and a lot of fun.

Now for some big news, today is my last day at Microsoft after over 15 years (that's 17-1/2 minus the 2 years at Amazon).  My wife, Lisa, and I have decided to move to downtown San Franciso.  We will be only about 30 minutes drive from the grandkids, my daughter, and son-in-law.  We will be about an hour from two of my brothers.  Since my father passed away in April, 2010, there has not really been a close family tie to keep us in Seattle.  We are really longing to be down where we can see the grandkids.

While I will miss my wonderful colleagues on the Cosmos team, it is time to search for something new.  I don't have a new job lined up and plan to move to SF, help unpack, and attend the bi-annual High Performance Transaction Systems workshop (www.hpts.ws) in late October.  HPTS has been a big part of my life since 1985.  I will see my friends who work in scalable computing and look around to see what opportunities are available in the Bay Area.  I know there's lots going on and I am looking forward to returning to California after my 17+ years in the Seattle area.

It is my hope to get a new website and blog going at www.pathelland.com but there's nothing there yet.  It's amazing how busy you can be working at Microsoft and preparing to move to California!

If someone wants to contact me, my personal email is phelland0215@hotmail.com.  

Thank you all for your interest and kindness through the years...  I would really like to offer more thoughts and ideas moving forward but today I am off to my last day of work and my exit interview.  You'll hear more from me soon!

All my best,

Pat

Recently, I’ve been trying to pull the various sub-teams together and open up communications and camaraderie within them.  This is an approach I’ve taken numerous times since the 1980s and I believe it is super powerful and important.  The idea is SO simple that I am always amazed it is not more common.  We pick a regular day of the week and meet for lunch on that day.  Simply reserve a conference room and have everyone run to the cafeteria (or wherever) and grab their lunch.  As people trickle into the room at a few minutes after noon, the discussion begins.  Here are some basic rules:

1) You want to pick a team size of 10-25ish which has a set of common technology interests. It should include development, test, and program management.

2) No topic is off limits but it needs to be broadly interesting to the folks on the team.  Realistically, this quickly settles into some challenge the team is facing, some architectural issue, or some customer issue.  What I like to do is let it go wherever it goes but watch the folks in the room.  Once in a GREAT while, some small group gets into something that is kinda boring the others and you need to steer the discussion back to one of general interest.  After a team is used to the meetings, it happens less often.

3) One person at a time talks as the discussion moves back and forth.  It is important to listen to everyone and let the discussion evolve.  No one should dominate the time.  The more senior people should contribute (they usually have lots of valuable stuff to add) but they shouldn’t talk for too long.

4) It is important to try and get as many people (senior and junior) to participate as possible.  When someone fresh out of college and new on the team feels they can contribute to the discussion, they feel more like they belong.  To ensure this is safe and OK, it needs to be understood that critical responses are not OK.  Gently discussing where a proposal may not be the best for some concrete reason is great as long as everyone contributing can feel safe.  Of course, healthy and safe teasing and joking around are fun.  One thing I’ve learned (by screwing it up) is that you shouldn’t call on the shy people to participate.  Sometimes, a really shy person would rather hide under the table than talk.  If you call on them, they’ll stop showing up for lunch.  Now, when they WANT to speak, it is OK to stop the table to listen (unless the folks at the table have already done the polite thing as they should).  It is normal for me to look around the room through the hour and make a mental note of who has not yet contributed.  If I have sufficient context, I may try to steer the topic to something near to their interests to make it easy and natural for them to chime in… that’s not threatening to them.  When this goes well, you will find that in a team of 15 to 20 everyone (or almost everyone) will have participated during the hour.

5) As much as possible, it is good to not cancel the weekly lunch discussions.  We all find crunches and deadlines.  The team gets busy and has to work its butt off for a few weeks.  Still, everyone needs to eat lunch and the emotional decompression with your friends (and talking about the fun challenges in your product) is a good diversion from tracking down those bugs.

6) I think it’s important to not prepare for the lunch nor to announce a topic for the lunch.  If the topic is known in advance, sometimes people say: “That’s not my favorite topic” and blow off the lunch.  If, on the other hand, they get used to not expecting to know the topic and that the sessions are fun, they will show up and it can evolve into something fun.  Also, if you have announced: “Sam will talk about XYZ”, it gets really tough to handle when Sam is a bore.  Not everyone is capable of keeping a lecture interesting.  It is better to say that the lectures are not during the “Friday Lunch Discussions” but are at some other time.  You will learn that having a pre-planned topic takes away from the team’s sense of ownership of THEIR time to chat together.  If there is a speaker, their won’t be as heavy a participation and it is a different event. 

I like the following definition of the word “Culture”.   Culture is “the way we do things around here”.    It’s my experience that once the expectations are set, the team just naturally behaves in accordance with how they are used to behaving.  That means you can set this discussion group up and it just happens moving forward.

There are many positive results from having these weekly meetings.  The members of the team get to know the some of the larger team better than they would have… healthy fun discussions make you LIKE your colleagues.  With a team size of 10-25ish, there will always be folks you don’t know as well as you’d like to know.  The discussion of the main challenges for the team can forge and evolve the shared vision for the team.  It is essential that the broader team understand the big picture and then work down into some of the details of the other pieces.

For me, this is both a major investment and a really nice treat.  I’m working to provide architectural guidance to all of COSMOS and Autopilot.  It’s pretty hard to keep up with all of these efforts and the people involved in them.  I’ve wiped out my lunches for the week.  Monday is Autopilot’s discussion meeting.  Tuesday’s is the COSMOS Store team.  Wednesday this week was the first for the SCOPE Compiler and Optimizer team.  [Thursday’s are my day to write quietly… like this blog entry.]  Friday’s are the Execution Engine’s meeting.  I am still trying to figure out which of these days will have TWO meetings so the Manageability team can have a discussion (and I will have to switch off between them).

Each time I’ve started a team going with this, there’s quite a bit of confusion before the meeting but afterwards we have a good consensus that it was fun and useful.  For me, it allows a precious opportunity to learn about the technology, the people, and how the people interact with their technology.  Sometimes, I can contribute some comment here and there which moves the way people think about the goals and direction of the product.  Always, it gives me an opportunity to work to help all the members of the team to learn and grow.  A healthy team is built out of people who are growing and stretching in their abilities.  This requires all the members of the team helping their friends in that process.

While typing this blog entry, an email came through from a teammate wanting to start a small effort on a utility.  The discussion at lunch yesterday encouraged him to push forward with an idea he and another teammate were thinking about.  I’m going to meet with them next week and get to know them better.   It is tough to quantify all the ways these discussion help the team!

Anyway, I just wanted to share this experience with you.  Software is much more than just bits and bytes.  Many times, it is building the caring and comfort that allows you to be friends with shared responsibilities and shared goals.   Sharing meals and camaraderie make a big difference.  An occasional beer is good, too.

Great to be blogging!

- Pat

Overview:

HPTS was founded by Jim Gray, Dieter Gawlick, Andreas Reuter, and some other industry luminaries in 1985.  It is a workshop designed to bring together practitioners and academics working in super large, scalable, high throughput systems.  There is a lot of overlap between this community and the folks doing DBMS systems.  In the early days, the biggest challenges included getting 1000 transactions per second through a mainframe.  At the time, that felt like getting a man to land on the moon and come back safely.  Now, this includes huge web based sites like Amazon, eBay, Google, and Bing.  Our area of interest also includes integrating massively complex heterogeneous Enterprise and Internet (B2B) systems.  Of course, we are fascinated with hardware trends, networking, and all sorts of geeky stuff.

I first attended HPTS in 1985.  It was a magic opportunity to be with legends in the industry.  Gene Amdahl drove up in his Rolls Royce and parked next to my beat up old van.  Mike Blasgen(System R), Mike Stonebraker, Bruce Lindsay, Mohan, Andy Heller, Dieter Gawlick, Hector Garcia-Molina, Irv Traiger, Phil Bernstein, Jim Gray, and many more folks who had become well known to me through their papers were there in the flesh and treating me (a 29 year old punk) with kindness and respect.  It was a magical 3 day gathering.

Let me explain the Asilomar Conference Center (www.visitasilomar.com).  Asilomar sits right by the Pacific Ocean between Pebble Beach and Monterey in the Pine Trees and includes dunes leading right to the beach.   It was built as a YWCA around 1910 or so and contains a number of beautiful historic buildings in the arts and crafts style built by architect Julia Morgan.  In the 1950s, it was sold to the state of California which has operated it as a conference center (and built more buildings in the 1960s).  The rooms are rustic (no TVs or phones – crappy mattresses), the food is served in a large dining hall when the bell rings on the roof and it has a magic feel.   It is normal to walk out of your room, hear the surf, and see deer walking around.  I absolutely love the evenings by the real wood fireplaces.

In 1985, Gray decided we could all go cheap and lined us up to sleep four in a room.  I shared a room with Dieter Gawlick (inventor of IMS/FastPath among other things), Hector Garcia-Molina, and Theo Haerderer (a wonderful professor from Germany).   After the first night, Hector informed us we ALL snored too much and moved to stay in a hotel down the street.  For the next 8 or 10 gatherings, we were two to a room but now most of us pay a bit more to have a private room.

I’ve attended HPTS every two years (for three days) since 1985.  I was program and general chair in 1989 and general chair in 2007.  Since Jim Gray’s disappearance, I’ve assumed the role of local arrangements chair and “godfather” – kinda’ watching out for the whole workshop.  This year, James Hamilton (now of Amazon) was general chair and Margo Seltzer (of Harvard) was the program chair.  We had a wonderful (but exhausting) three days and I will tell you about them soon.

One of the traditions is that the folks leading HPTS wander around during the first two days trying to figure out who gets suckered into doing the work as the next workshop happens two years later.  I am thrilled to report that Mohan (from IBM) will be the general chair in 2011 and Mike Carey (now of UC Irvine) will be the program chair.  I will remain as local arrangements chair and “godfather”.

So, every HPTS starts with Sunday dinner and a casual beer/wine afterwards.  Monday is all day presentations and after dinner we have poster sessions (volunteer 10 minute talks – we had 15 this year).  Tuesday is all day presentations and this time we had a 90+ minute panel after dinner.  Finally, we have sessions Wednesday morning and adjourn after lunch.  It is a long, exhausting, and exhilarating 3 to 4 days.

Of course, the most important part of the workshop is the breaks and the opportunity to talk with people working in the same area as you are working.  In addition to the grizzled old luminaries, this year we went out of our way to get more graduate students and invited 16 of the brightest and best from Berkeley, Stanford, MIT, Harvard, Brown, and more.   I spent each of the three evenings staying up late talking by the fire with these wonderful young folks (and some other diehard old farts).   As I type this the following weekend, I am still tired from the sleep shortage!

Session 1) The New Memory Hierarchy –
                            Session Chair: David Cheriton (Stanford)

Steve Kleiman – Chief Scientist at NetApp:   “Flash on Compute Servers”

So, Steve Kleiman is the Chief Scientist at NetApp so you would expect him to be focused on network attached storage and we were not disappointed.

Steve started with a discussion of hardware trends (not all of which I was able to capture in my notes).  He observed that with multi-core and VMs, more bandwidth (BW) and IOs per Second (IOPS) are needed for each node.  Also, as a trend, Steve pointed out that flash memory has more than 5X the IOPS and less than 100 microseconds access time.

The original case for network storage lies around the benefits of aggregating disk which include more random IOPS and bandwidth as well as the management simplicity of concentrating data into a centralized place.  It has been the case that a single application per host has somewhat limited ability to consume bandwidth.

Considering flash in a networked storage environment, it does not have the same benefits of sharing.  Since there are enough IOPS in a flash drive to placate most apps, the equation changes.  Is network storage fast when the network latency is noticeable compared to flash?  We can do 5 SSDs to get 40,000 IOPS but will the latency be a problem?

We will see flash on compute servers.   Servers can already accommodate flash and it is cheap enough with 250GB < 10% of the server cost.

A new tier in the storage hierarchy:  Steve argues we will see storage split into two tiers: the IOPS tier and the capacity tier.  The IOPS-Tier will offer the best $/IOPS with low latency and will have sufficient capacity for the active data in most typical enterprise systems.  It will be the new “primary storage”.   The Capacity-Tier will offer the best $/GB – Deduplication, compression, and good serial performance.

Steve then outlined three broad categories of Host-Based Flash usage.  DAS (which I think means “Direct Access Storage but my notes failed me), Primary Storage Service, and Cache.

·         Host-Base Flash: DAS:  Single point of failure and so you would use normal backup or application based replication like Exchange or Oracle).  You can have boot files, VM, temp files. 

·         Host-Based Flash: Primary Storage Service:  File or Block.  Requires mirroring or RAID to peers.  Allows capacity sharing.  Steve believes it is likely that Microsoft and VMWare are likely to compete here – the integration of this is complex.  Making reliable primary storage is hard and this may impact the roles of folks in IT.

·         Host-Based Flash: Cache –File or block.  This will be a writethrough cache.  If you do a write-back you will occasionally lose stuff.  Using a cache can allow for a centralized data management (in a remote network storage server).  This can help with keeping the current IT storage management roles by providing an automatic “tiering”.

So, data management spans the IOPS tier and the Capacity tier.  You need to have automated data movement and global access.  It is impractical to have manual placement at huge scale.  It is going to be important for us to have a technology independent language to specify the desired properties for the data.  Say what properties you want and not how to get them.   New SLO (Service Level Objectives) for Max Latency, Bandwidth, Availability, etc.

In summary, the new host-based flash is the new IOPS-Tier.  It will replace high-performance primary storage in the cloud.  This IOPS-Tier s is not going to be economical as a secondary or as archival since the $/GB is higher than spinning media.   The Capacity Tier will be implemented as network storage.  It is optimized for the best $/GB.  Data management between these two tiers will be important.

There was a lot of discussion around the SLO (Service Level Objectives).   Steve indicated that their vision was not super-flexible set of knobs but more a small set of options for defining the data characteristics.  Margo offered that one aspect of what was important was the “value” of different parts of data which can, in turn, tell the system how hard to work to make the data remain available.

Someone asked if there was any performance analysis for 3-tier storage (memory, IOPS-Tier, Capacity-Tier) over more traditional 2-tier storage (Memory and Disk).  Steve answered that there had not been an explicit performance analysis.

Stonebraker observed that enterprises seem to like NAS/SAN but that the Cloud guys are using direct attached storage.  Are the forward looking guys walking from NAS/SAN?  Steve replied there are people very interested in NAS/SAN because it gives them a way to control the data and make servers stateless.  Adding in the IOPS-Tier is a simple extension.

This was a fascinating and engaging presentation!

Memory Technologies for Data Intensive Computing –Andy Bechtolshein (Sun Computers)

What an honor to get Andy to come and present to us.  Unfortunately, I didn’t get a chance to chat with him personally as he was only at the workshop for a short while.

Andy is a well respected and talented hardware engineer who was a founder of both Sun Microsystems and of Arista Network.  His slides are not going to be posted and I typed as aggressively as I could during his lightning fast presentation.  I would recommend you look both at my friend James Hamilton’s blog post of Andy’s talk at: http://perspectives.mvdirona.com/2009/10/26/AndyBechtolsheimAtHPTS2009.aspx and then read my notes attached below.  I have not attempted to coalesce the information between James’ notes and mine and there may be overlap.  James has done a nice job cleaning his up and I will focus on some of the other sessions in my editing.

Here’s my rough notes:

è Andy speaks VERY fast and I lost a lot of what was covered – The slides were very dense – Also, I apologize for some inaccuracies… These are RAW notes that I made during the fast paced presentation and some of them don’t completely make sense to me. 

Semiconductor Technology Roadmap

Still increasing

Number of devices on the chip still doubling

Flash is increasing faster than logic and DRAM – Flash faster than Moore’s law

Between 2010 and 2022, Flash should increase by 100X – After that (10nm) optical problems may become severe – don’t know after 2022

<more content lost…>

Emerging devices

·         Carbon-nano tubes

·         Graphene Nanoribbons

·         … <more that I lost…>

Silicon Roadmap Predictions

·         128X increase in transistors by 2022

·         1000 cores per chip by 2022

CPU Module Roadmap

·         IO bandwidth will not allow the same bandwidth per core.  IO going a little faster but number of pins hasn’t increased in 10 years.  

·         1000 cores would need several terabytes of bandwidth to memory.  Of course, this would need power and cooling

·         <more lost…>

CPU Challenge

·         Power per core -- 

·         Clock rates not increasing – need lower power level per core

·         <more lost…>

Power efficiency – Increasing frequency – a Square effect on the power

·         CPU power strategy –

·         Clock rates look flat

·         High clock rates are less power efficient

·         To reduce power, simplify architecture

·         <more lost…>

Multi-Chip 3D packaging

·         Thru-Si via Stacking

·         Need to combine CPU and Memory on one module

High Density 3D Multi-Chip Module

·          

Benefits of MCM Packaging

·         Enables much higher memory bandwidth

·         More channels, wider interfaces, faster I/o

·         Greatly reduces power

Challenges with MCM packaging

·         Total Memory – 64 device <…??...>

·         With 64 GNB/device – 4TB

·         Assuming 1K cores – 4BG/core

MCM enables  Fabric I/O

·         2010 –1 32 Gb/direction

·         2020 1.73Tbps/direction

·         Mesh or higher radix fabric topologies

·          

Expected Link Data Rates

·         10GBps shipping today

·         2012 – 25 GBps

·         2016 – 50 GBPS

·         2020 – 100 GBps

ExaScale Storage

·         Forget Hard Disks – Disk are not getting faster

Flash Experience so far

·         Not easy to get to 1000000 IOPS

·         Writes are a problem

·         SAS/SATA interface not optimal

·         Direct PCI-Express interface looks more promising.  – Move FLASH closer to the CPU  -- Lower latency and more throughput

Flash in the Memory Hierarchy

·         Flash is not random access  more like a block device  -- random access about 1000 slower than DRAM

·         Flash can be used as a stable storage device  -- writes are committed – Supercap magic behind the scenes.

·         Tremendous throughput and size – terabytes of capacity

Flash Latency

·         Latency expected to drop by a factor of 10

·         Flash transfer rates will double each year for the next several years. – 250MBS/sec 2012

Flash prices

·         2004 -- $100 GB

·         2009 $1 – GB

·         2012 -- $.25 / GB 

·         Prices are dropping dramatically –

Flash Summary

·         FLASH will be very expensive, very dense,

·         Costs falling by 50% per year

·         Access times falling by 50% per year

Tech Summary

·         Moore’s Law for at least 10 more years

·         Frequency Gains  not happening

·         Flash will be interesting

Software Challenges

·         Limits of App Parallelism

·         Need to exploit Parallelism

Session 2) Using the New Memory Hierarch
                                          Session Chair: David Cheriton (Stanford)

Scaling-Out without Partitioning – Phil Bernstein and Colin Reid (Microsoft)

This session was, in many ways, one of the most interesting at HPTS.  It was not new to me since I had been privileged to spend a number of hours talking to Colin (and a bit to Phil) about Hyder back at Microsoft.  Phil did a great job explaining a radical new approach to scale-out and serializable database!

Hyder is a software stack for transactional record management.  It can offer full database functionality and is designed to take advantage of flash in a novel way.

Most approaches to scale-out use partitioning and spread the data across multiple machines leaving the application responsible for consistency.

In Hyder, the database is the log, no partitioning is required, and the database is multi-versioned.  Hyder runs in the App process with a simple high-performance programming model and no need for client server.  This avoids the expense of RPC.

Hyder leverages some new hardware assumptions.  I/Os are now cheap and abundant.  Raw flash (not SSDs – raw flash) offers at least 10^4 more IOPS/GB than HDD.  This allows for dramatic changes in usage patterns.  We have cheap and high performance data center networks.  Large and cheap 64-bit addressable memories are available.   Also, with many-core servers, computation can be squandered and Hyder leverages that abundant computation to keep a consistent view of the data as it changes.

The Hyder system has individual nodes and a shared flash storage which holds a log.  Appending a record to the log involves a send to the log controller and a response with the location in the log into which the record was appended.  In this fashion, many servers can be pushing records into the log and they are allocated a location by the log controller.  It turns out that this simple centralized function of assigning a log location on append will adjudicate any conflicts (as we shall see later).

The Hyder stack comprises a persistent programming language like LING or SQL, an optimistic transaction protocol, and a multi-versioned binary search tree to represent the database state.  The Hyder database is stored in a log but it IS a binary tree.  So you can think of the database as a binary tree that is kept in the log and you find data by climbing the tree through the log.

The Binary Tree is multi-versioned.  You do a copy-on-write creating new nodes and replace nodes up to the root.  The transaction commits when the copy-on-write makes it up to the root of the tree.

For transaction execution, each server has a cache of the last committed state.  That cache is going to be close to the latest and greatest state since each server is constantly replaying the log to keep the local state accurate [recall the assumption that there are lots of cores per server and it’s OK to spend cycles from the extra cores].  So, each transaction running in a single server reads a snapshot and generates an intention log record.   The transaction gets a pointer to the snapshot and generates an intention log record.  The server generates updates locally appending them to the log (recall that an append is sent to the log controller which returns the log-id with its placement in the log).  Updates are copy-on-write climbing up the binary tree to the root.

Log updates get broadcast to all the servers – everyone sees the log.  Changes to the log are only done by appending to the log.  New records and their addresses are broadcast to all servers.  In this fashion, each server can reconstruct the tail of the log.

Now we get to transaction commit.  All machines are marching ahead rolling the log forward.  When an intention record is generated (indicating a proposed new root for the log), each machine in the cluster checks for a conflict.  If any of the records updated conflicts with previously committed work, the transaction is aborted.  Conflict in this case is traditional read/write conflict using all the classic set of rules for various degrees of consistency.  If there are no conflicts, the transaction is committed.  If there are conflicts, the transaction is aborted. 

Because there is an append in the log controller which assigns order, each intention record will race to the log and the controller will pick a winner.  Because log ordering is deterministic, each of the server nodes can reliably come to the same conclusion about which transaction won the race and, hence commits rather than aborts in the face of conflict.  Conflict detection and transaction commit is done redundantly and predictably at each node.

So, each node generates an intention log for a transaction it wants to do.  Every server checks for conflicts and then either aborts or commits.  The log order provides determinism.

Performance of Hyder:  The system scales without partitioning.  The system-wide throughput of update transactions is bounded by the update pipeline.  It is estimated this can perform 15K update transactions per second over a 1GB Ethernet and 150K update transactions per second over a 10GB Ethernet.  Conflict detection and merge can do about 200K txs per second.

The abort rate on write-hot data is bounded by the transaction’s conflict zone.  This is determined by the transactions end-to-end latency.  This conflict zone is about 200 microseconds in the prototype.  It is estimated that this would handle about 1500 TPS (assuming all the transaction have conflicts).

Major Technologies:  This is using flash as append only (and it is not using SSD flash but raw flash with a special controller).  The custom controller over raw flash has mechanisms for sync and fault tolerance.  Storage is striped with a self-adaptive algorithm for storage allocation.  It includes a fault tolerant algorithm for a totally ordered log.  Also, it has a fast algorithm for conflict detection and merging of records.

Status:  Most of Hyder has been prototyped but there is a long way to go.  Phil Bernstein and Colin Reid are working on a paper and HPTS is the first time Hyder has been discussed publicly.

Again, this was a wonderful talk with some fascinating and different ideas.  Great work!

A Performance Puzzle:  B-Tree Insertions are Slow on SSDs (Bradley Kszmaul – Tokutek – Research Scientist at MIT)

So, this talk was about some surprises in SSD performance in supporting MySQL.  Much of the discussion was about the datasheets for some of the drives not matching the measured reality.

Bradley had measured the value of running MySQL against SSDs and expected the SSDs (with higher IOPS) to get better Btree performance.  Here’s my rough and raw notes:

Intel X25E Specs

·         Read BW up to 250 MB/s

·         Write BW up to 170 MB/s

·         Random 4KB read rate – 35 KIO/s

·         Random 4KB

·         <lost some…>

Measure Berkeley DB

·         Trending to 4500 write per second

Performance Model  --

·         Startup cost plus bandwidth cost

·         Looking at read performance as a function of block size

·         As block size gets bigger, bandwidth goes up

Discussion à Margo points out that many Filesystems see large blocks and do read ahead.  That can perturb the results…

When you read and write large blocks, Bradley sees Filesystems doing the right thing and keeping contiguous blocks adjacent.

The datasheet for the drive is not matching the reality for larger block sizes

Mixing reads and writes seems to impact the performance quite a bit…

What Block Size to Use?

·         For point-queries, BTrees are insensitive to block size. 

·         For range queries, block size if important

·         Tension: Large block sizes make range queries faster; Small block sizes make exact match faster

·         Set block size so half of the time is accounted by seek, half by bandwidth

·         Reads: 50KB for SSD but 500KB-1000KB for rotating disk

·         Writes 10KB SSD , 500KB-1000KB for HDD

·         Mixed read/write 21KB for SSD

Cache-Oblivious Approach

·         Use data structures that are fast for any block size

·         Also can speed insertions without slowing searches

·         Tokutek’s MySQL storage engine uses these ideas.

Discussion: James Hamilton layering of software and release by release changes makes it hard to get predictable behavior.   When all these releases are separately evolved, the performance behavior is very unpredictable.   Bradley: fragmentation starts messing with this, too.  James: all the layers interact and we need to measure by the workload and adapt.  The complexity makes performance very hard to predict.

Implications of Storage Class Memories (SCMs) on Software Architectures (C. Mohan – IBM)

Storage Class Memory (SCM)

Mohan gave a very thought provoking talk on the implications of Storage Class Memories.  Where do we need them?  What good will they do?

If we have non-volatile memory that is as easy to access as DRAM, will we have corruption problems?  Will we see a running system trash the non-volatile memory like it trashes DRAM?  One of the advantages of our current system design is that we have to pass a “sanity test” of an IO driver or messaging stack to get nastiness outside of a running system.  That dramatically reduces the chance of this happening.

Anyway, Mohan’s talk was really thought provoking and enjoyable!  It was good to see my old friend back at HPTS after taking a stint in India for a few years.

Raw notes:

·         Blurs the distinction between DRAM and Disk

Industry SCM Activities

·         IBM

·         Intel/ST spun out of Numonyx (FLASH and PCM)

·         Samsung, Numonyx sample PCM chips –

·         <lost som>

Speed/Volitility/persistenct

·         DRAM plus SWCM – Fast and persistent

·         Persistent storage will not lose data

Competing technologies for SCM

·         Phase Change RAM (most promising)

SCM as a part of the storage stack

·         The distinction between RAM and Disk – the line in between is blurring

·         Memory-like and also Storage-Like

SCM Design Triangle

·         Speed – endurance – Cost/bit

·         Speed with Write endurance is more memory-like

Phase-Change RAM

·         When density increases, the failure rate goes up as bits flip more often

·         SCM is on Moore’s law

PCM is fast

·         100-1000 ns

Architecture

·         SCM may be used as Memory

·         SCM may, alternatively be used as a storage (through IO controllers)

·         Discussion à Pat:  Bugs in non-volatile memory can sometimes be preserved

PCM Use Cases

·         PCM as Disk

·         PCM as Staging device

·         PCM as memory

·         PCM as extended memory

Exploring DBMS as an exploiter of PCM

·         Should log records be written directly to PCM?

·         Should log be more like Disk (first to buffer and then forced to disk)?

·         PCM is not as reliable as disk à do you need to offload to a disk

·         If you use Group Commit, it is unclear that PCM helps a DBMS

PCM replaces DRAM – Buffer Pool in PC

·         The PCM Buffer Pool access will be slower than DRAM

·         Writes will suffer more than reads!

·         Should we instead have a DRAM BP backed by a PCM BP?

·         Similar to DB2 z in parallel sysplex environment with BPs in coupling facility (CF)

·         But DB2 situation has well defined rules on when pages a move from DRAM CP to CF BP

·         Variation was used in SafeRAM work at MCC in 1989

Assume the whole DB fits in PCM

·         Apply old main memory DB concepts design directly

·         Shouldn’t we leverage persistence specially?

·         Every bit change persisting may not be good!

·         Today’s failure semantics has flexibility on tracking changes to DB pages.

·         Memory overwrites will cause more damage

·         If every write assumed to be persistent as soon as write completes, then L1 and L2 caching can’t be leveraged.

Assume whole DB fits in memory

·         Need to externalize periodically since PCM won’t have good endurance

·         If DB spans both DRAM and PCM, hard to figure out what goes where!

What about Logging?

·         If PCM is persistent and whole DB is in PCM, do we need logging?

The end of Monday morning!  More soon… 

-          Pat

The idea is that there are three different types of components. 

• Baby-Bears are much like the classic object with inheritance, reference-passing, and synchronous interactions with their callers.  
• Momma-Bears (closely related to what is provided by MEF), are comprised of multiple Baby-Bears.  While exhibiting synchronous call patterns, they do not support pass-by-reference nor inheritance.  They will, however, allow for support of dependency injection.
• Papa-Bears are the biggest of the three and are most similar to a SOA Service.   A Papa-Bear will typically be composed of multiple Momma-Bears.   Papa-Bears interact with their partners asynchronously, are frequently remote, do not support pass-by-reference or inheritance, and may be offline.

My interest has always been in the realm of the Papa-Bear and other projects have focused more in the realm of the Baby-Bear and the Momma-Bear.   I will leave it to Krzysztof to explain MEF (which I believe is more akin to the Momma-Bear) type of component.

Anyway, this is what I like to call “Building Blocks and the Three Bears”.

- Pat

If you are interested in cloud computing, you want to understand these ideas.

Also, I highly recommend the Berkeley blog on cloud computing available at:  http://abovetheclouds.cs.berkeley.edu.

Thanks to the folks at Berkeley for their hard work and permission to show this to you.

- Pat

- Pat

In August of 2001, the SQL Service Broker team did a design preview and I wrote five presentations totaling 280 slides.  I presented these over a 12 hour period (actually, that time also includes a sixth talk with another 60 slides on "Fiefdoms and Emissaries" but I'll push that out later).

CAUTION:  These slides are from August of 2001 and SQL Service Broker shipped in late 2005.  Many things are changed!  Do NOT count on these slides for accuracy of the product but they can be a big help with inspiration.

The following slide decks are now out in my new public skydrive folder.

SQL Service Broker Overview

An overview and introduction to SQL Service Broker.  This slide deck has the following major sections:

• Introduction:  Motivates the challenges in loosely-coupled messaging, the advantages of deeply integrating messaging with the database, support for scaling and fault tolerance, and some scenarios for SQL Service Broker.
• High-Level Architecture: This covers web services, brokers, message fragmentation, service environments (which provide a namespace for services), message-types, service-contracts, services, and service programs.
• Programming Model:  This section covers an overview of the programming model, service instances and state management, dialogs, and again ties these programmatic concepts to the service instance.  Caution: the syntax has changed between this and the shipped product!
• Relationship to Other Stuff: Here we visit management issues, interoperation, and look at products that create services.
• Conclusion:  We revisit the scenarios and summarize the points made.

The SSB overview talk has 87 slides and can be found here.

SQL Service Broker Messaging

This talk examines the message protocols used in SQL Service Broker.  Reminder:  This is not an accurate depiction of what is in the product!

This slide deck has the following major sections:

• Introduction:  This covers the concepts of reliable messaging, brief discussion of web services, Internet and Intranet issues, transport support, binary and lossless transformations, and a sketch of the protocol stack.
• Adjacent Broker Protocol: The adjacent broker protocol talks to a nearby SQL Server to propagate messages and get them closer to their destination.  This section outlines the challenges at this level with delivery (sometimes tunneled through firewalls), lifetime management, and format negotiation.
• Endpoint Broker Protocol: The endpoint broker protocol functions across the SQL Server engines that actually constitute the two endpoints for the communication (and it layers atop the adjacent broker protocol).  This section covers dialog endpoint state, fragmentation and sequence numbers, piggy-backed acknowledgements, and unsequenced ack messages.
• Message Formats: This covers pieces of the messages, encryption challenges, and ASCII versus binary sections of the SOAP message.
• Monolog Contracts: Monologs were not implemented but they are VERY interesting if you care about Pub-Sub.  This short section on Monologs covers some of the issues related to the messaging formats.  It discusses dialog contracts to implement monologs.
• Web Services and SOAP Bubbles:  Some brief framing of the relationship of the SSB protocols to Web Services and SOAP.
• Conclusion: a summary of the content covered.

The SSB messaging talk has 31 slides and can be found here.

SQL Service Broker Performance

This talk examines performance, scaling, and high-availability issues with SQL Service Broker.   Reminder:  This is not an accurate depiction of what is in the product!

This slide deck has the following major sections:

• Introduction 
• Work Within a Service: This covers messages in and out of a service, service instance state management, data access from services, and scalable and highly-available ASP.NET apps.
• Fault Tolerant Services: In this section, we cover designing services for fault tolerance, deploying services for fault tolerance, and more issues with fault tolerance.
• Scaling and High-Performance: Here, we examine application issues for scale-up, execution of scale-up services, application issues for scale-out, and execution of scale-out services.
• Conclusion

The SSB performance talk has 46 slides and can be found here.

SQL Service Broker Monologs

One of my favorite talks, this describes the unshipped feature of SSB Monologs.  Monologs implement a very flexible form of pub-sub.  While they are an unimplemented feature in SSB, the serious implementer could follow the guidance in this talk and avoid a lot of difficulties.   Reminder:  This is not an accurate depiction of what is in the product (it has not shipped)!

This slide deck has the following major sections:

• Monolog Overview:  This covers "What's a Monolog?", programmatic interfaces, and broker-to-broker contracts.  Included are the notions of "History versus Currency", Monolog-Groups, and Subscriptions.
• Programming with Monologs:  This section includes programming a publisher, requesting a subscription, and monolog delivery in a subscriber.
• Some Special Monolog Features: A really cool concept called a Monolog-Checkpoint is covered as well as the lifetime of monologs.
• Plumbing Monologs Over Dialogs:  We examine fan-out trees for scale, plumbing the fanout using dialogs, publisher agents, and subscriber agents.
• Conclusion:  Here we examine content syndication and content aggregation and then summarize the talk.

The SSB monologs talk has 53 slides and can be found here.

SQL Service Broker Security

Again, one of my favorite talks, this describes some of the special challenges with security that arise in a messaging and services system like SQL Service Broker.  Classic session-oriented security only works when messages are received in order.  That is just one of the challenges.   Reminder:  This is not an accurate depiction of what is in the product!

This slide deck has the following major sections:

• Security Behavior: This section covers identity, dialog authorization, monolog authorization, and auditing.
• Encryption Concepts:  Here we examine symmetric key encryption, public key encryption, and some practical rules for using encryption.
• Encrypting Dialogs:  In this section, we introduce a concept called "handshakes" which helps optimize the performance of certificate management in a dialog-based system.  We look at handshake certificates, messages and dialogs, and cryptographic guarantees.
• Encrypting Monologs: Monologs are N-party communications.  We examine in this section why 2-party security works best and then how we can use a fan-out tree to implement a secure scalable publisher.
• Conclusion

The SSB security talk has 61 slides and can be found here.

I hope these talks are fun as there's still lots of interesting concepts in them 8 years later.

- Pat

I just couldn't resist sharing some fun, though...

As I was sorting through old files, I came across a video of Transaction Processing and the Barbarian Hordes and placed it up for all to see.  This was my favorite moment on stage (at least so far) and was the source of a rare bout of stage fright before starting the presentation...

This was at the Long Beach PDC in November of 1996.  The PDC was titled "The Renaissance of Distributed Computing" and I had a one hour General Session to introduce Microsoft Transaction Server.  We wanted to show that MTS was easier to install than a classic Transaction Processing Monitor.  So, my friend Greg Hope came out to do a 2-1/2 minute setup on MTS.  While this was happening, I told a five minute joke...

There were 7000 people in the room and (I think) 10000+ on video feed.  They had NO idea what I was doing for at least three minutes.

- Pat

First, let me talk for a minute about Asilomar. It is located in the forest right off the beach in Pacific Grove, CA (right next to Monterey).  From 1913 to 1956, it was a YWCA and has a number of magnificent and beautiful historic buildings.  In 1956, it was sold to the State of California which has run it as a conference center since then.  Asilomar has a deep and special place in my heart.  I've spent 3 days every two years there since 1985 at the HPTS (High Performance Transaction Systems) workshop.  This gathering has been attended by many wonderful friends in the transaction processing and database community.  Every job I've gotten since the 1980s has been through contacts forged at HPTS.  I chaired the HPTS workshop in 1989 and (believe it or not), my first grandson was born DURING the workshop (after the keynote, before the first session, and immediately after I ordered more coffee in a crisis for the first break).  I still smile at the location of the (now removed) pay phone where I continually called to find out how the birth was progressing.  Nick (my grandson) is now at Cal State San Jose.    It was at Asilomar that I last saw my dear friend, Jim Gray.  Now, I go two times each two years for CIDR (January of the odd years) and HPTS (October of the odd years).  The smells, the feel, the fireplaces, the wonderful Chapel Building (where we meet to present to each other), and the amazing participants offer me a rich combination of memories and current stimulus.

So, this year's CIDR was a good one.  The agenda was packed.  There was a healthy combination of seasoned database folks and young grad student.  We listened to a lot of presentations and had a lot of stimulating conversations.  I saw a bunch of old friends and made a bunch of new ones.  Someone (who... me??) managed to bring some nice sipping whiskey each night and get a gang off to chat quietly by the fire and ensure we ended up sleep deprived but with a better understanding of our new and old friends.  It was a very, very nice event.

On Tuesday evening, I was on a panel with my old friend from Tandem days, Shel Finkelstein.  We were joined by a friend of mine I met last year, Michael Armbrust, who is a graduate student at UC Berkeley studying in the Berkeley RAD Lab.  The panel was on the ways in which cloud-based systems are, in many cases, evolving towards availability over consistency.  I presented a summary of "Building on Quicksand".   Shel did a marvelous job presenting "Principles for Inconsistency".  Finally, Michael presented about his project at Berkeley called "SCADS: Scale-Independent Storage for Social Computing Applications" and then we had a joint Q&A.  You can find the papers for these talks at http://www-db.cs.wisc.edu/cidr/cidr2009/program.html and then navigating the program to Tuesday evening.

The panel was a blast...  I've known Shel for almost 20 years and knew he would do a great job.  He did.  This was Michael's first conference but I was THRILLED with his talk.  He was clear, organized, and thought provoking.  An absolute home run for someone's first conference! 

FInally, I want to thank Mike Franklin, Joe Hellerstein, and Gerhard Weikum (and many others) for all their hard work to make CIDR 2009 possible.

- Pat

BuildingOnQuicksand-V3-081212h-pdf.pdf