Li Xiong

A true story about debug

Li Xiong — Tue, 14 Nov 2017 07:52:16 GMT

My first job is Support Engineer for developer tools at Microsoft China. Most of the time we remotely help Microsoft customer to solve their development issues, like deadlock, heap corruption and performance problem. For difficult case we would connect to customer machine to perform remote debug. In rare case our engineer would travel to customer site to do onsite support. At that time there was a special engineer title called CPR, meaning Critical Problem Resolver. Only very few engineer were entitled for that. Any problem that other engineers could not solve, CRP takes it and solves it. In the earliest stage of my career I worked closely with a CPR who was considered the youngest CPR. Getting mentoring from him is still the most important career experience for me. Some small trick I learned from him 10+ years ago still shocks my team members today when I demonstrated it. The story I want to tell today is not about him. I heard the story from him. It’s a story of another legendary CPR. I don’t remember the name of the protagonist. It was around year 2000. A traditional company decided to build their IT solution by using Microsoft tech stack. At that time, only banks and ISP would invest on IT infrastructure. The common IT infrastructures were mainframe system from IBM or SUN, based on UNIX like Solaris. Obvious this was a big bet for that customer. The customer signed a contract to use Microsoft consultant to help build the service. In the pre-production test, the new developed server application hit random crash. The same application worked fine in dev environment. When running on production environment it crashed in different places. Our support team had been following up for weeks, but not clue for the root cause. The customer was losing confidence on Microsoft. With the deadline approaching, we sent our CPR to customer’s site. Our CPR engineer sit on customer site, debugged day and night, analyzed tons of dumps, and tried different approaches. After several days, he (or she) told the customer: you got a bad hardware, contact the vendor to replace your motherboard. The customer was angry: “you cannot find your bug and you want to blame the hardware?” At that time, hardware vendor would not replace it for no reason. More importantly, shipping new hardware from USA takes time (computers were considered high tech, not made in China). If it would not fix the problem, we miss the project deadline. The customer eventually agreed to replace the hardware, but with a condition. If the problem is not related to the hardware, Microsoft should refund all the cost for this project including the consultant cost. The CPR explained the situation to the team leader. The team leader and other Microsoft leadership decided to support the CPR and agreed customer’s condition. Customer makes the order. The new hardware arrives. Power on, running for days, and the problem never happens. This is the end of the story. The story left a seed in my heart. I was wondering, if someday, I could be so skillful to tell from software bug and hardware bug in my debugging session. I could be so confident to ask customer to replace hardware when software crashes. Would I have the courage to trust the team’s judgement in similar situation? A decade passed. I am still in Microsoft but I am not in support role any more. I am working on Azure Compute team. We manage all Microsoft server asset like Bing.com and Azure. The whole cloud infrastructure is deployed and managed by my team, and we have our service code run on every server machines. 10+ years ago, the support team I worked was so worried about that single customer. We wanted the software running healthy, and any single crash mattered. Today, our team’s service is supporting every Microsoft cloud customer. A single crash still matters. With modern instrumentation, we know any crash almost immediately. For the past years I debugged crash dumps many times. I also introduced some bug that almost caused global outage. I saw disk corruption and memory corruption. I saw outlier crash callstack. From analysis some of them were likely caused by bad hardware issue. But I have not caught hardware failure in debugger lively and directly, until last weekend, 11-11-2017. The following is my analysis on a live debug. I think I observed a hardware failure which caused software crash. The debugging process was not tricky, but I still feel so good and peaceful. Problem Description: GetAndXXXXXXX started constantly crashing on SXXPrdXXXXX XXXXXX-c125-XXXX-8202-513e917c5ddc since 2017-11-06 03:40:39.450 UTC. The same binary is running everywhere without any issue and we verified the binary running on the problematic machine is not corrupted. Dump analysis: From mini dump we learned that the crash is Access Violation in string operation. Since this is AMD64 with optimization turned on, we could not confirm the actual broken pointer. Since this is consist repro, we setup live debug. From the live debug, the crash happens in inside this function call. A bad pointer was passed into the AppendF call. (line 1658) serviceListIniBuffer.AppendF("ServerList=%s\r\n", m_newServiceList.config->GetStringParameter("Manifest", "ServerList", "").c_str()); We set breakpoint just at above line and got: 00007ff6`36961932 488b00 mov rax,qword ptr [rax] 00007ff6`36961935 4c8bc0 mov r8,rax 00007ff6`36961938 488d1541270b01 lea rdx,[00007ff6`37a14080] 00007ff6`3696193f 488d4d20 lea rcx,[rbp+20h] >>>>>>>> 00007ff6`36961943 e8981ef3ff call GetAndXXXXXXX!YYYYY<127>::AppendF (00007ff6`368937e0) From there we know (we push parameter from right to left. this pointer goes to RCX.) r8 -- output of m_newServiceList.config->GetStringParameter rdx -- should be hardcoded string "ServerList=%s\r\n" rcx -- this pointer to serviceListIniBuffer The memory info for these: 0:000> da r8 00000001`0a3a48c0 "25.66.164.187:25.66.164.4:25.66." 00000001`0a3a48e0 "164.251:25.66.165.4" 0:000> da rdx 00007ff6`37a14080 "????????????????????????????????" 0:000> dc rcx 00000001`075be190 00000239 00000000 0a3a5710 00000001 9........W:..... 00000001`075be1a0 3b0a0d3b 69685420 69662073 6920656c ;..; This file i 00000001`075be1b0 65672073 6172656e 20646574 47207962 s generated by G From above, we know the bad pointer is the 1^st parameter. Interestingly, the 1^st parameter is actually hardcoded string “ServerList=%s\r\n”. How could this be corrupted! If we look carefully, the rdx value is 00007ff6`37a1408. Let’s flip a bit and change it to: 00007ff6`36a14080, we get: 0:000> da 00007ff6`36a14080 00007ff6`36a14080 "ServerList=%s.." This is 1 bit corruption. Conclusion: We high suspect this is hardware failure. My guess there is a CPU/Memory/BUS failure that when a specified memory sequence is loaded, a bit get corrupted. That’s the explanation I can think about why the same application fails all the time at the same place. BTW, the memory address does not change because the string is loaded from EXE, not a DLL. EXE always maps to the same start address.

软件测试随想

Li Xiong — Mon, 19 Nov 2012 00:02:25 GMT

今天看到一则有意思的题目：

http://www.weibo.com/1401880315/z5UBR5NN6

对我来说，这类问题和讨论我已经自问自答地探讨过无数次，以至于除了问题本身外的讨论都显得无聊了。

以前我把我整理的思路总结成了一篇pdf叫做“软件测试随想”。我知道里面的观点在很长一段时间都不可能被接受，所以我写下来后从来都没有发表过。反正我现在也不干test了，不妨今天放上来，请见本帖附件。

软件测试随想.pdf

Web服务的性能，和BCrypt性能问题的解决方法

Li Xiong — Sat, 24 Dec 2011 11:54:21 GMT

在昨天的研究中，发现BCrypt最大的隐患在于性能。BCrypt的安全性是通过牺牲性能来获取的。BCrypt比传统hash+salt要安全一万倍，但是代价是使用BCrypt做认证对比密码和密文时候，性能也比hash+salt要慢一万倍。

所以，我得出一个结论。如果使用传统hash+salt需要N台认证服务器的话，那用BCrypt就需要添加10000*N台服务器才能达到同样的性能。比如一个邮件系统使用了2台服务器来专门作认证，那使用BCrypt的话就需要再购买2万台。当然，小的应用，如果使用一台服务器1%的性能就可以做完认证的话，使用BCrypt只需要100台服务器。

这个听起来比较吓人，但是我通过另外一种计算方法，也能得出同样的结果。比如说163的邮件系统，峰值的时候每秒钟可能需要处理10000个登陆请求。比如每天早上9点到9点半之间，很多人都会登陆邮件系统。按照BCrypt提到的，每个请求需要花费0.3秒，那么处理10000个请求需要3000秒。这显然是不能接受的。如果要让客户都能在1秒钟登陆，那就需要添加服务器，把登陆请求均分到各个服务器上。对于单CPU的服务器，就一共需要3000台。

我把上面的想法和网友讨论后，得到了不同的意见。有人觉得0.3秒的处理时间其实不多阿，因为通常一个web请求，内存，磁盘数据库转一圈下来，离0.3秒也差不多了吧，所以BCrypt的0.3秒的开销是一个正常值，不会带来什么问题。有人提到普通web服务器每秒处理1000个请求是很常见的。如果按照我上面的计算方法，这就表示每个请求在1ms内完成了，这听起来有点不可能嘛。

根据上面的意见，我又仔细思考了一下。我想借这个话题，先讨论一下Web服务器的性能，再看看BCrypt带来的问题应该如何解决。

注意，下面所叙述的，是根据我现有知识和经验做的逻辑推理。我没有写半行web程序的code来验证，我也没有任何大规模web项目的经验。但是我非常喜欢先做这样“纸上谈兵”的推理，因为这样可以让我先仔细想清楚后再行动。

网页的加载时间，和单个request的处理时间是不一样的。单个网页中往往包含多个子元素，比如图片和AJAX调用。这些子元素的加载都会在额外的request里面完成。另外网页加载的时间除了web服务器的运算处理时间外，还有数据在网络上的传输时间，脚本运行时间，浏览器绘制时间，当然如果是第一次访问，还包括DNS查询和TCP握手时间等等。现在主流网站的页面加载时间都在2秒以内，每个页面大概包含10-50个子元素。客户端脚本和浏览器的处理占用了大概一半时间，所以跟服务器相关的时间大概就是1秒。由于部分子元素可能在浏览器中有cache，所以这1秒钟对应的大概就是5-25个请求。这样算下来每个请求大概在200-40毫秒之间。这里面大概web服务器的运算开销可能只有25%，那大概就是50ms到10ms之间。注意，50ms-10ms这个估算是指平均时间。

由于网络传输和客户端导致的开销不是今天的讨论范围，所以下面就只讨论web服务器运算一个request需要怎样的开销。处理一个web request，通常牵涉到下面一些工作。输出流添加，数据库读写，文件读写，复杂计算。输出流添加是最常见的，比如服务器会把Response.Write里面的内容加入到Response里面。输出流添加是非常快的。速度和string.format或者StringBuilder.Append应该在一个数量级。肯定比毫秒要快多个数量级。复杂计算是指需要CPU密集超过50ms的运算，比如正则表达式运算和BCrypt操作。数据库读写和文件读写所需要的时间往往也超过50ms，但是这两者不一定需要占用web服务器的CPU资源。

由此，在考量web服务器运算一个request所需的开销的时候，先看这个request是不是需要复杂运算。如果需要，那么这个复杂运算肯定是一个瓶颈。

那么数据库和文件操作是不是瓶颈呢。从处理request所需的时间上来说，这两者肯定也是瓶颈。因为再快也得先等到这两者返回。但是和复杂运算相比，复杂运算损害的不单单是request所需的时间，另外还损害了服务器的可伸缩性。

由于数据库和文件操作不占用web服务器的CPU，所以web服务器在等待数据库和文件操作结果的时候，可以把CPU资源用来处理新的request。比如来了100个请求，每个请求都需要2秒钟的数据库操作，不等于说服务器需要200秒才能处理完所有请求的。因为服务器可以让100个请求都进来，然后发送者100个请求的数据库操作，然后让这100个请求同时等待。2秒钟以后，这100个数据库操作的结果都返回回来后，服务器可以把这100个请求的结果都返回了。

但如果请求中牵涉到了复杂运算呢，服务器就没办法并发了。如果来了100个请求，每个请求都需要运算2秒钟，而服务器如果只有一个CPU的话，那就一定需要200秒。

由此可见，请求的平均处理时间和服务器的吞吐量不是简单的比例关系。这牵涉到这个请求具体是如何处理的。如果服务器的吞吐量是1秒钟1000个请求，不能简单地用1000毫秒除以1000，得到每个请求只需要1毫秒来处理。这个运算法则只在没有任何并发可能性的情况下才有效。这也在此证明了，合理使用异步调用，对服务器性能会有多大的提高。

接下来反过来想，对于一个不涉及数据库，不涉及文件，不涉及复杂运算的请求，合理的处理时间应该是多少呢。这个处理时间其实等同于同等复杂程度的函数调用时间。换句话说，如果只是计算出一个几十k的页面，花费的时间肯定在1ms的数量级。你肯能觉得这里没有考虑web服务器的“仪式性”开销，比如说处理TCP报文，分析HTTP头等等。其实，主流的web服务器在这方面已经足够优化，比如从IIS6开始，对于HTTP协议的处理都直接在内核里面做了，所以就算把这些开销都算上，也能做到1ms的数量级。对于计算机来说，1ms其实很长的，1ms表示一秒内调用函数1000次而已。不信你可以估算一下下面的代码，循环了10000次，大概需要多少时间：

StringBuilder dummy = new StringBuilder();

for (int i = 0; i < 10000; i++)

{

dummy.AppendLine((string.Format("this is {0}", i)));

}

var len = dummy.Length;

另外，上诉的讨论还没有考虑一个非常重要的武器:Cache。如果使用cache, 整个页面可能根本不需要任何运算，都不需要跑一行处理代码，web服务器就把结果给返回了。如果大量使用cache，别说一千次，一秒钟处理上十万次都可以。大多数的网站已经广泛使用cache了，图片是cache的，blog文章的内容是cache的，甚至评论都是cache的。正因为如此，把包含复杂运算，数据库和文件处理的慢操作跟带cache的块操作一平均，结果落到前面估算的50ms-10ms是很自然的事情。

有了上面的分析后，再来看文章前面的两个问题。第一个问题是，BCrypt的300ms开销，和数据库，磁盘文件兜一圈下来的开销也差不多阿，为什么说BCrypt就更严重呢。原因是BCrypt的300ms都压在了CPU上面，而人家的是可以并发的。当然，动不动就要去兜数据库和磁盘文件的web应用，设计上也有问题的。第二个问题是，我的服务器可以一秒钟处理1000个请求，按照原来的计算方法，就是1ms处理1个了，这个不可能吧。这个问题就要区分情况来回答了。首先，1ms处理1个请求是可能的，比如使用了cache，绝对能。其次，如果考虑并发的话，的确不能简单做除法。但问题关键是，BCrypt是不支持并发的阿。

所以，BCrypt最大的隐患的确就是在性能上。我所做的下列判断是正确的：“如果使用传统hash+salt需要N台认证服务器的话，那用BCrypt就需要添加10000*N台服务器才能达到同样的性能”。

接下来的问题是，BCrypt性能问题有解决办法么。在前面的逻辑推理中，我想到了一个最常见的办法：用空间换时间。我可以在内存中建立cache，把密码原文和用BCrypt计算后的hash缓存起来。这样每次处理认证请求的时候，就不需要计算了。当然，这里其实带来了新的安全隐患。如果有人盗窃了服务器的内存转储(memory dump), 那么就能看到cache里面的东西，那可就是密码的原文明文阿。这有何去何从呢？

引起这片文章的讨论来源:

http://weibo.com/1709648133/xDtMxivam

http://weibo.com/1709648133/xDy7YjYX0

Retrospect different Microsoft technologies from Win8

Li Xiong — Mon, 05 Dec 2011 07:35:00 GMT

The Chinese version is available here (中文链接在):
http://blogs.msdn.com/b/lixiong/archive/2011/12/03/win8.aspx

Learning how things work internally has always been my favorite. The first thing I did was to launch debugger when I got Win8 installed. Based on my research, Win8 is an amazing product. There are many old and new technologies involved in Win8. I’d like to use this blog to retrospect different Microsoft technologies.

Please note, the following analysis was based on Win8 CTP. I expect many changes when Win8 gets to RTM.

This above information is just my person opinions.

COM -Component Object Model

COM is a great technology invented in mid 90’s. COM was indented to address the reuse of software components. Prior to COM, software reuses were usually done at source code level. The C/C++ library was a typical example. Source code level requires recompilation, and it is difficult for distribution. COM made it possible to reuse software components at binary level. Starting from Win95, OLE and Office95, COM has been a cornerstone of Microsoft platform. From DirectX API to the most common clipboard, and even the host interface of .NET Framework are all based on COM. But, there is no technology which is good for everything. The problem of COM is the following:

The threading model like STA/MTA/NTA is over complex

The threading model of COM was designed to help developer. The implementation of threading model in COM Runtime was based on many system components like Win32 Message, RPC and Windows Services. COM developers must be familiar with these components if they want to use COM correctly. For example, a developer must understand the message loop and maintain it for any thread that hosts STA, otherwise deadlock is likely to happen. The complexity of threading model was a blocker for many developers who wanted to use COM at expert level.

Lack of development tool support

The relationship between COM and Visual Studio 6 is very similar as CLR2/VS2005, CLR3.5/VS2008 and CLR4/VS2010 nowadays. You had to choose either VB6 or ATL to create COM component with VS6. VB6 was good for business applications, especially the popular MIS system at that time. But VB6 was not powerful and flexible enough. Due to the lack of muti-threading support, there was no way to use MTA in VB6. ATL was powerful and flexible, but it was difficult to use. Every time you want to extend an interface, you had to do a lot of ceremony works like using muti-inheriency, defining new templates and using lots of C macros. It was never a pleasure of using ATL.

The complexities were out of control when it expanded to DCOM, COM+, and DTC

To fit into enterprise development, COM was extended and evolved to DCOM and COM+. “Enterprise development” means the requirements of security and scalability. The classic COM was an in-process model, in which the caller and callee ran under the same user account. Using impersonation was helpful, but a completed solution was to make different components run under different security boundaries. DCOM was born to make COM component possible to run from a different process, run from a different computer, run with different security levels for different interfaces, authenticate with domain account, and support different levels of encryption. You can click start, run and type dcomcnfg.exe. Try to expand some nodes and open the property page to see how complex it could be. For scalability, Microsoft further extended DCOM to COM+ by introducing object pooling and new sync model. The most famous COM+ based solution was ASP, which dominated the web development for decades on Windows platform, and eventually evolved to ASP.NET. Furthermore, Microsoft built DTC and MSMQ which were also based on COM+. You may not be aware of the niubility of DTC. DTC is short for Distributed Transaction Coordinator. It allowed you to run SQL statement to operate different database systems (like Oracle and MSSQL) under the same transaction without writing special code. This feature was super important when MSSQL competed with Oracle/DB2 at earlier days. These things made COM more horrible to use especially when development tools did not catch up.

.NET Framework/CLR

In my mind, CLR is flawless in almost all areas. But just because CLR was so good, Microsoft reduced the investment on the un-mamanged components starting from 2003.

Spreading and pushing CLR is never bad at all. Most developers write CRUD and UI code. You cannot let them to use C/C++ all the time. CLR is performant, the development tools are strong, and every new version brings huge improvements. But, there is a theory that has always been true: it is possible to write a C# application to get similar performance as using C/C++, but it costs more than using C/C++ at the first place. There are many reasons: the underlying API is unmanaged and the interop hits performance; the C# cannot control the details as much as C++; the JIT compiler has to generate additional code to achieve the CLR exception model and security model; this pointer is compared to test NullReferenceException as a cost before use. The situation is, there are many applications which still require performance strictly, but they were ignored during the fast development of CLR. From the years between Windows Server 2003 and Windows Server 2008 R2, the platform, API and development tools for unmanaged application had no equal improvement as CLR. This may not be a problem for Windows+PC platform, but you cannot expect people write mobile application with C#. The unbalance led little improvement on mobile and consumer areas for a long while.

WPF

WPF is a product with beautiful design. It solved 4 problems in traditional Win32 UI. 1) Win32 UI’s rendering is Window object and GUI based. WPF used a new model with dedicated rending thread, which improved rendering performance and was able to use GPU for acceleration. 2) Win32 depends on GDI and HWND object. For complex UI application which creates many handle, it is common to get out of resource error. Many developers used tab control to reduce the usage of handle to solve this problem. WPF does not use any native child window. The control concepts are defined at an upper layer. This design avoided the massive usage of native resources. 3) Win32 relies on many low level Win32 APIs, which are not convenient to use and are difficult to learn. WPF introduced Dispatcher class and BeginInvoke pattern, which simplified these issues. 4) The data, visual and code were not well separated in Win32. WPF used data binding, XAML and code behind to make the patterns clean. It is more easy to apply modern patterns like MVC and MVVM in WPF. All of these are the beauty of design.

The side effort of this was over design. WPF is based on CLR, and the over design kills performance. If you are familiar with WPF, you can find half of the design patterns in textbook be used in the dependent property. WPF is an encyclopedia of design patterns. In my calculation, there are about 7 different ways to hook up a mouse click event. The 7 ways were designed to use in different situations. It is good to be flexible, but the cost of flexibility is performance. The magic of patterns and data bindings is reflection in CLR. This is why you cannot get the same sleek feelings as native Win32 UI when a WPF application gets bigger.

The interoperability of Microsoft products

Products from Microsoft are always interoperable. C# is able to invoke C++, and vice versa. Any application could embed a browser control to use the function of IE. VBScript is able to automate office applications. Management tools like MMC and Explorer expose programming interface to allow user to plug in new functions. I saw customer use VBScript generate excel at backend, send it to client with ASP.NET, use JavaScript to edit it in browser, and send back to MSSQL backend. Interoperability protects customer investment, and the magic behind the interoperability is using COM as the bridge. This leads to three issues. First, it increases the complexity due to COM as discussed previous. Second, it hits performance with COM interop. Last, the development experience is not good enough because of the necessary ceremony. For example, when CLR interop with COM, the CLR engine needs to build RCW/CCW, transit between security model, and marshalling data here and there. When the call happens, you can even have a sense of performance hits by the noise caused by sector movements inside the HDD.

Win8

It’s time to discuss why I feel so good about Win8.

Win8 introduced Windows Runtime (WinRT). WinRT is an OS component, runs at user mode, between Win32 and the application, is intended to provide efficiency and friendly interfaces for developers. At the binary model, WinRT copied classic COM. There was no dependency between WinRT and CLR, and they could invoke each other. WinRT was implemented in C/C++. Performance was one aspect, and the other important aspect is projection. Projection is a new concept introduced by Win8. As the name says, it projects WinRT features to upper level programming languages to use in the most effective manner. The languages could be C#, C or JavaScript.

You can image projection as a new Windows API model. The classic API model was either DLL exported functions or COM interface. Both were not performant when invoking from CLR. The root cause is the incompatible memory and security models between CLR and unmanaged code. With projection, WinRT created the best way for upper language to invoke. This design avoided the cost and improved the programmer experience. Of course, this is also benefit by the development of Visual Studio tools, C# language and the CLR. For example, C# 5 introduced the await keyword, and WinRT introduced async operation pattern. Projection made the await statement to the WinRT async operation call without any redundant code. There was no CLR Engine involvement, no context switch, no marshaling and no proxies. It was just a simple assembly call to a stub from C# code, and then it jumped to the WinRT implementations. Please refer to Build conference to learn more about this. The callstack at the end is also vivid to explain this.

WinRT took the good part of COM as an efficiency and lightweight binary reuse model, and got rid of the complexity to be omnipotent. It let the upper programming languages and tools to achieve the flexibility. With projection, the performance issue got solved for interoperability. Microsoft changed the plan to use C/C++ to create a performant WinRT, and let CLR to do upper layer work. C/C++ is good at performance and bad with development cost. C# and CLR is opposite. Win8 makes the best of both.

This is why I think in Win8, different technologies are now in its best places. It is obvious that how Metro could fix WPF’s problem: The rendering model and XAML are continued in Win8, but the infrastructures and controls are moved to WinRT with native implementation. It fits for C and JavaScripts, and the upper level is flexible to use any design patterns.

Appendix

I attached some exemplary call stacks in Win8 here:

Stack1:

Windows_UI_Immersive!`Windows::Internal::CMessageDialog::ShowAsync'::`50'::::

Windows_UI_Immersive!Windows::Internal::CMessageDialog::ShowAsync+0x1a0

image08ee0000!DomainNeutralILStubClass.IL_STUB_CLRtoCOM()+0x8c

application1!Application1.MainPage+d__0.MoveNext()+0xcd

application1!Application1.MainPage.button_Click(System.Object, Windows.UI.Xaml.RoutedEventArgs)+0x80

stack2:

combase!CComActivator::DoCreateInstance+0x121

combase!CoCreateInstanceEx+0x51

combase!CoCreateInstance+0x65

Windows_UI_Immersive!Windows::Internal::CPopupWindowImpl::_TryToUnregisterForIHMNotifications+0x3b

Windows_UI_Immersive!Windows::Internal::CPopupWindowImpl::_HideWindow+0x31

Windows_UI_Immersive!Windows::Internal::CPopupWindowImpl::HideWindow+0x9

Windows_UI_Immersive!Windows::Internal::CPopupWindowImpl::DestroyWindow+0x24

Windows_UI_Immersive!Windows::Internal::CPopupWindow::Destroy+0x57

Windows_UI_Immersive!Windows::Internal::CClosePopupCommandHandler::Invoke+0xe 0x73

Windows_UI_Immersive!Windows::Internal::CPopupWindowImpl::_OnButtonPress+0xc0

Windows_UI_Immersive!Windows::Internal::CPopupWindowImpl::OnMessage+0x18b

DUI70!DirectUI::NativeHWNDHost::WndProc+0x73

USER32!InternalCallWinProc+0x23

USER32!UserCallWinProcCheckWow+0x100

USER32!DispatchMessageWorker+0x3d4

USER32!DispatchMessageW+0x10

Windows_UI_Immersive!SHProcessMessagesUntilEventsEx+0xe2

Windows_UI_Immersive!Windows::Internal::PopupWindowOperation::Show+0x103

Stack3:

Windows_UI_Xaml!HWWalk::RenderChildren+0x7a

Windows_UI_Xaml!HWWalk::RenderContentAndChildren+0x2d1

Windows_UI_Xaml!HWWalk::Render+0x61e

Windows_UI_Xaml!HWWalk::RenderChildren+0x7a

Windows_UI_Xaml!HWWalk::RenderRoot+0x1c4

Windows_UI_Xaml!CCoreServices::NWDrawTree+0x698

Windows_UI_Xaml!CCoreServices::NWDraw+0x1d6

Windows_UI_Xaml!CRenderTarget::Draw+0x13

Windows_UI_Xaml!CWindowRenderTarget::Draw+0x40

Windows_UI_Xaml!CXcpBrowserHost::OnTick+0x88

Windows_UI_Xaml!CXcpDispatcher::Tick+0x184

Windows_UI_Xaml!CXcpDispatcher::OnReentrancyProtectedWindowMessage+0x133

Windows_UI_Xaml!CXcpDispatcher::ProcessMessage+0xa4

Windows_UI_Xaml!CXcpDispatcher::WindowProc+0x69

USER32!InternalCallWinProc+0x23

USER32!UserCallWinProcCheckWow+0x100

USER32!DispatchMessageWorker+0x3d4

USER32!DispatchMessageW+0x10

windows_ui!Windows::UI::Core::CDispatcher::ProcessMessage+0xc7

windows_ui!Windows::UI::Core::CDispatcher::ProcessEvents+0x6c

Windows_UI_Xaml!CJupiterWindow::RunCoreWindowMessageLoop+0x3b

Windows_UI_Xaml!CJupiterControl::RunMessageLoop+0x1d

Windows_UI_Xaml!CJupiterControlLight::RunMessageLoop+0x8

Windows_UI_Xaml!DirectUI::DXamlCore::RunMessageLoop+0x15

Windows_UI_Xaml!DirectUI::ViewProvider::Run+0x11

twinapi!Windows::ApplicationModel::Core::CoreApplicationView::ViewProviderThreadProc+0x27

This is the end

从Win8回顾微软平台的各种技术

Li Xiong — Fri, 02 Dec 2011 22:17:00 GMT

我安装好Win8 CTP后做的第一件事情就是用调试器研究Win8各个组件的协作关系. 从我半天的研究结果看来, Win8真是一个让我爱不释手的产品. Win8里面涉及到的很多技术正好也是我的兴趣所在. 这篇文章简单回顾一下这些技术的变迁, 优缺点, 和对Win8的影响.

注意, 下面提到的对Win8的分析, 是基于公开的Win8 CTP来做的. 相信Win8面世的时候, 这些技术和细节, 都会发生重大改变. 所以这篇文章不具备实践上的指导价值.

COM -Component Object Model 通用组件模型

COM是上个世纪中期设计出来的伟大产品. COM旨在解决软件复用的问题. 在COM以前, 大家都是用代码级别的复用, 常见的就是C/C++的库, 无论是原代码库还是lib库, 都是需要编译后才能重用的. COM使得技术人员可以在二进制上进行复用. 从Win95, OLE32和Office95系列开始, COM就是微软平台上的一个技术基石, 无论是DirectX API, 还是最常见的剪贴板, 以及后来.NET Framework的host接口, 都离不开COM. 但任何伟大的产品, 都有局限的一面. COM在局限性在下面一些地方

STA/MTA/NTA等等线程模型过于复杂

线程模型, 特别是STA, 设计的目的是方便使用者. 但COM的线程模型严重依赖于太多系统组件, 比如Win32 Message, RPC和Windows系统服务, 使得程序员需要熟悉和了解太多系统知识才可以正确地使用线程模型. 否则用STA导致死锁简直就是家常便饭.

开发工具没有提供足够支持

COM和Visual Studio 6.0的关系, 就如同现在CLR2/VS2005, CLR3.5/VS2008和CLR4/VS2010的关系一样铁. 使用COM开发, 当时的选择要么是VB6, 要么用ATL. 这两者都有天生的局限. VB6适合企业开发, 特别是当时流行的MIS系统, 数据库系统这样的CS应用, 但是VB6不够灵活. 而且VB6里面由于缺少多线程支持, 无法用MTA的模型. ATL功能强大, 足够灵活, 但是使用起来特别复杂. 每次实现一个接口, 都要做一大堆C++的仪式性工作, 比如实现多重继承, 定义新的模板, 使用大量的C宏. 神经再粗大的程序员, 都经不起这样用C++的.

无止境地扩充到DCOM, COM+, DTC, MSMQ以及后来的.NET Remoting/WCF, 使得最后的复杂度无法控制

为了更好地适应企业级别的开发, COM被进一步延伸和演化成了DCOM和COM+. 所谓"企业级别", 其实是指对安全性和可伸缩性的更高要求. 经典的COM是一个进程内模型, 无法让不同的代码运行在不同的帐号下的, 因为同一个进程只能启在唯一的帐号下. 虽然通过impersonate等方法可以适当地解决问题, 但是为了让安全模型更为全面, 正确的做法是让不同的接口实现, 能够跨越进程甚至跨越机器等安全边界运行, 要能够赋予不同的接口不同的安全级别, 能够和域帐号集成, 支持不同等级的加密等等.远程通用组件模型, 也就是DCOM就这样诞生了. 读者可以尝试运行以下dcomcnfg.exe这个工具, 展开一些节点, 看看属性页, 就能体会到DCOM的功能是多么让人眼花缭乱了. 对于可伸缩性, 微软更上一层楼, 在DCOM的基础上加入了对象池和新的同步模型做成了COM+. 风靡十年的ASP, 就是运行在COM+框架下的最好例子. 更让人叹为观止的是, 微软把DTC和MSMQ的设计也和COM+模型绑定起来. 陌生的读者可能不了解DTC是个多么nb的东西. 简单说DTC是可以让程序员一行代码都不用写, 就让SQL Server和Oracle的数据库操作运行在同一个事务边界里面. 可以想象, 在MSSQL Server刚进入市场的时候, 微软推出这样的功能, 对于抢占Oracle的市场份额有多么重要. DTC和COM+当时成为了很多大公司的标准配置, 以至于后来设计.NET Remoting和WCF的时候, 都还是要考虑对DTC的支持. 这些强大而且复杂的功能, 让本来就复杂的COM更加恐怖. 这样的复杂度虽然也体现了当时的市场需求, 但由于缺乏配套的开发工具, 新的开发语言和更优美的抽象, 使得这条路最后越走越窄.

.NET Framework/CLR

在我眼中, CLR的各方面简直是无可挑剔的. 但可能正是因为CLR太好了, 让微软从2003年开始, 对unmamanged world的投资就不大了.

为了争取企业客户, 全力推广CLR是最正确的做法. 毕竟绝大多数的程序员, 一辈子都是和数据库, 和UI代码打交道. 你总不能让他们一辈子都用C去管理内容, 创建窗口句柄吧. CLR的性能也很有竞争力, 与之对应的编程语言和开发工具也非常给力, 每个新版本都带来长足进步. 但是, 这些再好, 也无法掩饰一个悖论: 要用用C#写出性能可以和C++一个数量级的程序, 不是不可能, 而是花费的代价往往比直接用C++写更大. 这里面有很多原因. 比如系统最底层的API还是unmanaged的, 通过CLR作interop的性能损失无法忽略. 比如用C#的话程序员的控制力很弱, 从工具和语言层面上很难对性能精雕细作, 一不注意就box/unbox了. 比如JIT编译器为了实现自己的安全模型和异常处理, 每次访问成员函数的都是都要生成代码确认this指针是否为空. 这个世界上还是有很多程序, 是需要对性能做严格控制的. 在CLR高速发展的几年中, 对应的系统平台, Win32 API, C++开发工具, 只有完善性的改善, 并没有重大突破. 这个问题对Windows平台本身影响不大, 但如果寄希望于在移动设备上, 大家都指着C#来开发, 就有点天方夜谭了. 这样的失衡, 使得微软在移动设备和消费者产品这两个严重需要unmanaged和系统投入的领域缓慢发展了很长时间.

WPF

在我看来, WPF是一个设计得很美的产品. WPF解决了传统Win32 UI程序的四大局限. 1) Win32的绘图是由各自Window元素独立控制, 基于GDI的. WPF引入了rendering thread来提高性能, 优化算法, 借用GPU加速. 2) Win32依赖于GDI Object, 在开发复杂窗口程序的时候, 很容易就遭遇资源泄露和资源不足. 比如早期的淘宝旺旺, 开到几十个窗口的时候, 程序就会出问题. 所以淘宝针对这个问题, 使用了统一控制台, 合并多个窗口到标签页的方法来解决. 而WPF只有最外面的窗口使用了Win32 Window和GDI, 内部的元素都是抽象成了WPF自己的元素, 不额外占用Win32 GDI资源的. 3) Win32窗体程序严重依赖Windows Message模型. 这要求程序员对系统知识有深入的了解. 而且Win32 API并不是非常利于使用, 比如要进行UI thread和Worker thread之间的通信, 往往需要和SendMessage这样的API打交道. 在WPF中, 引入了Dispatcher类和BeginInvoke方法, 把这些复杂问题抽象了. 加上CLR提供了更方便高效的开发环境, 使用WPF是很愉快的工作. 4) Win32缺乏数据, 设计和代码三者之间的模式抽象. 这三者在WPF中对应了数据绑定, XAML文件, 以及后台代码. 在WPF中可以更直观地使用各种模式比如MVC和MVVM. 这些都体现了WPF设计上的优美.

再优美的东西都还是有局限性的. WPF的问题在于过多的模式和对CLR过度的依赖. 了解WPF框架的人都知道, 就一个简单的dependent property, 就把设计模式这本书里面的模式用掉一大半了. 分析WPF框架代码的话, 简直就是看一本设计模式的百科全书. 我曾经统计过, 关于mouse click这样一个event回调, WPF里面有7种不同的实现方法, 分别各有好处, 旨在解决不同问题. 在这样高度灵活的背后, 牺牲的是程序性能. 无论是五花八门的模式, 还是最常用的数据绑定, 背后的主力都是CLR的reflection. 过度依赖于reflection导致WPF程序规模一大, 性能上就出问题. 就算再怎么优化, 也总找不到原生Win32程序那般流利的感脚. 使用reflection也体现了对CLR的依赖. 所以前面CLR的局限性, 也适用于WPF.

微软产品的互操作性

微软的产品线虽又长又多, 但是各个产品之间一定是能够互操作的. 比如C#可以和C++互相调用. 任何语言开发的程序都可以嵌入Browser Control来借用IE的功能. Office暴露了VBA接口, 通过VBScript都能够自动化Office程序. 各种管理工具无论是Explorer还是MMC, 都暴露了编程接口可以让程序员添加自己的功能. 我见过客户用ASP.NET在后台用VBScript生成Excel表格, 然后把表格嵌入在 IE浏览器中, 再使用JavaScript来帮助客户编辑, 最后提交回MSSQL数据库做完成报销功能的. 完善的互操作性充分保护了用户的投资, 使得客户对微软平台用一次就上瘾, 几乎没有不可能完成的任务. 仔细分析, 其实这些互操作有一个共性, 都是把暴露COM接口作为内部实现原理. 这个做法导致了三个局限性, 首先是牵涉到了前面提到的COM的复杂度. 其次是潜在的性能损耗. 最后是在具体开发的时候, 都需要一些仪式性的工作来引入或者定义COM接口, 使得开发过程不够自然流畅. 在CLR和COM互操作的调用栈里面, CLR的RCW, CCW, 安全处理, 列集拷贝等等, 耗费的时间带来的性能开销简直是可以到了肉眼可察觉的地步(听硬盘的声音和看任务管理器里面CPU的波动).

Win8

在讨论完这些技术背后的故事后, 再看看为啥我就对Win8爱不释手了.

Win8引入了Windows Runtime, 简称WinRT. WinRT是一个操作系统模块, 运行在用户态, 介于Win32的上层和应用程序的下层, 目的在于提供更高效友好的开发接口供Win8的程序员使用. WinRT在二进制模型上基本就是照搬了经典的COM. WinRT和CLR互不依赖, WinRT可以被CLR使用. WinRT通过C/C++实现, 效率高是一个方面, 更重要的时Win8引入了projection的概念, 就是可以把WinRT的API用最直接最高效的方法, 提供给上层的编程语言调用. 这个语言可以是C#, C或者JavaScript.

对于第一次接触Projection的朋友, 可以把Projection认为是一种新的Windows API模型. 传统的操作系统API, 要么是暴露DLL的方法, 要么是通过COM接口. 无论是哪一种, 在CLR中调用的时候都有不小的开销. 使用这些传统API的效率, 比调用一个C#自己的方法, 效率差了多个数量级, 根本的原因在于CLR的安全模型, 内存模型和传统的unmanaged模型不兼容, 所以跨越边界的调用需要额外的代码来处理. 而Projection提供的模型, 是在提供新功能的同时, 还针不同编程模型和语言, 提供了最利于它们调用的方法. 这样就主动避免了不同模型之间为了互相兼容导致的开销, 也使得程序员写代码的时候非常自然流畅, 调用的时候根本感觉不到和调用本地函数的区别. 当然, 能够实现这一点, 也是得益于CLR, C#语言和VS开发工具这十年的长足发展. 举个例子, C# 5.0中引入了await关键字, WinRT中引入了async operation. Projection技术把C#中的await语句转换为WinRT async operation的调用, 而且这个调用直接从managed code直接跳到unmanaged code, 中间没有任何冗余, 也不需要CLR Engine的介入. 进一步的信息, 可以参考Build大会关于WinRT的多个演讲. 后面的callstack也提供了直观的例子.

前面提到了COM的局限性在于一个轻量的二进制模型, 被硬生生的扩展成一个无所不能的框架. WinRT取其精华, 去其糟粕, 借用了COM的轻便, 舍弃了复杂性, 在扩展性上依托于上层的编程语言和工具. WinRT通过projection的技术解决了传统互操作性效率不高使用不方便的问题. 以前的路线是希望所有的产品和技术最后都统一到CLR上来. 现在修正为底层模型通过WinRT和C来实现, 然后把这一层高效的组件无缝提供给上层的开发技术比如CLR来使用. 这个转变重新重视unmanaged层面的二进制模型. 归纳为, unmanaged模型的优势在于执行效率高, 可以通吃所有场景, 缺点在于开发和使用成本也高. CLR的优势在于开发成本低, 缺点在于无法通吃各种需求. 现在微软自己用unmanaged来做WinRT, 然后把WinRT提供给上层语言, 这两者就可以取长补短了.

有了WinRT, 有了unmanaged的回归, 再加上微软开发工具和C#语言的长足发展, 前面介绍的各种技术在Win8里面就相得益彰了. Metro是如何修复WPF的缺陷就显而易见了: Win8的metro程序继续使用WPF中引入的rendering模型和XAML, 但是在control的基础设计和实现上, 从CLR转移到了C, 然后通过WinRT来暴露给使用者. 至于使用的灵活性, 比如要不要实现数据绑定, 就看上层使用者自己的选择了.

附录

下面是我研究过程中看到的一些Win8的callstack. 基于这些callstack和我以往的经验, 才写了上面的文章.