In pre-6.5 versions of Windows Mobile, the primary surface was responsible for all compositing which is no longer the case. The new compositing surface does not carry the orientation information of the primary surface so when processing the BMF_SRCPREROTATE flag we end up rotating the wrong surface. This results in camera preview incorrectly oriented and ultimately failing the test.

The fix for the problem can be found inside of AnyBlt() where you need to specify the assignment of pGPEDst while handling the BMF_SRCPREROTATE flag  - specify your primary surface as:

            GPESurf * pGPEDst = pGPE->PrimarySurface();

Instead of what is was previously assigned to. The location of the exact change will depend on if your driver implements a private copy of the function. It is also possible that your driver has undergone significant changes from the public copy, in which case I recommend understanding the change in regards to the public driver before integrating to your private one.

If you are using the default, public copy of this routine, then you should only need to re-link with a recent AK since it has been fixed in the public tree. Good Luck and happy hunting!

1.       Locate the Hopper.bak file, this is ‘backup’ log file that was written 5 minutes prior to the Hopper.log – often this file contains matching values and can be trusted.

2.       Read the persisting value from HKLM\SOFTWARE\Microsoft\Hopper\RunTime. This value is maintained during the Hopper run and will reflect the true time the test lasted and will persist from reboots

3.       Follow the instructions provided in the LTK to establish the correct runtime. This is the 'official' way to determine runtime and is the process used most often by 3rd-party testing organizations. Re-running Hopper will conveniently read the registry value above and prompt the user with this information – this is often easier than running registry tools.

Happy Hunting!

This API is documented in the OEM docs as well as: https://msdn.microsoft.com/en-us/library/aa450797.aspx. Most notably, the article includes sample code that will help get your display driver working under the new method quickly. Please make use of the sample, but I also tune the Allocator and de-Allocator functions to work specifically for your needs. I would recommend requesting a large VM allocation, but only commit the physical memory when needed and returning the physical when no longer needed.

The ability to try out bug fixes is a valuable part of the Hopper process and re-building images quickly is one of the easiest ways to improve your Hopper numbers. We recently went about trying to find the combination of build server parts that makes short work of our builds and wanted to share with the OEM community.

Step 1: Start with a fast quad-core computer with dual hard-drives. Doesn’t have to be the fastest processor group out there, but use something capable. Don’t worry about the size of the hard-drives.

Step 2: Buy two (2) 1-Terabyte SATA drives. The more expensive ones are rated for RAID, we went “on the cheap” and opted for the non-rated drives. You will also need to buy bay adapters and SATA cables for final connections. Hint: get the 90-degree angle connectors on the SATA cables.

Step 3: Connect the new drives per your manufacturer’s documentation. This step is intentionally vague since each build machine will have different configurations – please do you best. Enable SATA Raid in the Bios, reboot. Configure RAID-2 Array in BIOS, reboot. We found it best to stripe the two smaller drives as your C: drive and then stripe the 1TB drives as D:.

Step 4: Reload the OS making sure the OS is running from C. Install your Windows Mobile AK to the D: drive and build from there. Verify that all indexing programs are turned OFF. We have also seen some gains by pointing the TEMP directory to a RAMDRIVE.

Happy Building!

Using correct tools to collect relevant and sufficient data, and to analyze them not only makes the life a little easier for the engineers, it also helps making sure correct information can be gathered and correct conclusions can be reached, that would otherwise be missed if things were done only manually. It also shortens the investigation time and thus leaves more time to better assessing and implementing resolution alternatives.

Below, some of the important tools are discussed briefly that are available to Windows Mobile partners, with the goal of presenting where they are applicable/usable, how they can be located and available references for detailed usage.

CELog is an event-tracking tool that can log both a set of predefined kernel events (related to processes, threads, synchronization, memory, etc.) as well as application defined ones. It collects the data in a user-configurable location that can later be viewed using various tools. The data captured can be divided into “zones” to be able to filter out events that are relevant to the problem. More details can be found in the following pages:

-          MSDN documentation

-          Windows CE Base Team Blog

o   https://blogs.msdn.com/ce_base/archive/2005/11/30/a-tour-of-windows-ce-performance-tools.aspx

o   https://blogs.msdn.com/ce_base/archive/2006/01/23/516586.aspx

Perfman is an on-device tool that helps collecting CELog data on a Windows Mobile powered device without needing any connection to the device. It provides a simple user interface to make it easier to start and control the logging on the device. More details can be found in the following pages:

-          OEM documentation topic “Tuning the Platform for Optimum Performance”

-          Windows CE Base Team Blog

o   https://blogs.msdn.com/ce_base/archive/2005/11/30/a-tour-of-windows-ce-performance-tools.aspx

Even though this is obvious, it can easily be missed (if it is not being overused). For issues that originate from applications or components for which you have access to the sources, application level logging/tracing can be very powerful to help narrow down the problem area.

If a consistent repro of the issue is available, you have a good understanding of the possible components that the issue might be related to, and you have access to a KITL enabled device, live debugging using Platform Builder would be the best way to go. This is true even if you don’t have the correct symbols. Following pages provide good tricks that can aid in debugging using Platform Builder:

-          Resolving symbols manually on Windows CE

-          Windows CE Virtual Memory Layout for Debugging

-          Platform Builder Debug Symbols

-          Virtual Memory and Thread Stacks

Netlog is an on-device network packet sniffer (a.k.a. network monitor) that can be used to log network activity on the device. The output of Netlog can be viewed with popular desktop viewer tools like Microsoft Network Monitor, or Wireshark/Ethereal. More details can be found in the following pages:

-          MSDN Documentation

o   https://msdn.microsoft.com/en-us/library/ms883125.aspx

o   https://msdn.microsoft.com/en-us/library/aa926774.aspx

o   https://msdn.microsoft.com/en-us/library/aa922639.aspx

o   https://msdn.microsoft.com/en-us/library/ms886701.aspx

-          Windows CE Networking Team Blog

o   https://blogs.msdn.com/cenet/archive/2004/10/11/getting-network-captures-on-windows-ce.aspx

RIL Proxy is a component that is part of the Radio Interface Layer (RIL) architecture. Besides its main function, it also serves the purpose of being able to log radio events that originate either from the radio or from the operating system. RIL Proxy logs can be very useful especially for troubleshooting radio or connectivity related issues. More details can be found in the following pages:

-          MSDN Documentation

-          Windows CE Networking Team Blog

o   https://blogs.msdn.com/cenet/archive/2005/09/27/474650.aspx

-          Hopper Blog

o   https://blogs.msdn.com/hopperx/archive/2007/07/26/using-the-radio-interface-layer.aspx

RapiConfig is a desktop configuration tool, part of the Windows Mobile 6 SDK, which allows applying a provisioning XML on a device/emulator connected using ActiveSync. It can be used both to make quick configuration changes (e.g. change Connection Manager settings, make registry changes, etc.), and to capture current configuration on the device (e.g. read Connection Manager settings, read a registry value, etc.).

Usage of RapiConfig is controlled by security settings on Windows Mobile which means that any “RapiConfig request” could be rejected if RAPI API usage is disabled (look for policy 4097 in the “OEM Documentation”). In such a case security configuration of Windows Mobile device must be changed before using RapiConfig tool.

More details can be found in the following pages:

-          MSDN Documentation

o   https://msdn.microsoft.com/en-us/library/aa454232.aspx

o   https://msdn.microsoft.com/en-us/library/ms889520.aspx

o   https://msdn.microsoft.com/en-us/library/aa924367.aspx

o   https://msdn.microsoft.com/en-us/library/ms889589.aspx

o   https://msdn.microsoft.com/en-us/library/30dtsstx.aspx

o   https://msdn.microsoft.com/en-us/library/bb384149.aspx

-          MSDN Blogs

o   https://blogs.msdn.com/marcpe/archive/2005/01/18/355158.aspx

-          OEM Documentation

o   “Configuration Service Providers”

This tool, which runs on Windows XP/Vista and which is also part of Windows Mobile SDK, can generate a CPF file (which is special CAB file with only settings but no files) from a provisioning XML. The CPF file can be put on the device and run to make the actions (read or change some configuration on the device) defined by the provisioning XML it contains.

Installation of a CAB/CPF file is controlled by security settings on Windows Mobile which means that CAB/CPF file installation could be rejected (look for policy 4101 in the “OEM Documentation”). In this case the file(s) have to be signed with a trusted certificate that is already installed on the Window Mobile device (provided by OEM or Mobile Operator).

More details can be found in the following pages:

-          OEM Documentation

o   “How to Create a .cpf File”

o   “Cab Provisioning Format (CPF) File”

o   “Packaging the XML File for Delivery”

-          MSDN Documentation

o   https://msdn.microsoft.com/en-us/library/ms889557.aspx

o   https://msdn.microsoft.com/en-us/library/aa455993.aspx

Platform Builder comes with various remote tools that target specific troubleshooting/debugging scenarios. OEM documentation includes comprehensive documentation about each of these tools. Some of these topics in the documentation are listed below:

-          “Remote Tool Connectivity”

-          “Remote Tools for Debugging”

-          “Tools for Performance Tuning”

-          “Remote Tools for Information Management”

CEDebugX is an extension to Platform Builder that helps collecting system information from a device, to better understand the state of the device at a given time. More details can be found in the following pages:

-          MSDN Documentation

o   https://msdn.microsoft.com/en-us/library/bb509784.aspx

-          MSDN Channel 9

o   https://channel9.msdn.com/posts/mikehall/Using-CEDebugX-with-Windows-Embedded-CE-60-SP1/

This mechanism (a.k.a. “Windows Error Reporting” or “Watson”) takes a snapshot of the system memory and system state upon a system exception (e.g. memory access violation, division by 0, etc.). It is by default installed in all Windows Mobile devices and can be turned on or off using “Error Reporting” settings UI or alternatively via using some registry keys.

When “Windows Error Reporting” is enabled, you should take care that as soon as the Windows Mobile device is connected to a PC via ActiveSync, it will by default try to remove the dump file from the device and to upload it to “Windows Error Reporting” server. This behavior can be turned off using registry keys. If you want to get the dump file, you should either disable the upload mechanism or copy the file before connecting the device to ActiveSync. More details can be found in the following pages:

-          OEM Documentation

o   “Error Reporting Overview”

o   “Using CELog and Watson to Debug a Driver Exception in a Retail Device Without KITL”

o   “Generating A Dump From A Handled Exception”

o   “Exception Mode Error Reporting”

-          MSDN Documentation

o   https://msdn.microsoft.com/en-us/library/bb905581.aspx

o   https://msdn.microsoft.com/en-us/library/aa935972.aspx

o   https://msdn.microsoft.com/en-us/library/aa935583.aspx

o   https://msdn.microsoft.com/en-us/library/aa935778.aspx

-          MSDN Blogs

o   https://blogs.msdn.com/hopperx/archive/2005/10/07/getting-help-from-the-doctor-dr-watson-that-is.aspx#478380

o   https://blogs.msdn.com/hopperx/archive/2005/10/12/not-all-watson-dumps-are-created-equal-watson-part-ii.aspx

In addition to Just in Time Debugging capabilities, Platform Builder also provides post-mortem debugging capabilities via an extension called “Post Mortem Debugger”. It is used to debug Watson dump files using Platform Builder, as if what is being debugged is a live device under debugger. More details can be found in the following pages:

-          OEM Documentation

o    “Post-Mortem Debugging”

o   “Types of Crash Dump Files”

o    “Using CELog and Watson to Debug a Driver Exception in a Retail Device Without KITL”

o   “Generating A Dump From A Watched Process”

o   “Capturing a Dump File on a Standalone Device”

o   “Capturing a Dump File While Debugging”

 Jumpstart Guide to Troubleshooting on Windows Mobile – Part I

Every Windows Mobile device goes through a full software project cycle, before it can be commercialized, and made available to mobile operators and is working in customers’ hands. In this cycle, development, customization and testing phases play a crucial role in achieving partner and end user satisfaction, and in making sure Windows Mobile phones are of high quality.

Like in any other project, during these phases, often various technical problems are discovered related to various aspects of the device and the experience it offers. In terms of their effects in overall device quality and project schedule (i.e. time to market), these issues or challenges can range from insignificant ones to device blockers. Needless to say, regardless of their size or effects, all of these issues need to be attacked by engineers and experts, to identify and implement effective, timely and high quality resolutions.

Troubleshooting (i.e. investigation of issues’ root cause for the purpose of removing it) is the first step in reaching an acceptable resolution for any issue we are challenged with during device/application development. It is more accurately a series of steps that make sure issues are correctly identified with all their aspects, so that effective, timely and high quality resolutions can be reached. To achieve that, using sound strategies, techniques, and tools is crucial. Without the correct approaches and tools used, troubleshooting could take more time than it actually requires or is available, or even worse, could lead to incorrect resolutions.

This article will present general guidelines and strategies that should be employed while troubleshooting Windows Mobile device issues. It is the first part of a series of articles I’d like to do. In the second part of this series, I will provide an overview of the tools that are available to make life a little easier for engineers.

Using sound strategies while approaching a problem and analyzing it directly effects how sound the resolution at the end will be, how long it will take to reach to it, and how painful the process will be. The strategies briefly discussed below are actually general in nature, and can be applied to any development problem, but still act as the basis for our purposes.

The issue in question might be found by the same engineers who  will be troubleshooting it, or by some other team/partner. Regardless of the source, getting a clear picture of the problem at hand is the most important point.

Asking and finding the answer to the following questions would be helpful to better understand the problem:

-          Which version of the device is the issue reported/found for? The version applies to WM version, BSP version, radio version, etc.

-          What is the expected behavior?

-          What is the actual behavior?

-          Did the issue exist in previous versions?

-          Does the issue occur only on one device (a particular device), one set of devices (with same hardware, WM version, BSP version, configuration and customizations), or all devices with the same hardware and WM/BSP versions?

-          Is the reported issue an isolated one? This means finding out whether a similar problem (e.g. connectivity, etc.) exists on the device on another area or use case that might not have been reported initially, but might be related.

-          What is the use case and the expected user experience? Although this is closely related to the “expected behavior” mentioned above, in some cases it might be completel different. An example is the case where the expected behavior might be related to a part of the user interface, but the actual expectations underlying this might be completely different (which would cause us to look at the problem from a completely different angle).

Even if it is the same engineers who reported the issue and working on it, it is always a good exercise to write down the steps to reproduce the problem in detail (commonly referred to as “repro steps”). Having this is especially important, if the issue is being reported by another team, or partner.

Repro steps should at the minimum include the following data and characteristics:

-          The pre-conditions that existed before the repro is done. This can include the same information that we mentioned above for understanding the problem, or some other relevant data specific to the issue itself (e.g. is there radio connectivity, is  the SIM used, what is the meta network used, what are the applications that were running on the device, etc.)

-          What are the steps, in order, that were done until the issue occurred? Supporting these with screen captures from the device, and including the user interface (UI) elements’ names as they appear on the device, would be more than helpful.

-          What was the result? Supporting this with data related to the result, e.g. screen captures, error message texts/numbers, etc. would be very helpful.

-          Is the repro consistent? What s the failure rate?

Although not always possible, one thing that would ease the analysis of the issue is getting the same repro on a device you have access to. This can reveal a lot of new info, especially the ones that you might not have gotten when the issue was reported.

Indirect and direct factors can include a lot of things. They are the data points that can give you important clues on the environment and pre-conditions on the device when the issue occurred.

Direct factors can simply include the configuration on the device relevant to the issue at hand (e.g. connection manager configuration for a network issue, the active theme for a UI issue, etc.), or they can be the pre-conditions for the repro that might not have been mentioned before (e.g.  the initialization API’s used before the failed API, the radio state for a connection issue, available memory for an issue involving processing a large amount of data, etc.).

Indirect factors are similar to the direct ones, which at first look don’t seem related to the issue.  Collecting this data can prove itself useful for some issues, especially if there is a chance a different view on that data could reveal some indirect relationship to the issue (e.g. another application running on the device holding a lock on the same file, another application keeping the radio busy causing your connectivity issue, etc.).

Attacking a problem effectively always require knowing the problem surface. Some issues require a deep level of knowledge about the problem domain, while for some issues, a general familiarity with the workings is sufficient. If you feel there are things that you can’t explain in the problem, or don’t know enough to collect relevant data or analyze them, take a look at the available documentation, and if possible, find an expert who can quickly demystify things for you.

To make sure the investigation has the chance to reveal useful results, all possible data that relate to the issue should be collected. Since the data should be relevant to the issue and should be sufficient (not too little or too much), a good understanding of the end-to-end scenario as well as the underlying modules/architecture/inner-workings is almost a must. An example is, for a connection issue, it is almost always useful to collect the Connection Manager configuration on the device as well as a capture of the network traffic. Another example is collecting Radio Interface Layer (RIL) logs for an issue that involve radio connectivity.

Collecting these logs will require knowing and having access to the right tools for the job. Some of these tools will be discussed in the second part of this series.

Once all data is collected, analysis phase also require using the right tools, which can convert your raw data to useful information and clues about your issue. There are a variety of tools available for Windows Mobile, which will be discussed later.

Having access to an unlocked device can save you a lot of time, when you need to collect data from the device. Having a KITL-enabled image on the device, on the other hand, is the ideal situation that would allow you to connect Platform Builder to the device and run the repro under a debugger.

The goal of the data collection and analysis is obviously to find the root cause of the issue as well as a resolution. But since in most cases, it won’t be possible to reach this goal right away, intermediate goals are required, where the problem area is narrowed down gradually and isolated from all other irrelevant factors one step at a time. At each step, new data might need to be collected. As the problem is isolated more and more, the problem area will get more manageable and have a better chance to leading to the root cause.

While trying to understand the problem, to identify different factors that are in play and to gradually isolate the problem, the need for tracing back to a previous step and re-doing some steps is sometimes unavoidable. But to improve this, a good strategy is to review the assumptions made (consciously or not) and to evaluate them, so that unnecessary trace backs are avoided.

Especially for tough problems, where either it is never enough to collect sufficient data, or to have a deep-enough understanding of the factors in play, it might be tempting to cut some shortcuts and eliminate alternatives by jumping to conclusions. Although not completely avoidable especially for cases where trial-and-error (the ones that are done based on sound and informed assumptions/decisions) is the only way left to go, for most issues, considering all available/seen alternatives and having high degree of confidence for all the conclusions, will make sure a possible rework is eliminated. One principle that is always useful to aid in this is to separate facts or observations that are based on concrete data collected, from possibilities and conclusions that are not yet proven.

For some issues, troubleshooting might end up identifying bigger underlying problems. For those cases, it is important to evaluate the cost of the resolutions offered with the project timeline and time-to-market requirements. If the proposed resolutions are costly, the investigation might need to re-focus on identifying cheaper alternatives for employing in the short term, as well as the longer term resolutions.

It is targeted as an “OS” book, so if you are curious about VM, threads and other nuts & bolts it makes for a real page-turner.

However, if you are intentionally including the “W” (write) attribute on a code section and your DLL is greater than 64kb, then you may be affected. Luckily, dumpbin.exe already knows to display a warning:

C:\WM612\release>dumpbin myDll.dll

…

…

60000020 flags

         Code

         Execute Read

DUMPBIN : warning LNK4078: multiple '.text' sections found with different attributes (E0000020)

SECTION HEADER #2

…

…

E0000020 flags

         Code

         Execute Read Write

There is nothing programmatically wrong with the above and most modules will work without modification. However, if you are seeing some strange results in recent versions of Windows Mobile, you may want to try and restrict your module to Slot 0 by adding the following registry key:

[HKEY_LOCAL_MACHINE\System\Loader\LoadModuleLow]

    "MyDll.dll"=dword:1  << change DLL name to match

Please note that using the above key will force your entire module into Slot 0 and prevent other modules from loading in that slot. The above registry setting should be used with caution and only for modules with a writeable code section.