The topic is about the screen resolutions and orientations supported by the display driver. It is really a simple thing, but there are a set of rules that driver writers must know to get it right. When it is not correctly followed, your device might still boot, but you may start noticing weird glitches here and there.

A GPE-based display driver has to override a function called GetModeInfo() which is a pure virtual function defined by the GPE base class. What it does is very simple; It just returns the screen resolution (and some extra information such as bit depth) supported by the device at the given index. It is used in a combination with NumModes() which gives the number of modes that the GetModeInfo() supports. What is important here is that the display driver framework requires that the GetModeInfo() returns the screen resolution relative to the current screen orientation. Not the native screen orientation. Not the boot-time screen orientation. It is always the current screen orientation.

The screen can take four possible orientations; 0 degree, 90 degrees, 180 degrees, and 270 degrees. Unless you are only dealing with square displays, then depending on the angle, the width and height have to be swapped. For example, if your screen resolution is 800 x 600 at the 0 degree and that is how your GetModeInfo() reports, then it has to return 600 x 800 when the current orientation is at 90 or 270 degrees. Many GWES components that deal with screen resolutions and orientations are dependent upon this behavior, but this doesn't become a problem until you actually try to rotate the screen. So, initially it may look like it is working perfectly, but once you try to rotate it, then you start seeing some issues.

As far as I can think of off the top of my head, most APIs (if not all) that deal with resolutions such as ChangeDisplaySettingsEx() and SetDisplayMode() all expect resolution parameters to be relative to the current screen orientation. So, the rule of the thumb appears to be this: Always think in terms of the current screen orientation.

We recently got a new codec library drop which we integrated into our mainline code tree. The codec team spends alot of time developing optimized ARM-versions of windows media codecs, and every once in awhile we get a new library that we need to integrate into our build system.

When we checked the libary into our source tree and ran a Smartphone build, we got roughly this error from one of our build tools:

    wmvdmod.dll(0) : fatal error RM0024 : Input File has more than 16 sections

In the ensuing investigation we discovered two things we hadn't previously known:

1. Our codec team has been subdividing their C/C++/Assembly language routines into multiple sections to keep certain code paths together and improve cache/page hit rates. As a result, they had created about 14 extra sections with names like ".decodeX_Pass1" (names changed to protect the innocent ;-). In general, one can view this type of information for any lib or dll by running "dumpbin -headers" on it.

2. Windows CE has some limitations on the number of sections that a module can contain (due to design decisions in the kernel and ROM image filesystems). Ultimately this results in a limit of 16 sections for some scenarios, which is the case we hit in our build tools.

The simplest short-term solution to this problem was to use the merge linker directive to force the linker to merge the different sections in the library back into the .text section. To accomplish this, we added something like the following to the appropriate sources file. This solved the build error without the need to rebuild the library (at the expense of removing all the goodness of using multiple sections to control code placement).

LDEFINES=$(LDEFINES) \
 -merge:.decodeX_Pass1=.text \
 -merge:.decodeX_Pass2=.text \
 -merge:.decodeY_Pass1=.text \
 -merge:.decodeY_Pass2=.text \
...

Note: I'm told one can accomplish the same feature within c/cpp files using #pragma comment(linker,"-merge:.foo=.bar")

In the ensuing discussion of how to fix this in the correct way (e.g. removing the restriction on the number of sections, or using fewer sections in the codec lib), our compiler/linker guru came down firmly on the side that there's no reason to need more than 16 sections (or really more than four or five), and noted that this whole situation could have been easily avoided using the following techniques:

For performance, if you want page alignment, use __declspec(align).  If you need to control code layout, use the linker’s /ORDER switch with a file containing the symbol ordering you need.  Alternatively, use the linker’s automatic sorting of section suffixes, e.g. .text$FOO_A, .text$FOO_B, and .text$FOO_C are automatically merged with .text in alphabetical order.

We didn't previously know about the linker options to automatically sort and merge sections using the $ delimiter, and I suspect that most other people don't either. We'll now go back to the codec team and suggest that future drops can just use the automatic sorting mechanism to ensure that code is grouped as needed while keeping all the code in the .text section. As a nice side benefit, grouping code into the same section saves on the amount of ROM required for the code. Each section must start on a 4k boundary, so on average each section will waste 2k or ROM. Note that section names are case sensitive, so .TEXT is not the same as .text. 

Here are some other related details about sections which I've shamelessly stolen from some other developers here at MS:

Paging:

Code may be paged into a size-limited RAM buffer called a "page pool". The page pool helps limit the RAM impact of code by keeping resident only the code pages currently in use. Code that must always stay resident in RAM can be marked as non-pageable, but this will cause the full extent of that code section to be copied into RAM for as long as the module is loaded.

To limit the footprint of a module in the page pool, it’s best to group the functions and constant data that are in the working set together.  This will allow the working set of code to exist in the page pool in the smallest number of pages. You can group them together using custom section naming. If section names are unique they will each be page-aligned (4k), so unless they truly need unique attributes, it’s best to name them such that automatic section merging can take place. Automatic section merging happens on sections named using a “section_name$subsection_name” convention, such that they all merge into one section named “section_name”.

For readability, give the subsection a name related to the grouping reason, such as “initialization”, “debug”, or “core”.

Example

To group function1 and function3 together in a custom subsection, you can do the following.

Non-Pageable Sections

If you need only a small bit of the code to stay in RAM always for performance or reliability reasons (like time-critical driver code), you can make the module partially pageable by creating a completely new section with custom attributes.

The following pragma defines a section called "NonPageableCode" which is set to non-pageable.

There is also an newer, more readable way of specifying the section properties which has been available since CE5:

Now, in the source code, to make a section of code non-pageable, put the following line before the code:

Afterward, you may use the following line to force following code to be placed back in the default .text section:

Tools

DUMPBIN /HEADERS FOO.DLL (to see what sections exist in the module)

That's it for now.

A DLL Forwarder is used if you want to export an entry point from one DLL (or, more likely, for historical purposes you've already exported it from one dll), but you want to actually implement it in a different DLL.

For example, suppose you want to implement a function Foo() in a DLL I'll call impl.dll, but for whatever reason you need to export it from a DLL I'll call export.dll. Of course, one simple solution is to add code to export.dll to explicitly call into impl.dll. However, there are a couple of issues with this:

- There's an additional function call/return overhead.

- If the functions exported from the two DLLs have the same name, you might have some trouble convincing the linker to call the the Foo() in impl.dll from the Foo() in export.dll.

A forwarder solves this problem by directly interacting with the loader to forward exports from one DLL to another DLL without actually adding anything to the code path.

Forwarders are implemented in the .def file of the DLL you're forwarding through, and have the following syntax:

EXPORTS

    <FuncName>=<ForwardedDll>.<ForwardedFuncName>|#<ForwardedOrdinal> @<FuncOrdinal> NONAME

Where: 

<FuncName> is the name of the function as exported (e.g. from export.dll).

@<FuncOrdinal> is the ordinal of the function as exported (e.g. from export.dll). Note the use of the '@' symbol. Optional.

 

<ForwardedDll> is the name of the DLL into which you're forwarding the call (e.g. into impl.dll). Optional; if not specified, the forwarded function is assumed to be in this dll.

<ForwardedFuncName> is the name of the function in the ForwardedDll.

#<ForwardedOrdinal> is the ordinal of the function in the ForwardedDll. Note the use of the '#' symbol.

 

NONAME is the keyword that causes the linker to throw away the name of the function you're exporting so that it can only be referenced by its ordinal. This saves some space in the DLL and forces all callers to use the ordinal to link or GetProcAddress on the function. Optional.

Note: You need to specify the forwarded function name or ordinal (not both). You'll get slightly better load perf and smaller code size by specifying the ordinal. The ordinal is also necessary if the function in the DLL you're forwarding to is specified as NONAME in its def file.

Using our example, to forward Foo() from export.dll to impl.dll, the export.def file would have a line that looks like this:

EXPORTS 

    Foo = impl.Foo

If you run "dumpbin /exports" against export.dll, you should see an entry for showing the forward that looks something like this:

    ordinal  hint   RVA      name
    nnnn    mm                Foo (forwarded to Impl.Foo)

Tricky detail 3: Forwarders are used at load time to forward references between DLLs. The linker will not use forwarders at link time to resolve links within your DLL. Therefore, if the DLL you're forwarding from includes code with references to the function you're forwarding, the link phase of you DLL will fail with an unresolved extern error.

Tricky example: If you want to export it a function at ordinal 1000 (and not export it by name), and want to forward it to  DLL which exports it at ordinal 2000 (without a name), the syntax in your .def file is:

EXPORTS

    SHIM_ORD_1000=IMPL.@2000 @1000 NONAME

The SHIM_ORD_1000 is an arbitrary name; it's only there to satisfy the def file syntax rules. It doesn't really matter what you call it as long as it doesn't alias to anther function exported in your .def. If you then run dumpbin /exports on the resulting dll, you'll see something like:

    1000 [NONAME] (forwarded to IMPL.@2000)

About Reference Clocks and Stream Time

All fiters in a filter graph are synchronized to the same clock, the reference clock. The stream time is based off the reference time, but it is relative, and depends on which state the graph is. For instance, stream time doesn't move when the graph is paused; stream time goes back to 0 after a seek.

DirectShow provides a base class CBaseReferenceClock that implements the IReferenceClock interface. The base class clock object maintains two times internally:

• internal private time
• reference time

The internal private time is the actual time kept by the clock, and can be accessed through GetPrivateTime(). The internal private time can go backwards for brief periods of time. The reference time is based off the private time, and cannot go backwards.

Whenever a filter provides the reference clock, it will usually inherit from CBaseReferenceClock. It can either override the GetPrivateTime() function to return directly the time from the device (if available), or it can issue adjustments to the stream time through the SetTimeDelta() function. If it chooses the second method, it will need to monitor the difference between the system time and the time provided by the device.

The default reference clock in WindowsCE is provided by our audio renderer. It uses the SetTimeDelta() method to issue adjustments to the stream time. To change the reference clock in a filter graph, the interface IMediaFilter needs to be queried from the filter graph. Then use SetSyncSource() to change the reference clock.

All filters can access both the reference time and the stream time. The base filter class CBaseFilter has a m_pClock member. The reference time can be accessed by doing m_pClock->GetTime(), and the stream time is a member function of CBaseFilter, StreamTime().

About Time Stamps & Stream Time

The samples being processed in the filter graph may or may not have a time stamp, which is the media sample start and finish time. The time stamps are used in conjuction with the stream time. If a sample has a time stamp that is greater than the current stream time, it means that the sample is early. If a sample has a time stamp that is smaller than the current stream time, it is late. In a playback scenario, usually a splitter is the one that attaches time stamps to the samples. Filters may use time stamps for different reasons. For instance, time stamps may be used for presentation purposes, or to control the amount of buffering. The video renderer will use time stamps to schedule the samples for presentation, and thus, will end up throttling the video playback pipeline. When a sample arrives at the video renderer, there are several possibilities:

• no timestamp - sample is scheduled immediately
• in the future (timestamp > stream time) - video renderer needs to schedule the sample, and will usually call m_pClock->AdviseTime()
• in the past (timestamp < stream time) - may render immediately, or not render at all.

The Reference Clock & The Audio Renderer

Let's assume in this section that the default Windows CE audio renderer is the reference clock. The audio renderer uses time stamps and stream time in a different way. Being the reference clock implies that the stream time is controlled by this component, so it will not follow a behavior similar to the video renderer.

In this case, as soon as the audio renderer receives a sample, it is ready to send it out to the audio driver. If the sample is late, it will drop it. Otherwise, it will send them immediately if there's buffer availability from the audio driver. It will never wait for the time to be right. For cases where the media sample times are not contiguous, that is, the end time of a sample is smaller than the start time of the next sample, then the audio renderer will write silence to the driver, and will wait until the start time for the second sample has arrived. In the normal scenario, there is going to be no space between media samples, so the audio renderer will write samples as fast as it can.

When the default audio renderer has finished processing a sample, it will read the device clock and the system clock, and compute the difference between them. Unfortunately, there is no way to get to the device clock directly, so the audio renderer uses the amsndOutGetPosition(), which can be imprecise. The audio renderer will accumulate differences, and will use a low pass filter on these differences. Whenever the average difference has gone above a certain threshold, then it will issue an adjustment to the stream time, through the usage of the SetTimeDelta() function. As soon as it does that, all filters calling GetTime() will receive the adjusted time - so the stream time will not be continuous. Note that all other filters that used m_pClock->AdviseTime() to get notified when a certain stream time has arrived (such as the video renderer) will not have to know of the stream time "change" that happened because of an adjustment. They will be advised when the stream time reached the desired value.

Live Sources & Clock Slaving

If the default Windows CE audio renderer is not the reference clock, it will write samples as it receives them. There's no automatic slave mode in WindowsCE, so the audio renderer will not wait for the time to be right before it sends the next sample.

For the case of live streams, there is one interface in our audio renderer that causes a speed up or slow down in the audio driver so that we try to match against the live source. The source filter will usually be using the IAudioRenderer->SetDriftRate() to control the audio matching speed. In this case, the audio renderer continues to be the master clock.

Another possibility when the audio renderer can't be the master clock is to simulate a slaving mode by inserting a filter in front of the audio renderer. This filter's responsibility would be to throttle the samples, so that they will be delivered just when it is almost time to send them to the audio driver. Of course, more complicated schemes are possible to try to match the rate of the incoming samples, but we will not go there here...

I've been working on a wavedev2 porting guide over the last few weeks and decided that it's better to post what I've go so far rather than wait until it's what I would consider finished. Expect future updates/additions as time allows, and feel free to ask for specific information in the comments. 

Overview

This whitepaper gives an overview of porting the wavedev2 audio driver to new hardware. For additional background on the history and features of wavedev2, please refer to other articles on http://blogs.msdn.com/medmedia.

Like most software, each release of Windows CE includes new features and bug fixes. For this article I’ll be referring to the version of wavedev2 which shipped with Windows CE 6 (AKA Yamazaki) under public\COMMON\oak\drivers\wavedev\wavedev2\ensoniq. This version is backward compatible with previous versions and the porting process is comparable. I’ve included some notes on the differences between Windows CE 6 and previous implementations at the end.

File Layout 

All the files needed to build wavedev2 are in a single directory. For porting purposes, these files can be grouped into the following categories:

1.       Files which are device independent, and which you should not need to touch during the porting process beyond just copying them:

audiosys.h: Proprietary wave message definitions used by wavedev2.

devctxt.cpp: Implementation of device context class.

devctxt.h: Definition of device context class.

input.cpp: Implementation of audio input streams.

makefile: Used by build system

midinote.cpp: Implementation of tone generator.

midistrm.cpp: Implementation of MIDI stream and MIDI parser.

midistrm.h: Definition of MIDI note and stream classes.

mixerdrv.cpp: Implementation of Mixer API classes (* may need to change if you want to take advantages of mixer API extensions).

mixerdrv.h: Definition of MIXER API classes.

output.cpp: Implementation of audio output streams.

strmctxt.cpp: Implementation of base audio stream class.

strmctxt.h: Definition of base, input, output stream classes.

wavemain.cpp: Device driver interface.

wavemain.h: Common include header used by all source files.

wavepdd.h: Basic PCM sample definitions.

wfmtmidi.h: MIDI structure definitions.

2.       Files which are device dependent but are logically part of the wavedev2 driver infrastructure. These files will need to be copied and modified during the port. These files are:

hwctxt.cpp: Implementation of hardware context class.

hwctxt.h: Definition of hardware context class.

oemsettings.h: HW-specific definitions used by hw-independent code.

sources: Used by build system.

wavedev2_ensoniq.def: Driver exports. Probably just need to rename.

wavedev2_ensoniq.reg: Registry entries used to install driver.

3.       Files which are device dependent and are not logically part of the driver infrastructure. You can ignore these files in your port (unless they happen to be appropriate to your hardware). For the Ensoniq sample driver, these are:

AC97.H: AC97 codec-specific definitions.

Es1371.cpp: Ensoniq 1371-specific functions.

Es1371.h: Ensoniq 1371-specific header.

Hw_ac97.cpp: AC97 codec-specific functions.

In the above list, note that the vast majority of files shouldn’t need to be touched, and there is really only one source file and two headers you will need to modify to bring up a driver.

Class Descriptions

Class Overview

The wavedev2 driver largely consists of three main base classes:

HardwareContext

This class represents the actual audio hardware. This is the only class you will typically need to modify to port the driver. It is the only device dependent class, and takes care of hardware initialization, power management, DMA and Codec control, handling audio interrupts, and any other proprietary features the driver may implement. There is one instantiated HardwareContext object in the driver which is pointed to by the g_pHWContext global variable. I’ll go into detail about each HardwareContext’s methods later.

DeviceContext

This class represents a specific audio device. You should not have to modify anything in this class to port the driver. There is a DeviceContext virtual base class, from which are derived an InputDeviceContext and an OutputDeviceContext. A typical wavedev2 audio driver (such as the Ensoniq driver) implements a single input device represented by an InputDeviceContext; and a single output device represented by an OutputDeviceContext. In the Ensoniq sample, these objects are directly embedded within HardwareContext class as member variables.

DeviceContext methods include:

StreamContext

This class represents a specific audio stream. You should not have to modify anything in this class to port the driver. There is a StreamContext virtual base class from which are derived a variety of stream classes for various flavors of PCM audio and MIDI data. Each stream is associated with a specific device context. This association is implemented as a linked list of stream contexts hanging off of each device context. In addition, each stream context includes a pointer back to its associated device context.

The class hierarchy is roughly as follows:

StreamContext

        CMidiStream 

        WaveStreamContext

                InputStreamContext

                OutputStreamContext

                    OutputStreamContextM8

                    OutputStreamContextM16

                    OutputStreamContextS8

                    OutputStreamContextS16

The reason for the multitude of output contexts is that the mixing/sample-rate-conversion code on the output side is optimized for each type of PCM data (Stereo/Mono, 8/16-bit samples). This avoids some tests in the inner loop. The same optimization wasn’t done for the input side (input isn’t typically used as often as output, and the code is a little simpler).

StreamContextMethods include:

Porting HardwareContext

In this section, I’ll go over each of the methods in HardwareContext and describe what they do.

HardwareContext::CreateHWContext

This is a static method which is called during driver initialization (from the WAV_Init code in wavemain.cpp). This function should create and initialize the global g_pHWContext with a new HardwareContext object and call g_pHWContext->Init. You probably won’t need to change this function, as most changes will be in the Init method.

HardwareContext::Init

This method is only called by CreateHWContext, and is where initialization of the Hardware is typically implemented.  Portions of this function may need to be modified for new hardware. Its role is to initialize any hardware, allocate DMA buffers, and startup the interrupt service thread. In addition, during initialization it needs to call into some of the device independent sections to initialize them; specificially:

-          Call SetBaseSampleRate on each device context to tell it what sample rate the hardware is running at. Note that these functions can be called at any time to tell the device context that the hardware sample rate has changed, but for devices with a fixed sample rate setting this up during initialization makes sense.

-          Call InitMixerControls to initialize the Mixer API support.

HardwareContext::Deinit

This method is called when the driver is unloaded and the system calls WAV_Deinit. In the current design, wave drivers are never unloaded so this method has limited usefulness.

HardwareContext::UpdateOutputGain

HardwareContext::UpdateInputGain

HardwareContext::SetOutputGain

HardwareContext::SetOutputMute

HardwareContext::GetOutputGain

HardwareContext::GetOutputMute

HardwareContext::GetInputMute

HardwareContext::SetInputMute

HardwareContext::GetInputGain

HardwareContext::SetInputGain

These methods are associated with the master input and output gain controls provided by the default mixer API implementation and the device gain waveOutSetVolume API. In the Ensoniq implementation these defer processing to the DeviceContext SetGain methods, which automatically handle volume control in software. There is no need to modify the existing code unless you want to handle some aspects of volume control in hardware. However, keep in mind that individual stream gain controls are still handled in software, and there is no additional overhead in handling device gain as well. Therefore, there is no performance advantage in modifying this code to use hardware gain controls.

HardwareContext::StartOutputDMA

This method starts the DMA controller for audio output. This includes:

1.       Check to see if output dma is already running and ignore the call if it is.

2.       Clear the variables that track how much “live” data is in each DMA buffer.

3.       “Prime” the output DMA buffer with data.

4.       Start the DMA channel if (and only if) data was available to be transferred.

The only line you should need to change is the one that specifically turns on the DMA channel, which in the Ensoniq implementation is:

m_CES1371.StartDMAChannel( ES1371_DAC0 );

HardwareContext::StopOutputDMA

                This method stops the audio output DMA controller. The only line you should need to change is:

m_CES1371.StopDMAChannel( ES1371_DAC0 );

HardwareContext::StartInputDMA

HardwareContext::StopInputDMA

These methods are analogous to the methods described above for the output DMA. However, note that the code to start the input DMA doesn’t need to “prime” the buffer or keep track of how much application data is in the buffer, and is therefore somewhat simpler than the output case.

HardwareContext::GetDriverRegValue

HardwareContext::SetDriverRegValue

These methods relate to reading driver-specific registry keys. You should not need to change them.

HardwareContext::InitInterruptThread

This method initializes the audio driver’s IST thread and sets it to a realtime priority. If your driver has a single IST thread shared by both input and output you will not need to modify this code.

HardwareContext::PowerUp

HardwareContext::PowerDown

These methods are called by the system’s power management subsystem. In the Ensoniq driver they are stubbed out.

HardwareContext::TransferInputBuffer

HardwareContext::TransferOutputBuffer

This method is called from the IST to transfer one data into our out of the DMA buffers. The code determines the starting address and size of the DMA buffer and passes the information to the device context, which performs the actual transfer. You will not need to modify this code unless you change the organization or data structures representing the DMA buffers.

HardwareContext::InterruptThread

This method implements the Interrupt Service Thread which is shared by both input and output DMA. It’s operation is basically:

1.       Wait for an input or output DMA done interrupt.

2.       Determine whether an input or output (or both) DMA interrupt occurred.

3.       If an output DMA interrupt occurred:

§  Transfer/mix application data into the DMA buffer that was just completed.

§  If there is no application data remaining in either DMA buffer, halt output DMA

4.       If an input DMA interrupt occurred:

§  Transfer data out of the DMA buffer that was just completed into application buffers.

§  If we were unable to transfer any data (due to no application buffer being available), halt input DMA.

5.       Go back to step 1.

HardwareContext::SetSpeakerEnable

HardwareContext::RecalcSpeakerEnable

HardwareContext::ForceSpeaker

These methods handle the WODM_FORCESPEAKER message which may be used to request that audio data be routed to an auxiliary speaker on the back of the phone (this speaker is typically larger and more powerful than the earpiece speaker, and is used for ringtones). If you hardware supports this functionality, you will need to add code to the SetSpeakerEnable to switch the speaker on or off.

HardwareContext::PmControlMessage

This method receives messages from the Power Manager IOCTL calls:

IOCTL_POWER_CAPABILITIES

IOCTL_POWER_QUERY

IOCTL_POWER_SET        

IOCTL_POWER_GET

                You probably will not need to modify this code.

HardwareContext::IsSupportedOutputFormat

This method is called during waveOutOpen to allow the OEM to support additional custom audio formats beyond the standard PCM functionality. Normally this method should just return FALSE. The Ensoniq driver supports directly playing WMAPro compressed audio content over its S/PDIF interface and therefore returns TRUE for this specific case.

DMA Buffer Organization and Data Transfer

In the Ensoniq implementation, which is fairly typical, input and output are each allocated a DMA buffer using HalAllocateCommonBuffer. The size of each buffer is only 4k (the same as a memory page), so it’s unlikely that the allocation will fail (especially since it takes place during boot). Other implementations may choose to preallocate a fixed area of memory for the audio DMA buffers.

During audio transfer, each DMA buffer is logically subdivided into equally-sized DMA pages 0 and 1, and the hardware is programmed to:

a.       Transfer nonstop from the DMA buffer to the codec, and automatically reload the DMA address register with the start address of the buffer when it reaches the end.

b.      Generate an interrupt to the audio system whenever the DMA address moves either past the midpoint of the buffer (e.g. from page 0 to page 1), or reaches the end and restarts itself (e.g. from page 1 to page 0).

On each DMA interrupt, the HardwareContext code needs to determine which DMA page the DMA controller has just finished copying data into/out of and call the DeviceContext’s TransferBuffer method to copy application data into/out of that buffer.

Buffer Security/Copying

(Still working on this section) 

Support for S/PDIF

(Still working on this section) 

Differences between Windows CE 5 & Windows CE 6

All current versions of Windows Mobile (including Windows Mobile 6) are based on Windows CE 5 or earlier OS releases. However, the most recent version of wavedev2 is shipped with Windows CE 6, and that’s the version I’m examining for this guide. Therefore, it’s important to touch on differences between how the two OS’es interact with wave drivers.

For the most part, the audio driver architecture between CE5 and CE6 is the same. Audio drivers written for CE5 can generally run with little or no modification on CE6.

Virtual Addressing Differences

When porting the Ensoniq wavedev2 driver from CE5 to CE6, the only change specifically related to moving between operating systems was to surround the call to SetProcPermissions in hwctxt.cpp as follows:

#if (_WINCEOSVER < 600)

    SetProcPermissions((DWORD)-1);

#endif

On Windows CE6, the API’s SetProcPermissions and GetProcPermissions are no longer supported due to changes in the virtual memory architecture. They are still exported for backward-compatibility purposes, but they will have no affect (other than printing out a nasty warning message on the debugger). This change bears a little explanation:

On pre-Windows CE6 systems there is a limit of 32 processes, and all processes run in a shared virtual memory space. The system provides cross-process protection to ensure that processes don’t access each other’s memory, and this protection is enforced on a per-thread basis. Device drivers (such as the wave driver) run inside device.exe, which is one of these 32 processes. The Interrupt Service Thread (IST) in the driver is responsible for accessing audio data in various application buffers residing in multiple processes. The audio driver’s IST overrides this protection by calling SetProcPermissions(0xFFFFFFFF); each bit in the parameter represents one of 32 processes in the system, so 0xFFFFFFFF enables access to all of them.

Windows CE 6 adopts a more traditional memory architecture, with the kernel taking the upper 2GB of virtual space and each user process occupying the same lower 2GB region. Switching between user processes involves swapping a new process into the lower 2GB. While this greatly expands the amount of virtual memory available to each user process, it also means that the IST thread (now running in the kernel) may no longer freely access other arbitrary process’ address space.

To solve this problem, the waveapi middleware (which sits above the audio driver on the stack) now takes care of mapping each application data buffer into a kernel (via CeAllocAsynchronousBuffer). The memory mapping/unmapping take place during waveOutPrepareHeader/waveOutUnprepareHeader, so the cost of memory management doesn’t impact performance.

The video renderer operates in two distinct modes:

• GDI
• DirectDraw

When the graph is first connected, the video renderer always tries to connect using GDI, and for that, it will need a connection with an RGB media type that matches the display format of the primary monitor. Just when it goes into Paused mode, the video renderer will try to allocate surfaces using DirectDraw. This dual mode of operation was envisioned so as to always have a fall back plan in case DirectDraw surfaces were not available in some circumstances.

Choosing an Accelerated Media Type

When the video renderer goes into Paused mode, it is time to allocate the DirectDraw surfaces. The video renderer will do so by enumerating all media types of the upstream filter, and then trying to allocate a surface matching that media type. For instance, let's assume the upstream filter is the WMV DMO. It currently supports the following output media types (the preferred media type is the first one):

• YV12
• NV12
• YUY2
• I420
• IYUV
• UYVY
• YVYU
• RGB565
• RGB555
• RGB32
• RGB24
• RGB8

The video renderer will try to allocate flipping overlay surfaces first, then non-flipping surfaces:

• For each media type of the upstream filter, in the order dictated by the upstream filter
• try to allocate a flipping surface of that media type
• If it succeeds, call QueryAccept on the upstream filter's output pin
• If it succeeds, use it
• If the previous didn't succeed, try to allocate a primary flipping surface (if enabled)
• If it succeeds, call QueryAccept on the upstream filter's output pin
• If it succeeds, use it
• If the previous didn't succeed, for each media type of the upstream filter, in the order dictated by the upstream filter
• try to allocate a surface (not flipping) of that media type
• If it succeeds, call QueryAccept on the upstream filter's output pin
• If it succeeds, use it

In this way, if the upstream filter has optimized for certain YUV formats, it can control the choice of media type. In case the display driver can also provide a surface of that type, the accelerated media type is chosen. The whole process is driven by the upstream filter, with the display driver in a passive role.

Dynamic Format Changes from the Video Renderer

Of course, for an optimal pipe, we would like to always have overlay flipping surfaces available. Nevertheless, that may not be the case in some situations. For instance, depending on the display driver capabilities, the flipping overlay may just be available when the user is watching the video at its original size. When the user is stretching or shrinking, overlays might not be available. This is controlled by the DirectDraw hardware capabilities dwMinOverlayStretch and dwMaxOverlayStretch (see http://msdn2.microsoft.com/en-us/library/aa915204.aspx). So, if the display driver doesn't support overlay stretching, and the video renderer is currently using overlays, it will need to swap to GDI (and thus to RGB format), so that GDI will do the necessary scaling.

Note that every time the upstream filter requests a new buffer from the video renderer, the video renderer will try to return a DirectDraw buffer. If all the conditions to use the DirectDraw buffer are OK (clipping, stretching, video memory, etc.), then it will use it. Just in case one of the conditions fail it will resort to using GDI.

Debugging Video Renderer Connection Problems

We have seen some common connection problems when initially bringing up new decoder filters and/or capture drivers:

• Color space converter is inserted in the graph
• Video renderer doesn't connect
• YUV surfaces are not being used, just GDI

Analyzing the DirectShow logs 

The first step in this case is to turn on the debug zones for the DirectShow DLL, quartz.dll, and observe the connection and video renderer messages. 

• Set it on the client when quartz.dll is already loaded (my preferred method ;-):
• Go to Target/CE debug zones, choose quartz.dll, and check the first four zones
• Set it on the host-side registry if you want to start with the zones turned on:
• Go to [HKEY_CURRENT_USER\Pegasus\Zones]
• Add a DWORD value with the name quartz and set it to 0xf.
• (see more information on debug zones at: http://blogs.msdn.com/ce_base/archive/2006/12/18/debug-messages-and-debug-zones-in-windows-ce.aspx)

Run your test scenario, and save the debug output. Look for the section that says "Filter Graph Dump", and verify which filters got inserted in the graph. Here's an example of a filter graph dump:

Filter graph dump
Filter 1a199a30 'Video Renderer' Iunknown 1a199a20
    Pin 1a199f10 Input (Input) connected to 1a0e1880
Filter 1a0e1200 'WMVideo & MPEG4 Decoder DMO' Iunknown 1a0e11f0
    Pin 1a0e16e0 in0 (Input) connected to 1a0e0600
    Pin 1a0e1880 out0 (PINDIR_OUTPUT) connected to 1a199f10
    Pin 1a0e1a00 ~out1 (PINDIR_OUTPUT) connected to 0
Filter 1a0ecc60 'ASF ICM Handler' Iunknown 1a0ecc50
    Pin 1a0ecd70 In (Input) connected to 1a0aa3a0
    Pin 1a0e0600 Out (PINDIR_OUTPUT) connected to 1a0e16e0
Filter 1a0ec240 'Audio Renderer' Iunknown 1a0ec230
wo: GetPin, 0
    Pin 1a0ec4e0 Audio Input pin (rendered) (Input) connected to 1a0eb880
Filter 1a0eb220 'WMAudio Decoder DMO' Iunknown 1a0eb210
    Pin 1a0eb660 in0 (Input) connected to 1a0ea800
    Pin 1a0eb880 out0 (PINDIR_OUTPUT) connected to 1a0ec4e0
Filter 1a0e9380 'ASF ACM Handler' Iunknown 1a0e9370
    Pin 1a0e9490 In (Input) connected to 1a0aa000
    Pin 1a0ea800 Out (PINDIR_OUTPUT) connected to 1a0eb660
Filter 1a0a2ae0 '\Hard Disk2\clips\wmv\0-1.asf' Iunknown 1a0a2ad0
    Pin 1a0aa000 Stream 1 (PINDIR_OUTPUT) connected to 1a0e9490
    Pin 1a0aa3a0 Stream 2 (PINDIR_OUTPUT) connected to 1a0ecd70
End of filter graph dump

After that, verify which media type the video renderer is using when trying accelerated mode (and if it succeeded). Search for "Allocating video resources":

Allocating video resources
Initialising DCI/DirectDraw
Searching for direct format
Entering ReleaseSurfaces
Entering HideOverlaySurface
Enumerated 32315659
Entering FindSurface
Entering GetMediaType
Not a RGB format
Entering CreateYUVFlipping
Entering CheckCreateOverlay
GWES Hook fails surface creation. IDirectDraw::CreateSurface fails.
No surface
Entering ReleaseSurfaces
Entering HideOverlaySurface
Enumerated 3231564e
Entering FindSurface
Entering GetMediaType
Not a RGB format
Entering CreateYUVFlipping
Entering CheckCreateOverlay
GWES Hook fails surface creation. IDirectDraw::CreateSurface fails.
No surface
Entering ReleaseSurfaces
Entering HideOverlaySurface
Enumerated 32595559
Entering FindSurface
Entering GetMediaType
Not a RGB format
Entering CreateYUVFlipping
Entering CheckCreateOverlay
Entering InitOverlaySurface
Entering InitDrawFormat
Entering InitDrawFormat
Entering GetDefaultColorKey
Returning default colour key
Entering InitDefaultColourKey
Entering SetSurfaceSize
Preparing source and destination rectangles
Entering ClipPrepare
Entering InitialiseClipper
Entering InitialiseColourKey
overlay color key on
Colour key
No palette
Found AMDDS_YUVFLP surface
Proposing output type  M type MEDIATYPE_Video  S type MEDIASUBTYPE_YUY2

Note in the above log that the video renderer tried to create surfaces in the order specified by the WMV DMO. For the display driver in use for the above log, it managed to create a YUY2 surface, the third option for the WMV decoder. The last section in this blog entry has more information about FourCC codes.

Here are some solutions for common connection problems we have faced in the past.

Color Space Converter is inserted in the graph 

The number one problem is that the upstream filter doesn't report any RGB format, just YUV formats. If that's the case, the video renderer can't connect directly to the filter since it requires a matching RGB format. Usually, the color space converter will be inserted in the graph in these cases. We don't want this to happen, as it will imply a memory copy of each frame buffer, so we want to make sure the upstream filter does provide RGB formats.

Sometimes the color converted gets inserted in the graph even though the upstream filter does support the needed RGB format. This can happen because the upstream filter is requiring an alignment different than 1 when the allocator is being decided. Currently, the video renderer will just accept 1-byte alignments.

Another common reason for the color converter to be inserted in the graph is when the BITMAPINFOHEADER supplied by the upstream filter doesn't contain the bitmasks correctly at the end of BITMAPINFOHEADER that is passed when getting the output media types. Please make sure that the bitmasks are inserted correctly. For instance, for RGB565, we should have:

        *pdwBitfield++  = 0xF800;       // Red – 5

 *pdwBitfield++  = 0x07E0;       // Green - 6

 *pdwBitfield    = 0x001F;       // Blue - 5

Graph doesn't connect at all

If the upstream filter just supports a subset of YUV formats, and none of these are recognized by the color space converter, then it won't be possible at all to connect the video renderer. Again, in this case the solution is for the upstream filter to provide RGB formats.

YUV Surfaces are not used, just GDI

Another common occurrence is for the upstream filters to provide allocators. If this is the case, the video renderer will be tied to not using DirectDraw (as it can't pass upstream memory buffers to DirectDraw). If we want the optimal overlay flipping path, the video renderer *needs* to be the allocator, so that it is possible for it to provide DirectDraw surfaces upstream.

Surface Types: Controlling Which Surfaces the Video Renderer Creates

There are ways to control which accelerated surfaces the video renderer is allowed or not to create that are useful when debugging the connection process, specially to reduce the number of options and the number of tries in the display driver. This is controlled via a registry key (see http://msdn2.microsoft.com/en-us/library/aa930626.aspx):

HKEY_LOCAL_MACHINE\Software\Microsoft\DirectX\DirectShow\Video Renderer\SurfaceTypes

The following table shows the AMDDS values for use with the SurfaceTypes named value.

If you just want to enable YUV overlay flipping surfaces for debugging purposes, you should set the SurfaceTypes registry key to AMDDS_YUVFLP. Remember to turn on all surfaces back on after you finished debugging your problem...

About FourCC codes:

Note that in the example log we list the FOURCC codes that are being used in the line "Enumerated 32315659". Here's how to map this hex number into a character sequence that will help identify the code:

 Enumerated 32315659

0x32 = '2', 0x31 = '1', 0x56 = 'V', 0x59 = 'Y' ===> 0x32315659 = YV12

 Enumerated 3231564e

0x32 = '2', 0x31 = '1', 0x56 = 'V', 0x4e = 'N' ===> 0x3231564e = NV12

 Enumerated 32595559

0x32 = '2', 0x59 = 'Y', 0x55 = 'U', 0x59 = 'Y' ===> 0x32595559 = YUY2

 Enumerated 56555949

0x56 = 'V', 0x55 = 'U', 0x59 = 'Y', 0x49 = 'I' ===> 0x56555949 = IYUV

 Enumerated 59565955

0x59 = 'Y', 0x56 = 'V', 0x59 = 'Y', 0x55 = 'U' ===> 0x59565955 = UYVY

 Enumerated 55595659

0x55 = 'U', 0x59 = 'Y', 0x56 = 'V', 0x59 = 'Y' ===> 0x55595659 = YVYU

Also, the file in public\directx\sdk\inc\uuids.h contains several FOURCC media subtypes definitions:

• // 32595559-0000-0010-8000-00AA00389B71 'YUY2' == MEDIASUBTYPE_YUY2
• OUR_GUID_ENTRY(MEDIASUBTYPE_YUY2,
• 0x32595559, 0x0000, 0x0010, 0x80, 0x00, 0x00, 0xaa, 0x00, 0x38, 0x9b, 0x71)
• // 55595659-0000-0010-8000-00AA00389B71 'YVYU' == MEDIASUBTYPE_YVYU
• OUR_GUID_ENTRY(MEDIASUBTYPE_YVYU,
• 0x55595659, 0x0000, 0x0010, 0x80, 0x00, 0x00, 0xaa, 0x00, 0x38, 0x9b, 0x71)
• // 59565955-0000-0010-8000-00AA00389B71 'UYVY' == MEDIASUBTYPE_UYVY
• OUR_GUID_ENTRY(MEDIASUBTYPE_UYVY,
• 0x59565955, 0x0000, 0x0010, 0x80, 0x00, 0x00, 0xaa, 0x00, 0x38, 0x9b, 0x71)
• // 31313259-0000-0010-8000-00AA00389B71 'Y211' == MEDIASUBTYPE_Y211
• OUR_GUID_ENTRY(MEDIASUBTYPE_Y211,
• 0x31313259, 0x0000, 0x0010, 0x80, 0x00, 0x00, 0xaa, 0x00, 0x38, 0x9b, 0x71)
• // 32315659-0000-0010-8000-00AA00389B71 'YV12' == MEDIASUBTYPE_YV12
• OUR_GUID_ENTRY(MEDIASUBTYPE_YV12,
• 0x32315659, 0x0000, 0x0010, 0x80, 0x00, 0x00, 0xaa, 0x00, 0x38, 0x9b, 0x71)
• // 36313259-0000-0010-8000-00AA00389B71 'YV16' == MEDIASUBTYPE_YV16
• OUR_GUID_ENTRY(MEDIASUBTYPE_YV16,
• 0x36315659, 0x0000, 0x0010, 0x80, 0x00, 0x00, 0xaa, 0x00, 0x38, 0x9b, 0x71)
• // 56595549-0000-0010-8000-00AA00389B71 'IUYV' == MEDIASUBTYPE_IUYV
• OUR_GUID_ENTRY(MEDIASUBTYPE_IUYV,
• 0x56595549, 0x0000, 0x0010, 0x80, 0x00, 0x00, 0xaa, 0x00, 0x38, 0x9b, 0x71)
• // 3231564E-0000-0010-8000-00AA00389B71 'NV12' == MEDIASUBTYPE_NV12
• OUR_GUID_ENTRY(MEDIASUBTYPE_NV12,
• 0x3231564E, 0x0000, 0x0010, 0x80, 0x00, 0x00, 0xaa, 0x00, 0x38, 0x9b, 0x71)
• // 30323449-0000-0010-8000-00AA00389B71 'I420' == MEDIASUBTYPE_I420
• OUR_GUID_ENTRY(MEDIASUBTYPE_I420,
• 0x30323449, 0x0000, 0x0010, 0x80, 0x00, 0x00, 0xaa, 0x00, 0x38, 0x9b, 0x71)
• // 56555949-0000-0010-8000-00AA00389B71 'IYUV' == MEDIASUBTYPE_IYUV
• OUR_GUID_ENTRY(MEDIASUBTYPE_IYUV,
• 0x56555949, 0x0000, 0x0010, 0x80, 0x00, 0x00, 0xaa, 0x00, 0x38, 0x9b, 0x71)

Please do leave feedback, and let us know if this has been useful. Thanks,

Lucia

The stream time is based off the reference clock, but relative to the time the graph last started running (so the stream time doesn't move if the graph is paused). If a media sample that enters a renderer has a time stamp t, then it means it should be rendered at stream time t. This is the basic mechanism by which a/v synchronization occurs.

There is usually a crystal in the audio hardware though, and no guarantees that the hardware timer will match the system clock. That's why usually we have the audio renderer being the reference clock for the whole DirectShow graph. If the audio renderer receives a sample late, or if the audio clock is consistently drifting from the system clock, then the audio renderer will issue stream time adjustments.

An audio renderer implementation will usually inherit from the CBaseReferenceClock class, and will call SetTimeDelta() function whenever it needs to do an adjustment to the stream time. Note that it should use a low pass filter before sending adjustments to the master clock so that no unnecessary jittering is introduced.

As the video renderer uses the incoming timestamps to schedule samples for presentation, the scheduler is based off stream time, and the audio renderer has control to change the stream time, the video and audio renderer will be using the same timeline.

About the Video Renderer & Frame Dropping

If the video is running slow, and all video frames are being rendered, then theoretically the video renderer will receive samples with timestamps in the past and schedule them for immediate rendering. If this situation continues to happen, what will happen is that the video is going to be behind audio. This shows the need for frame dropping.

In fact, audio and video synchronization in DirectShow works by a combination of two elements:

• Audio renderer controlling the DirectShow stream time;
• IQualityControl and IDMOQualityControl interfaces guiding frame dropping algorithm

Dropping frames at the video renderer level is of course not very effective. If using overlay flipping surfaces, for instance, dropping a frame doesn't get you much farther trying to catch up (because the flipping itself is very cheap). Even in the case of Blits, it is still going to help very little (rendering time is small compared to decoding time). That's why there is the need to indicate the state and lateness of the renderer to upstream filters/DMOs, which is done through the quality notification messages.

The video renderer originates the notification messages (since it is the filter that needs to run in real time), and sends them upstream. If the upstream filter is a decoder, and it can handle it, it doesn't pass the message upstream. If it can't handle, then it passes it upstream. Note that the video renderer will drop frames anyway if it is very late.

Here's a coarse example of how to use the Quality interface to be able to drop frames in a decoder filter:

HRESULT CDecoderFilterPin::Notify(IBaseFilter *pSender, Quality q)

{

       if (quality sink has been set)           // m_pQSink

       {

              status = Pass Notify on the quality sink   (base sender is the decoder filter now)

}

else

{

if (has frame dropping algorithm)

              {

                     status = Call decoder filter to do frame dropping

              }

              else

              {

                     if (upstreamQualityControl)

                     {

                           status = Pass Notify() on to upstream quality control interface (base sender is the decoder filter now)

                     }

                     else

                     {

                            status = not handled;

                     }

}

}

return status

}

CatchupPTS = q.Late + q.Timestamp

Drop all frames until PTS_frame >= Catchup_PTS

Of course, there are many variations for this. If B-frames are available, start trying to catch up by dropping B-frames. If the decoder has decoding decoupled from output generation (or color conversion etc.), a first step when you're late is to start dropping the output generation, and try to catch up. If neither is working, then you may have to drop P-frames, in which case you'll have to wait until the next I-frame, which is never a good user experience because the spacing between I-frames may be large.

When doing trick modes, the algorithm for dropping frames will usually be more forgiving, since frames are arriving at rates other than 1.0. In fact, at high-rates, all samples that the decoder receives are probably going to be key frames anyway. In this case, you may drop any frame without a significant penalty, making catch up much easier.

One issue in Smartphone devices is determining where to route ringtones and other high-priority notifications. This is driven by two scenarios:

1. A fairly common design feature of Smartphone devices is a separate "high-volume" speaker on the rear of the device to play ringtones or fairly loud notifications. The OEM may want most system sounds to play through the normal handset speaker, but to have incoming call ringtones routed through the rear speaker.

2. When the user has a headset plugged in, all audio is routed to that headset. However, what about the situation where the headset is plugged in but the user isn't wearing it? For this reason, some OEMs may choose to play ringtones through the device speaker even if a headset is attached.

The common thread to each of these scenarios is that the audio device driver needs to make some decision about where to route the audio based on whether the sound being played is a ringtone (or other high-priority notification), or is just normal audio such as music being played through media player.

The wave API doesn't include any built-in provision for the wave driver to determine who is trying to play a sound or where it should be routed. One might use the mixer API to address this, but the mixer API is somewhat complicated and doesn't lend itself to handling this on a per-wave-stream basis. Instead, when we first developed the wavedev2 driver back in 2000, we defined a proprietary wave message, MM_WOM_FORCESPEAKER, which is sent to the driver as a hint that the associated wave stream is a ringtone or notification that should possibly be played over over the device's speaker.

The format of this call is: 

                waveOutMessage(<DeviceID | hWaveOut >, MM_WOM_FORCESPEAKER, bSpeaker, 0);

The first parameter can be either a Device ID (e.g. 0 through N-1 if there are N devices in the system), or the handle of an open wave device. 

The third param tells the driver whether to route sound to the speaker (TRUE) or not (FALSE).

If param1 is a wave handle (and bSpeaker is TRUE), the sound will be routed to the speaker until you either call the API again with the same handle and bSpeaker==FALSE, or you close the wave handle (or your app exits). If an application calls this API with bSpeaker==FALSE, audio should be routed to the earpiece rather than the speaker.

If param1 is a device ID and bSpeaker==TRUE, all audio is routed to the speaker until there's a matching call to the API with that device ID and bSpeaker==FALSE. This behavior will persist even if the application that made the original call exits. We generally never use this form of the API.

Other design details:

-  This API is more of a hint than a command to the driver. It’s up to the OEM to decide how to interpret this message; we don’t place any requirements on how it is interpreted. Some drivers might just ignore it.

- The sample driver does a pretty good job of refcounting which steams have enabled the speaker, so if you turn the speaker on twice for a specific stream it won't really do any harm (we'll ignore the second call).

- If multiple streams are playing and only one of them has turned the speaker on, all of the streams will be routed out the speaker until the stream that turned it on either goes away or turns it back off. As an example, if you're listening to music over WMP and an a ringtone gets played, while the incoming ring is being played you may hear both the incoming ring and the music being played through the speaker. If this is objectionable, OEMs may choose to mute (in software) these other streams for the duration that the ringtone is being played over the speaker.

- Most of this design is geared toward hardware implementations where a single piece of audio hardware is multiplexed between one or more speakers and headset jacks (which was the only thing available when the API was designed).

- The EventSound subsystem (which manages playback of ringtones) keeps track of which tones should be played with ForceSpeaker turned on via a set of reg keys. OEMs can therefore customize which notifications should play over the speaker.

- Don't be surprised if this goes away or gets redesigned in some future release; it's really a Smartphone/PPC proprietary feature, and after 7 years it's getting a little long in the tooth.

Feel free to leave feedback (does anyone read this stuff?).

-Andy

Responses to questions:

1. Where is MM_WOM_FORCESPEAKER defined?

- It should be in audiosys.h, but that might have moved around a bit. The definitions you're looking for are:

    #define MM_WOM_SETSECONDARYGAINCLASS   (WM_USER)

    #define MM_WOM_SETSECONDARYGAINLIMIT   (WM_USER+1)

    #define MM_WOM_FORCESPEAKER            (WM_USER+2)

For the purposes of this blog I'll define multichannel audio to mean any audio stream containing more than two channel stereo. We'll further subdivide multichannel audio into three types, differentiated by how the audio data gets from your CE device (e.g. Set Top Box, Smartphone/PPC, whatever) to your receiver:

1. Analog Matrix Decoders: In this type of decoding, multichannel audio is sent as left/right stereo data to the receiver. In a simple stereo receiver the audio can still be played through left and right speakers and will sound more-or-less correct. However, a receiver supporting the appropriate decoder can use cues placed in the audio to decode to more than two speakers and synthesize additional channels. The most well known decoders are Dolby Pro Logic and Pro Logic II.

2. Compressed audio over S/PDIF: S/PDIF was originally designed to support a maximum of 4 decompressed PCM channels. While one might use this to pass four discrete audio signals to a receiver, today's multichannel content typically has at least six channels (e.g. 5.1). There's no way to squeeze 6 decompressed audio channels across S/PDIF. The solution to this has been to pass the compressed audio over S/PDIF and let the receiver decompress it (as long as the compressed data bandwidth is less than that which would be required for four decompressed channels). Apart from enabling use of a single cable from the device to the receiver, this has the added benefit of offloading the audio decompression processing into the receiver. The downside of this architecture is that it relies on the receiver to be able to correctly decode the audio data. This is complicated because S/PDIF is a one-way transmission mechanism, so there's no way to query the receiver at runtime to determine what it supports. There are potential timing/latency issues with lip synch.

Transferring compressed audio over S/PDIF typically involves massaging the compressed audio into a format that matches the S/PDIF frame format
(e.g. by padding the data with zero's as needed) and adding some header information that lets the receiver figure out that you're sending a
compressed audio stream rather than PCM. Both WMAPro and Dolby AC3 have a spec for this. Almost every receiver in the world supports AC3
decoding. A small (but a growing number) also support WMAPro (Pioneer in particular have spread WMAPro support to even the low end of their product line).

Info on WMAPro is here:
http://download.microsoft.com/download/5/b/5/5b5bec17-ea71-4653-9539-204a672f11cf/wmadrv.doc

Info on AC3-over-S/PDIF is here in Appendix B:
http://www.dolby.com/assets/pdf/tech_library/46_DDEncodingGuidelines.pdf

3. Multiple discrete audio outputs: In this type of connection multiple PCM audio channels are sent to the receiver. Until recently this has meant a separate RCA cable for each channel: a six channel (e.g. 5.1) signal would require six cables between components. HDMI has the potential to overcome this limitation by supporting 6 or more decompressed audio channels (and video) via a single cable.

Outputting to 6 DAC channels presumes that you've already got decompressed multichannel content or you've got a compressed multichannel content that is going to get decompressed before being sent to the wave driver (e.g. AC3 or WMAPro). The latter case is most likely, which means you'll need the appropriate DirectShow decompression filter for CE.

Now, on to what CE supports (and doesn't): 

Device Drivers 

If you want to support either S/PDIF or multiple discrete audio channels, you should probably want to start with the Ensoniq wavedev2 sample driver that
shipped in the Windows CE 5.0 Networked Media Device Feature Pack for CE 5 under public\fp_nmd\common\oak\drivers\wavedev\wavedev2\ensoniq. (Note: everything in this feature pack was rolled forward to CE6 as well, so there's nothing in the feature pack that isn't available in CE6 as well, although it might be in a different place).

This version of the Ensoniq driver has S/PDIF support built into it (the Ensoniq 1371 chip has a sort-of-undocumented S/PDIF mode which we
take advantage of), and supports passing WMAPro-over-S/PDIF compressed date. Support for AC3-over-S/PDIF would be a fairly trivial modification.

One other issue with passing compressed data over S/PDIF is that since the data isn't decompressed to PCM until it gets to your
receiver, there's no way for you to programatically control the volume or mix it with other PCM audio data. The former isn't really a big
issue (the user can always control the volume on their receiver). The latter doesn't have a really great solution.

In the sample Ensoniq driver, whenever we're playing compressed WMAPro out the S/PDIF port we just throw away any PCM data that we're asked to
play so it's never heard (although we maintain the appropriate playback timing, so from the application standpoint everything appears to behave as expected).

To support multichannel discrete outputs in the wave driver one would need to modify the driver to accept a WAVEFORMATEX structure which looks like a normal PCM format but for which the nChannels field is 6 (or more). This is not be a trivial exercise, but should be pretty straightforward. As part of this, for wavedev2 one would have to rewrite the output.cpp file to add a new output stream class that accepts 6 streams, and modify the render functions that handle sample-rate-conversion to support all 6 channels.

Note that the kernel software mixer only supports stereo streams, so it won't do any multichannel mixing for you. This is one reason wavedev2 is probably a good starting place, as it already has code built into it to mix stereo streams which could be extended to more channels.

DirectShow Filters

DirectShow is Windows CE's media processing infrastructure. The architecture is media-type agnostic, meaning that there's nothing in the overall design that makes it support one type of media any better than another. A number of outside customers are working on multichannel audio products using their own DirectShow filters (or filters they licensed from third-parties). The description below only discusses what Microsoft currently ships with CE5 and CE6. 

WMAPro-over-SPDIF filter: The abovementioned Feature Pack also includes a WMAPro-over-SPDIF DirectShow filter to massage WMAPro data into a format which can be sent over S/PDIF. Used in conjunction with a wave driver that supports WMAPro-over-SPDIF content and a receiver which supports decoding WMAPro, this allows a the best decoding quality and performance. To be honest, this isn't currently a terribly common scenario given the limited WMAPro receiver penetration in the market; we did this partly as a proof-of-concept, partly to support our own (Microsoft) technology, and partly because all the pieces were available to us within the company so it wasn't a major development effort. In addition, the architecture and driver changes are applicable to other more common formats (e.g. AC3); although we don't currently ship any explicit support for AC3 streams, OEMs have implemented AC3 support using a similar set of components based on some of this work.

WMAPro decoder: Windows CE includes a WMAPro decoder to decode 5.1, 6.1, and 7.1 compressed content. However, when CE5 was first shipped all our existing customers were still using stereo outputs, so there was no value in passing the discrete channels down to the wave driver. Therefore, while the version that shipped in CE5 decodes all the discrete channels internally, it downmixes them to stereo for output. Therefore, there is currently no way to get the discrete channels out of the WMAPro decoder. The NMD feature pack improved on this situation by introducing matrix-encoding into downmix algorithm: a receiver supporting Pro Logic or Pro Logic II should be able to make use of this information to partially regenerate the discrete channels which were lost during the downmix. We'll look into improving this situation if there's sufficient customer demand.

Dolby AC3: Dolby AC3 is  probably the most common/popular multichannel format. Microsoft doesn't currently ship a Dolby AC3 Directshow decoder, although there are probably lots of third party companies that produce such a thing and there may be open source versions (google "ac3filter").

That's all I've got for now. Please let me know if you found this useful, if there were any errors, or if you have any questions.

Responses to comments (if I misunderstood anyone's question, please let me know):

1. How can I playback audio content simultaneously to both analog audio jacks and S/PDIF (Ianbing)

If I understand correctly, you're trying to play the same audio content over two connections simultaneously (one RCA analog audio jack, and one S/PDIF jack). Assuming that's correct:
- If your audio hardware can simultaneously send a single audio stream over both connections, have the audio driver handle it internally and just expose a single device at the waveapi level.
- If you have two separate pieces of audio hardware (one to handle analog, the other for S/PDIF), you'll need to split the PCM output of the decoder (using a Tee filter- I think there's one under public\directx\sdk\samples\dshow\filters\inftee) and hook both outputs of the tee to the wave renderer.

The latter design causes an additional problem because you'll need a way to tell each renderer which audio device to playback to. To do this, you'll need to hand-construct the graph, get pointers to each of the two audio render filters, and tell each wave renderer which device ID to play to. I don't believe I've ever tried this, but it should be possible by creating an IPropertyBag object (I think you'll have to roll-your-own, but it's not too difficult), setting the "WaveOutId" property to the ID you want to use, and pass that propertybag to the IPersistProperty interface on the wave renderer.

Your code would look something like this (sorry, I haven't compiled/tested this):

    // CPropertyBag is your implementation of the IPropertyBag interface.

    // We might have a public sample of this (search for cpropertybag.cpp), but I'm not sure

    CPropertyBag PropertyBag;

    // Find the waveout renderer in the graph that you want to talk to...

    ...

    // QI for the IID_IPersistPropertyBag interface... something like this...

There are usually three different contexts in which "mixer" is used: the "Software Mixer", the "WaveDev2 Mixer", and the "Mixer API".

The "Software Mixer"

Inside the waveapi module there is a software mixer, sometimes also called the "kernel mixer", which can be used to mix and sample-rate-convert multiple PCM audio output streams. This software mixer was added in CE 4.2 to allow audio drivers which only support one output stream to automagically support multiple concurrent streams at different sampling rates.

Internally, the software mixer spins off a thread for each wave device to which it's attached. This thread takes application audio buffers and mixes them together into a set of mixer buffers which are then passed down to the audio driver. During this process the software mixer performs these tasks:

• Converts all data to a common 16-bit 2-channel (stereo) format.
  
  Note that this means that for an audio driver to work with the software mixer it must support 16-bit stereo data. If your underlying hardware only supports mono data and you want to make use of the software mixer, your audio driver will need to accept the stereo data and mix it down internally to mono. In this scenario if an application passes mono data down through the stack, the software mixer will duplicate it to the two audio channels and the driver will then remerge the data: not elegant, but the typical performance impact is trivial.
  
  The mixer only supports application buffers with PCM formats of 8/16 bit samples and mono/stereo channels. The software mixer can't handle compressed data and can't handle multichannel (e.g. 5.1) PCM data. If you need the software mixer to play any of these formats, they need to be converted to something the mixer understands (e.g. 16-bit stereo PCM) first. This is typically done at the DShow or ACM level, above the software mixer. It only becomes an issue if you want to pass compressed or multichannel audio to the driver. When the software mixer sees a format it doesn't recognize or support, it steps out of the way and passes the waveOutOpen request directly to the device driver. At that point it's totally up to the driver to decide how to handle the request.
  
  
• Sample-rate-converts all data to whatever sample rate the driver requires.
  
  The sample-rate-converter is currently a 5-point FIR. Without going into too much detail, the quality and performance have historically been a fairly good tradeoff for most devices, although there's certainly work we'd like to do in the future to improve the quality and performance.
  
  Note that to do the sample rate conversion the mixer needs to know what sample rate the wave driver requires (or pick a reasonable default). I'll cover how this works in another blog entry, along with other configuration details and some more info about the internal design.
  
  
• Performs per-stream gain control.
  
  When an application calls waveOutSetVolume and passes a wave handle, the software mixer absorbs the call and handles this, so it will never be seen by the driver (the driver will still see waveOutSetVolume calls using a device ID though).
  
  
• Implements support for waveOutSetRate.

The "WaveDev2 Mixer"

The WaveDev2 sample wave driver includes its own output mixer to mix PCM wave streams. This mixer performs basically the same function as the software mixer. Why do we have more-or-less the exact same feature in two different places? You can read my blog Windows CE Audio Driver Samples for more background, but basically because:

• The waveapi software mixer didn't exist at the time WaveDev2 was developed (it first ran on CE 3.0).
• The WaveDev2 mixer handles some proprietary calls that Smartphone/PPC need (see The Wavedev2 Gainclass Implementation). Someday we might fold those into the software mixer, but it hasn't happened yet.
• The WaveDev2 mixer uses a less-cpu-intensive linear interpolation algorithm to perform mixing, and runs in the context of the driver's IST rather than a separate thread. This yields better performance in terms of CPU bandwidth, battery life, and latency (all of which are really important on battery-powered mobile devices).

Having the same feature in two different places and with slightly different/incompatible feature sets is generally not a good thing. Someday we'll rationalize this situation so the mixer only exists in one place.

The WaveDev2 sample driver also supports "input mixing"; this isn't really mixing, but it allows a single hardware input stream to be split and sample-rate-converted to multiple input clients. I'm including it here because inside the driver it shares the same architecture and alot of the same code paths.

The other difference between the Software Mixer and the WaveDev2 Mixer is that the former is in private code and isn't generally modifiable by OEMs, while the latter, being part of the OEM device driver, may be modified as needed. This can be a good or bad thing (depending on your desire to change the code and your expertise at not breaking it ;-)

The "Mixer API"

While the Software Mixer and WaveDev2 MIxer are concerned with mixing PCM audio streams coming down through the Wave API, there's a totally different and unrelated thing called the "Mixer API".

The Mixer API is an API which conceptually sits alongside other top-level APIs like the Wave API, ACM API, TAPI API, etc. The role of the Mixer API is to expose various low-level audio-related controls at the application level (e.g. things like volume, bass/treble, surround sound, etc.). When someone talks about the Mixer API, they're talking about the set of APIs including mixerOpen, mixerClose, etc. An MSDN reference page is here http://msdn2.microsoft.com/en-us/library/ms705739.aspx, and there are some interesting discussions of the Mixer API here Mixer API and here Larry Osterman's WebLog : Mapping audio topologies to mixer topologies,

I believe the mixer API was first introduced as part of the Windows Sound System (WSS) DDK. The Windows Sound System was a hardware reference design that Microsoft developed in the early 1990's to evangelize audio hardware support on the PC platform. One of the features of WSS was inclusion of a Crystal Semiconductor 4231 codec, which eventually evolved into the AC'97 codec spec. This codec had a number of hardware mixing and volume control features to support multiple inputs and outputs (cs4231a multimedia audio codec), but there was no API defined to allow applications to access them. Thus was born the mixer API.

The Mixer API was designed to allow a mixer application with no knowledge of the underlying audio architecture to create interactive UI for the end user. As a consequence, the Mixer API allows the application to query information suggesting what type of UI element to use to represent a control (e.g. a pushbutton, slider, multiple-select, etc.), and even query the labels that the application should display. On the desktop windows, when you run SndVol32 to bring up the mixer control panel (which uses the desktop's implementation of Mixer API), remember that SndVol32 has absolutely no a priori knowledge of what your soundcard supports. All those labels and controls are derived by calling down to the driver level.

All MixerAPI calls into the audio device driver are done via an IOCTL_MIX_MESSAGE IoControl. An audio driver can add support for the mixer API by adding support for this call and the myriad messages that route through it. Although Windows CE audio sample drivers typically include sample code to support for the mixer API, in general it's an optional part of the wave driver and there are very few applications that make use of it (although there are some, notably VoIP).