Extended camera controls
Extended controls use the KSPROPERTY mechanism to expose camera controls to the application.
The following list of standardized extended controls (defined by Media Foundation) enable additional Windows camera features:
Some of the controls are exposed to applications as asynchronous controls and others are exposed as synchronous controls.
Synchronous controls
For these controls, the capture pipeline issues a KSPROPERTY camera control structure and expects the driver to synchronously return the request.
Asynchronous controls
For these controls, the capture pipeline issues a KSPROPERTY, enables a KSEVENT associated with that property, and waits on the event to get signaled. The driver must complete the KSPROPERTY synchronously and use that as a trigger to start the asynchronous operation. Upon the completion of the asynchronous operation, the driver must signal the associated KSEVENT on which the capture pipeline is waiting on. The capture pipeline notifies the application about the completion of the operation when it receives the signal.
If an asynchronous control can be cancelled, it must specify the flag KSCAMERA_EXTENDEDPROP_FLAG_CANCELOPERATION in the control. If the control cannot be cancelled, the control's operation must not exceed 5 ms.
These extended controls are part of the following KS property set defined in ksmedia.h:
#define STATIC_KSPROPERTYSETID_ExtendedCameraControl \
0x1CB79112, 0xC0D2, 0x4213, 0x9C, 0xA6, 0xCD, 0x4F, 0xDB, 0x92, 0x79, 0x72
DEFINE_GUIDSTRUCT("1CB79112-C0D2-4213-9CA6-CD4FDB927972", KSPROPERTYSETID_ExtendedCameraControl);
#define KSPROPERTYSETID_ExtendedCameraControl DEFINE_GUIDNAMED(KSPROPERTYSETID_ExtendedCameraControl);
Metadata
To retrieve metadata, the user mode component (DevProxy) must query the driver for the metadata buffer requirement. Once the user mode component has this information, it allocates the appropriate metadata buffer for the driver to fill and return back to the user mode component.
The KSPROPERTY_CAMERACONTROL_EXTENDED_METADATA property ID that is defined in the KSPROPERTY_CAMERACONTROL_EXTENDED_PROPERTY enumeration is used by the client to query for the metadata buffer requirements, such as required metadata size, memory alignment requirements, and desired memory allocation type, for metadata buffer allocation.
Once user mode component has obtained the metadata buffer requirements from the driver, it allocates the appropriately sized metadata buffer with the desired memory alignment from the desired memory pool. This metadata buffer, along with the actual frame buffer, will be sent to the driver to fulfill and then returned back to the user mode component when filled. For multishot scenarios, a corresponding metadata buffer is allocated and delivered to the camera driver for each frame buffer allocated.
The KSSTREAM_METADATA_INFO structure, along with the following flag, is used to send the metadata buffer to the driver.
#define KSSTREAM_HEADER_OPTIONSF_METADATA 0x00001000
Once the buffer (metadata + frame) is queued to the driver, DevProxy sends a standard KSSTREAM_HEADER structure, followed by a KS_FRAME_INFO structure, and followed by a KSSTREAM_METADATA_INFO structure. DevProxy will further mask KSSTREAM_HEADER.OptionFlags with KSSTREAM_HEADER_OPTIONSF_METADATA before it passes the buffer down to the driver.
If the driver does not support metadata, or if KSPROPERTY_CAMERACONTROL_EXTENDED_METADATA is not implemented, the KSPROPERTY_CAMERACONTROL_EXTENDED_METADATA property control will fail. In this case, DevProxy will not allocate a metadata buffer and the payload that is passed down to the driver from DevProxy will not contain the KSSTREAM_METADATA_INFO structure.
If the driver supports metadata and the client does not want any metadata, DevProxy will neither allocate metadata buffer nor pass down KSSTREAM_METADATA_INFO when sending the buffer to the driver. This behavior reduces the unnecessary metadata memory allocation if an app does not want metadata on a given pin.
The following structures describe the layout of the metadata items to be filled by the camera driver in the metadata buffer.
The list below contains the layout of a metadata item. This must be 8-byte aligned.
Metadata
The photo confirmation metadata is identified by MetadataId_PhotoConfirmation. When present, it indicates that the preview frame associated is a photo confirmation frame. Photo confirmation metadata is parsed by the DevProxy.
The custom metadata is identified by a MetadataId that starts from MetadataId_Custom_Start. The custom metadata item can contain a blob of metadata that can be a focus state and/or faces detected for the preview pin, EXIF and/or the OEM metadata for an image pin. The exact format of the custom blob is determined by the OEM who implements the driver and MFT0. The MFT0 is responsible for parsing the custom blob and attaching each metadata item as an attribute grouped under the MFSampleExtension_CaptureMetadata attribute bag in a format that is readable by the MF capture pipeline and/or WinRT.
The following IMFAttributes are defined in mfapi.h. These are required by the MF capture pipeline and/or WinRT. Note that MFT0 does not set any IMFAttributes for photo confirmation since the photo confirmation frame will not flow beyond DevProxy.
| Attribute (GUID) | Data type |
|---|---|
| MF_CAPTURE_METADATA_FOCUSSTATE | UINT32 |
| MF_CAPTURE_METADATA_FACEROIS | Blob |
| MF_CAPTURE_METADATA_FRAME_RAWSTREAM | IUnknown |
The MF_CAPTURE_METADATA_FRAME_RAWSTREAM IMFAttributes are created and attached to MFSampleExtension_CaptureMetadata by the DevProxy, which contains a pointer to the IMFMediaBuffer interface associated with the raw metadata buffer (KSSTREAM_METADATA_INFO.Data).
When the MFT0 receives an IMFSample, it gets the raw metadata buffer from the MF_CAPTURE_METADATA_FRAME_RAWSTREAM and parses any additional custom metadata items such as focus state and converts them into corresponding IMFAttributes defined above and attaches them to the MFSampleExtension_CaptureMetadata attribute bag. The following IMFAttributes must be carried over by the MF pipeline and any third-party supplied MFTs:
| Name | Type |
|---|---|
| MFSampleExtension_CaptureMetadata | IUnknown (IMFAttributes) |
| MFSampleExtension_EOS | UINT32 (Boolean) |
| MFSampleExtension_PhotoThumbnail | IUnknown (IMFMediaBuffer) |
| MFSampleExtension_PhotoThumbnailMediaType | IUnknown (IMFMediaType) |
To access the raw metadata buffer, the MFT0 does the following:
Calls GetUnknown on MFSampleExtension_CaptureMetadata from the IMFSample interface to get the IMFAttributes interface for the attribute bag.
Calls GetUnknown on MF_CAPTURE_METADATA_FRAME_RAWSTREAM from the IMFAttributes interface obtained from the previous step to get the IMFMediaBuffer interface.
Calls Lock to get the raw metadata buffer associated with IMFMediaBuffer.
To add the required IMFAttributes to the MFSampleExtension_CaptureMetadata attribute bag, the MFT0 does the following:
Calls GetUnknown on MFSampleExtension_CaptureMetadata from the IMFSample interface to get the IMFAttributes interface for the attribute bag.
Calls SetUINT32, SetBlob, or SetUnknown on MF_CAPTURE_METADATA_XXX from the IMFAttributes interface obtained from the previous step based on the GUID and data type specified in the table above.
Mandatory metadata attributes
The full list of metadata attributes available can be found at Capture Stats Metadata Attributes
Focus priority
The KSPROPERTY_CAMERACONTROL_EXTENDED_FOCUSPRIORITY property ID is the only control that is associated with the focus priority DDI.
Focus state
The KSPROPERTY_CAMERACONTROL_EXTENDED_FOCUSSTATE property ID is the only control that is associated with the focus state DDI.
Extended region of interest ROI
The following property IDs are the controls that are associated with the ROI DDI:
Photo confirmation
The KSPROPERTY_CAMERACONTROL_EXTENDED_PHOTOCONFIRMATION property ID is the only control that is associated with the photo confirmation DDI.
Photo sequence submode
The KSPROPERTY_CAMERACONTROL_EXTENDED_PHOTOMODE property ID is the only control associated with the photo sequence DDI.
EXIF and HW JPEG encoder
The pipeline is not required to process or warp any EXIF data for the HW JPEG encoder; therefore, the EXIF data format is provided by the driver, MFT0, and OEM HW JPEG encoder. OEMs partners can define any custom attribute GUID and variant type for the EXIF attribute and attach it to the MFSampleExtension_CaptureMetaData attribute bag for it to be consumed by the OEM components. If a HW JPEG encoder is available, the pipeline photo sink component will load the HW JPEG encoder and set the EXIF data held in the MFSampleExtension_CaptureMetaData attribute bag onto the HW JPEG encoder as an EXIF encoder option using the IPropertyBag2::Write method.
The encoder option property bag contains an array of PROPBAG2 structures that specify the available encoding option properties. The EXIF encoder option set onto HW JPEG encoder is identified by the following property in the encoder option property bag:
| Property name | VARTYPE | Value | Applicable codecs |
|---|---|---|---|
| SampleMetaData | VT_UNKNOWN | Pointer to an IMFAttributes interface for MFSampleExtension_CaptureMetaData attribute bag that contains an OEM sub attribute containing the EXIF data. | JPEG |
The MFSampleExtension_CaptureMetaData attribute bag can only contain any OEM defined EXIF sub attribute that the MFT0 and HW JPEG encoder can read to hold the EXIF data. To pass EXIF data from the driver to the HW JPEG encoder, the driver and MFT0 must do the following:
The driver provides custom EXIF metadata in the metadata buffer supplied by the pipeline. This is attached to MFSampleExtension_CaptureMetadata as an MF_CAPTURE_METADATA_FRAME_RAWSTREAM IMFAttribute by DevProxy when the sample is returned back to DevProxy.
When the MFT0 receives an IMFSample, it gets the raw metadata buffer from MF_CAPTURE_METADATA_FRAME_RAWSTREAM and parses the custom EXIF metadata item and converts it to an OEM defined IMFAttribute and attaches it to the MFSampleExtension_CaptureMetadata attribute bag.
To pass EXIF data from the MFT0 to the HW JPEG encoder, the pipeline photo sink does the following:
Calls GetUnknown on MFSampleExtension_CaptureMetadata from IMFSample to get the IMFAttributes interface for the attribute bag when IMFSample is received.
Calls IPropertyBag2::Write to set the encoder option property, identified by SampleMetadata, to the HW JPEG encoder. The encoder option property contains the IMFAttributes interface obtained from the previous step. This interface contains all custom sub attributes including the OEM EXIF sub attribute, and the standardized sub attributes in the Metadata section discussed earlier in this topic.
To retrieve the EXIF data for further processing, the HW JPEG encoder does the following:
Calls IPropertyBag2::Read to retrieve the property value for the property identified by the SampleMetadata property name and VT_UNKNOWN type. When returned, the VARIANT.punkVal receives the IMFAttributes interface for MFSampleExtension_CaptureMetadata.
Calls GetBlob or GetUnknown on the OEM EXIF sub attribute from the interface obtained from the previous step to get the EXIF data blob based on the GUID and data type of the OEM EXIF sub attribute.
Thumbnail
MFT0 is not required to produce any thumbnail for the camera driver. The camera app is expected to generate its own thumbnail. The thumbnail could be produced from the photo confirmation image, HW JPEG encoder, or resizing a full size image. This is up to the app developers. To maintain API and app compatibility with Windows 8.1 apps, the camera driver must not implement the KSPROPERTY_CAMERACONTROL_EXTENDED_PHOTOTHUMBNAIL control.
Integer ISO
The KSPROPERTY_CAMERACONTROL_EXTENDED_ISO_ADVANCED property ID is the only control associated with the integer ISO DDI.
Advanced focus
The KSPROPERTY_CAMERACONTROL_EXTENDED_FOCUSMODE property ID is the only control associated with the integer ISO DDI.
Flash
The KSPROPERTY_CAMERACONTROL_EXTENDED_FLASHMODE property ID is the only control that is associated with the flash DDI.
Zoom
The KSPROPERTY_CAMERACONTROL_EXTENDED_ZOOM property ID is the only control that is associated with the zoom DDI.
Scene mode
The KSPROPERTY_CAMERACONTROL_EXTENDED_SCENEMODE property ID is the only control associated with the scene mode DDI.