Timeline Markers and Events
Applications can handle timeline events that occur while a media stream plays or use markers to find specified locations in the stream. The StreamInfo class in the Microsoft.Web.Media.SmoothStreaming client library corresponds to StreamIndex element in the XML manifest.
This topic shows how to extract time and text data from StreamIndex elements and initialize TimelineMarker objects. The example extends the examples implemented in the Silverlight Applications, Events, and Position in Stream topics in order to create a table of contents that specifies chapter headings.
Defining Timeline Events in the Manifest
The client manifest file, which has the extension .ismc, uses XML to specify metadata for Smooth Streaming media. There are three types of StreamIndex elements in the manifest: Video, Audio, and Script. Each type is also identified as a member of the MediaStreamType enumeration.
Timeline events are specified in a StreamIndex element as shown in the following example. The manifest can contain as many StreamIndex elements as are required in order to implement markers, captions, animations, or other features that are activated at time-specified locations in the media stream. (For more information about captions, see Manifest Merge.) Using manifest syntax, you can identify each StreamIndex element by type and attributes.
The following XML creates a table of contents that lists chapter headings located at time intervals in the stream.
<StreamIndex Type="text"
Name="MARKERS"
Subtype="CHAPTER"
TimeScale="10000000"
ParentStreamIndex="video"
ManifestOutput="TRUE"
QualityLevels="1" Chunks="9"
Url="QualityLevels({bitrate},{CustomAttributes})/Fragments(MARKERS={start time})">
<QualityLevel Index="0" Bitrate="1000" CodecPrivateData="" FourCC=""/>
<c n="0" t="596416667">
<f>Q2hhcHRlciAx</f>
</c>
<c n="1" t="1192833334">
<f>Q2hhcHRlciAy</f>
</c>
<c n="2" t="1789250001">
<f>Q2hhcHRlciAz</f>
</c>
<c n="3" t="2385666668">
<f>Q2hhcHRlciA0</f>
</c>
<c n="4" t="2982083335">
<f>Q2hhcHRlciA1</f>
</c>
<c n="5" t="3578500002">
<f>Q2hhcHRlciA2</f>
</c>
<c n="6" t="4174916669">
<f>Q2hhcHRlciA3</f>
</c>
<c n="7" t="4771333335">
<f>Q2hhcHRlciA4</f>
</c>
<c n="8" t="5367750040">
<f>Q2hhcHRlciA5</f>
</c>
</StreamIndex>
The ParentStreamIndex="video" attribute indicates that the timing of this StreamIndex item is keyed to the video stream. StreamIndex elements are sparse streams. The data in sparse streams is not contiguous but occurs at distinct locations in the stream and must be keyed to the video stream.
The c elements are chunks of data that designate each packet using number (n) and time (t) attributes, such as the following:
<c n="5" t="3578500002">
The data itself is contained in the f element. In the previous example, the f elements are chapter headings that are specified as base64 encoded strings, such as the following:
<f>Q2hhcHRlciAx</f>
The TimeScale attribute (here set to 10000000) indicates the tick units by which time intervals and duration are measured: hundred nanosecond (hns) units. The manifest can contain data for each fragment <f> of the stream only when the attribute ManifestOutput attribute is set to true.
Extracting Timeline Events and Assigning Markers
When a SmoothStreamingMediaElement object loads Smooth Streaming media files, the Microsoft.Web.Media.SmoothStreaming.SmoothStreamingMediaElement.AvailableStreams property of the SmoothStreamingMediaElement contains metadata for all the streams defined by the manifest. StreamIndex elements in manifest XML correspond to StreamInfo objects in application code.
The following example shows how to use the MediaOpened event and the asynchronous BeginGetChunk and EndGetChunk methods to add the markers and initialize the Markers collection.
The first step is to assign a delegate to handle the MediaOpened event, as shown in the following example:
SmoothPlayer.MediaOpened +=
new RoutedEventHandler(SmoothPlayer_MediaOpened);
The next step is to access the StreamInfo objects from the AvailableStreams property of the first SegmentInfo instance in the ManifestInfo object. The following code shows how to loop through the available streams and identify the markers stream by type (Type) and attributes.
The StreamIndex element in the manifest that contains markers is identified in code as a System.Windows.Media.MediaStreamType.Script stream. This stream is not included by default in the SelectedTracks collection. The video and audio streams are required and are included by default. In order to access the markers script stream from application code, you have to identify the stream and include it in the SelectedTracks collection.
void SmoothPlayer_MediaOpened(object sender, RoutedEventArgs e)
{
foreach (SegmentInfo segmentInfo in SmoothPlayer.ManifestInfo.Segments)
{
List<StreamInfo> selectStreams = segmentInfo.SelectedStreams.ToList<StreamInfo>();
foreach (StreamInfo streamInfo in segmentInfo.AvailableStreams)
{
if (streamInfo.Type == System.Windows.Media.MediaStreamType.Script)
{
if (streamInfo.Attributes["Name"] == "ClosedCaptions" ||
streamInfo.Attributes["Name"] == "MARKERS")
{
selectStreams.Add(streamInfo);
segmentInfo.SelectStreamsAsync(selectStreams);
foreach (TrackInfo trackInfo in streamInfo.SelectedTracks)
{
foreach (ChunkInfo chunk in streamInfo.ChunkList.ToList<ChunkInfo>())
{
IAsyncResult ar =
trackInfo.BeginGetChunk(
chunk.TimeStamp, new AsyncCallback(AddMarkers), streamInfo.UniqueId );
}
}
}
}
}
}
}
The previous code example calls the BeginGetChunk method and specifies the AddMarkers callback method to handle the results. The AddMarkers method is shown below. It calls the EndGetChunk method and initializes the markers.
private void AddMarkers(IAsyncResult argAR)
{
if (!Deployment.Current.Dispatcher.CheckAccess())
{
Deployment.Current.Dispatcher.BeginInvoke(() => AddMarkers(argAR));
}
foreach (SegmentInfo segmentInfo in SmoothPlayer.ManifestInfo.Segments)
{
foreach (StreamInfo streamInfo in segmentInfo.SelectedStreams)
{
if (streamInfo.UniqueId == ((string)argAR.AsyncState))
{
List<ChunkInfo> markerChunks = streamInfo.ChunkList.ToList<ChunkInfo>();
foreach (TrackInfo trackInfo in streamInfo.SelectedTracks)
{
ChunkResult chunkResult = trackInfo.EndGetChunk(argAR);
if (chunkResult.Result == ChunkResult.ChunkResultState.Succeeded)
{
System.Text.Encoding enc = System.Text.Encoding.UTF8;
int length = (int)chunkResult.ChunkData.Length;
byte[] rawData = new byte[length];
chunkResult.ChunkData.Read(rawData, 0, length);
String text = enc.GetString(rawData, 0, rawData.Length);
TimelineMarker newMarker = new TimelineMarker();
newMarker.Text = text;
newMarker.Time = chunkResult.Timestamp;
SmoothPlayer.Markers.Add(newMarker);
}
}
}
}
}
}
Note
Applications should parse the manifest for markers during the MediaOpened event, but not earlier. Specifically, applications should not parse the manifest during the ManifestReady event, because all markers set before the MediaOpened event is raised are erased in Silverlight during the MediaOpened event.
Creating a Timeline Event Handler
After the TimeLineMarker objects are initialized in the Markers collection, application code can catch and respond to timeline events identified by markers.
To do this, assign a delegate to handle the MarkerReached event as shown in the following example:
SmoothPlayer.MarkerReached +=
new TimelineMarkerRoutedEventHandler(SmoothPlayer_MarkerReached);
Implement the delegate that will run when the event occurs. The following code shows how to write both the timeline position in ticks and the text of the marker to a Silverlight TextBlock object.
void SmoothPlayer_MarkerReached(object sender, TimelineMarkerRoutedEventArgs e)
{
OutputText.Text = " Marker Text: " + e.Marker.Text ;
}
To see another way to handle the MarkerReached event, see Scheduling Media Clips.
Specifying Navigation to Markers
The following example shows how to add a Chapter button to activate a chapter navigation window. (This example assumes that you are extending the application implemented in the topics Silverlight Applications and Events.)
<Button x:Name="ChaptersSeekButton" Width="50" Content="Chapter"
Click="ChaptersSeekButton_Click"/>
Set the enabled property of the ChapterSeekButton object to false in MainPage.xaml. Add the following code to the PlayButton_Click method to enable the ChaptersSeekButton instance only when there are markers in the Markers collection.
if (SmoothPlayer.Markers.Count != 0)
ChaptersSeekButton.IsEnabled = true;
else
ChaptersSeekButton.IsEnabled = false;
When a user clicks Chapter, the code uses the markers represented by the Markers property of the SmoothStreamingMediaElement object to navigate through the stream.
Application code can seek to positions in a stream from TimeLineMarker objects. When a user clicks Chapter, the following code finds the next marker and advances the media stream from the current position to the marker position. (Later in this document, you will modify this code in order to implement a navigation window that contains chapter headings.)
private void ChaptersSeekButton_Click(object sender, RoutedEventArgs e)
{
if (!SmoothPlayer.Markers.Equals(null))
{
foreach (TimelineMarker marker in SmoothPlayer.Markers)
{
if (SmoothPlayer.Position < marker.Time)
{
SmoothPlayer.Position = marker.Time;
break;
}
}
}
}
Creating Table of Contents Navigation Using Markers
A Smooth Streaming application can use the Markers property to implement a table of contents that supports navigation through the stream. The Markers collection provides access to various positions without scrubbing through the entire stream.
To implement navigation in this scenario, follow the instructions in Defining Timeline Events in the Manifest to define markers in the Smooth Stream client manifest .ismc file. Create a marker element in the manifest for each chapter heading that is needed in the media stream. Any media file editor that displays a timeline will provide the position in the media stream based on the interval from the beginning of the stream. Convert the time values to ticks (hundred nanosecond units) and assign them to the t attributes of the c elements as shown in the next example.
<c n="5" t="853120000">
<f>UHVsc2UgdG8gdGhlIGxpbWl0Lg==</f>
</c>
To set the value of the f element, you must convert the chapter texts to base64 encoded strings. Use the following conversion method, which is available in the Convert class.
System.Convert.ToBase64String(System.Text.Encoding.UTF8.GetBytes("Chapter 1"));
Save the changes to the manifest file in the IIS folder that contains the multiple-bitrate media files.
In the Silverlight application, parse the StreamInfo and TrackInfo objects, as shown earlier in this topic, and add the markers to the Markers collection.
Creating a Child Window Table of Contents
There are many ways to display these timeline markers to the user. The following example shows how to create a Silverlight child window that displays a list box of chapter headings as navigable options.
Add a Silverlight child window to the project by using the template provided by the Visual Studio Add New Item option. The following example names the child window ChapterSeekWindow and adds the XAML for it.
<controls:ChildWindow x:Class="Silverlight_SmoothStreaming.ChapterSeekWindow"
xmlns="https://schemas.microsoft.com/winfx/2006/xaml/presentation"
xmlns:x="https://schemas.microsoft.com/winfx/2006/xaml"
xmlns:controls="clr-namespace:System.Windows.Controls;assembly=System.Windows.Controls"
Width="400" Height="300"
Title="Chapter Navigation Window">
<Grid x:Name="LayoutRoot" Margin="2">
<Grid.RowDefinitions>
<RowDefinition Height="0.80*"></RowDefinition>
<RowDefinition Height="0.20*"></RowDefinition>
</Grid.RowDefinitions>
<ListBox x:Name="ChaptersList" Width="355" Height="25" Grid.Row="0"
SelectionChanged="ChaptersList_SelectionChanged" SelectionMode="Single"/>
<Button x:Name="CancelButton" Content="Cancel"
Click="CancelButton_Click" Width="75" Height="23"
HorizontalAlignment="Right" Margin="0,12,0,0" Grid.Row="1" />
<Button x:Name="OKButton" Content="OK" Click="OKButton_Click"
Width="75" Height="23" HorizontalAlignment="Right"
Margin="0,12,79,0" Grid.Row="1" />
</Grid>
</controls:ChildWindow>
To pass the Markers collection to the child window, add a ChapterSeekWindow constructor to the window's partial class definition that includes a parameter for the TimelineMarkerCollection object, as shown in the following example. This code uses the collection argument to populate the ChaptersList list box in the child window. You also must have the members (shown before the constructor in the following code) in order to keep the markers collection in scope and to return results from the child window to the main page.
TimelineMarkerCollection markersCollection;
public TimelineMarker chapterMarker { get; set; }
public ChapterSeekWindow(TimelineMarkerCollection markers)
{
InitializeComponent();
markersCollection = markers;
foreach (TimelineMarker chap in markers)
{
if(chap.Text.Contains("Chapter") && !chap.Text.Contains("Caption"))
{
ChaptersList.Items.Add(chap.Text);
ChaptersList.Height += 20;
}
}
}
To handle the user's selection from the list box, add the following method to the child window partial class. The single line of code in this handler assigns the user's selection to the chapterMarker variable declared in the previous partial-class code segment.
private void ChaptersList_SelectionChanged(object sender,
SelectionChangedEventArgs e)
{
chapterMarker = markersCollection[((ListBox)sender).SelectedIndex];
}
In the MainPage.xaml.cs file, change the ChaptersSeekButton Click event to instantiate the child window. You must also have a variable that holds a reference to the window class that is instantiated by the Click method.
// Child window for chapter selection.
ChapterSeekWindow chaptersWindow;
The if block in the following method assures that the method runs only when the Markers collection contains markers. The code passes the markers collection to the child window and assigns a delegate method to handle the user's selection that is returned from the child window. There can be several kinds of markers in the collection, so the following code checks the marker text for Chapter and eliminates those with an Action value that might indicate the marker is intended for captions or some other feature activated by markers.
private void ChaptersSeekButton_Click(object sender, RoutedEventArgs e)
{
if (!SmoothPlayer.Markers.Equals(null))
{
TimelineMarkerCollection chapterMarkers = new TimelineMarkerCollection();
foreach (TimelineMarker marker in SmoothPlayer.Markers)
{
if (marker.Text.Contains("Chapter") && !marker.Text.Contains("Action"))
{
TimelineMarker copyMarker = new TimelineMarker();
copyMarker.Text = marker.Text;
copyMarker.Time = marker.Time;
copyMarker.Type = marker.Type;
chapterMarkers.Add(copyMarker);
}
}
chaptersWindow = new ChapterSeekWindow(chapterMarkers);
chaptersWindow.Closed +=
new EventHandler(chaptersWindow_Closed);
chaptersWindow.Show();
}
}
The next code segment is the delegate method that gets the user's selection, if any, from the list box of chapter options. If there is no selection by the user, the dialog box result is false, so there is no reassignment of the Position property and no navigation, and the method returns without action. If the user has made a chapter selection, the position indicated by the marker is assigned to the Position property of the SmoothStreamingMediaElement object, which then causes the player to seek to the new position in the stream. The last few lines of code calculate the new position for the slider and adjust the thumb position as a proportion of its range.
double mediaProportionSeconds;
void chaptersWindow_Closed(object sender, EventArgs e)
{
ChapterSeekWindow window = (ChapterSeekWindow)sender;
if(window.DialogResult.Equals(false))
{
chaptersWindow.Closed -= new EventHandler(chaptersWindow_Closed);
return;
}
if (window.ChaptersList.SelectedItem != null)
{
chapterMark = window.chapterMarker;
SmoothPlayer.Position = chapterMark.Time;
// Calculate proportion of media length at current position.
mediaProportionSeconds =
SmoothPlayer.Position.TotalSeconds/SmoothPlayer.EndPosition.TotalSeconds;
// Set slider value (position).
SliderBar.Value = mediaProportionSeconds * SliderBar.Maximum;
}
chaptersWindow.Closed -= new EventHandler(chaptersWindow_Closed);
}