| Comments

A long while back I had written a quick sample when Silverlight introduced drag-and-drop into the framework.  Then I decided to show dragging MP3 files into a Silverlight app and reading the metadata and album art.  In order to accomplish this I had to read into the ID3 data from a Silverlight library.  I found a few libraries but settled on TagLib# to do the job.  I had to modify it a bit to get it working in Silverlight as the .NET profile wasn’t the same.  Recently a surge of people have been emailing me for the code.  I spent time searching and apparently I didn’t think the TagLib# modifications were that important because I never saved them anywhere!  A conversation started on Twitter and I decided to devote some “20% time” to making these modifications and take it a step further and make it into a Portable Class Library (PCL).  Here’s my journey…

Deciding to Fork

My first task was finding the source of truth for TagLib#.  The main link on the Novell developer site was broken and stale.  I found it on GitHub and started looking around.  It really hadn’t been updated in a long while (after all, it really didn’t need to be fore core .NET framework) and the project is very stale.  There is no open issues list on GitHub as you have to use Bugzilla, but that didn’t even look like it was getting much attention.  I emailed the maintainer listed in the authors file in the repository and he indicated he’s not really the maintainer.  This felt like a project kind of fizzling down (if not fizzled already).

In looking at the tests, the project structure, and taking into account that I may want to do some things different, I made the decision to fork rather than clone.  I’m not totally married to the decision but I don’t think anyone is keeping the lights on to take a pull request either though.  I forked the code and started with new projects and using Visual Studio 2013 as my tool of choice.

Using Shared Projects

At first I thought that I would be doing perhaps a few different flavors, portable and full .NET framework.  Because I thought I would I decided to use the new Shared Project system in Visual Studio 2013 and the new Shared Project Reference Manager VS extension that allows me to add references from any project to shared code easily.  This gives me the flexibility for the future and sets my project system up in advance.  You’ll see in the end I haven’t actually needed to take that step and perhaps won’t even need the Shared Project anymore, but for now I’m keeping it as it does me no harm.

First Compile

Once I moved the code over and set my target profile for PCL, I hit build.  Whoa.  About 140 compile errors.  Immediately I thought that I didn’t want to spend the time.  I took a look at the issues and quickly realized that the base code had, in fact, changed a bit from when I messed with it in Silverlight.  I started making a list of the things that weren’t compiling as I was targeting .NET 4.5, Silverlight 5, Windows 8+, Windows Phone 8+, and iOS/Android (Xamarin).  The biggest errors came from the fact that the library was still using XmlDocument and hadn’t moved to XLinq.  Beyond that there were things like Serializable, ICloneable, ComVisible and File IO that weren’t going to work.  I got really frustrated quickly and about gave up.

Working at Microsoft I am fortunate to have access to try more things and indeed I reached out to some folks for help.  I was able to get some things working continued with XmlDocument, but it didn’t feel right and starting to think about releasing this updated library I just realized this wasn’t going to work.  I remained frustrated.

Helpful Friends

Sometimes when you are frustrated you just want to vent to the world.  We call that Twitter these days.  I was pulling some hair out and posted a comment, which was quickly replied by a member of the .NET team with a bit of a touché comment. 

I chuckled but I also knew that David and others were going to be the key to helping me find the fastest path here.  I started emails with the PCL team, including David and Daniel, who are incredibly knowledgeable and responsive.  I finally got most working and then my colleague and I started chatting about my frustrations.  He worked on XLinq for a bit and basically told me to suck it up and do the conversions and that it wasn’t that bad.  We walked through a few of the scenarios and indeed it really ended up all being isolated into one area that I could quickly scan through.  I could now remove my dependency on XmlDocument and have no other dependencies for this portable library.

Hooray for helpful people!  Even when you vent, the good peeps will still help out!

Changes to TagLib# for portability

After the full conversion, a few things remain.  Right now I have #ifdef’d out come of the interfaces and attributes that weren’t working.  Once I get to a point of porting all the tests over, I’ll decide if they are even needed.  Perhaps the biggest change though for users of this lib will be the removal of the default string path of file access.  In discussing with some folks, I could have tried to make a portable storage layer work, but it started to make less sense quickly to do that in the library and leave that simple task to the app developer.  This provides flexibility for the app to do what they want without the library trying to work around how different platforms do their file IO routines.  What that means is that the default way of reading a file’s tags changes from:

var tagFile = File.Create("ironlionzion.mp3");
var tags = tagFile.GetTag(TagTypes.Id3v2);
string album = tags.Album;


' file is a StorageFile
var fileStream = await file.OpenStreamForReadAsync();
var tagFile = File.Create(new StreamFileAbstraction(file.Name, fileStream, fileStream));
var tags = tagFile.GetTag(TagTypes.Id3v2);
string album = tags.Album;

in a simple case.  Yes, you as the app author have to write a bit more code, but it puts you in control of ensuring the file location you are reading.  You can see here that I did add my StreamFileAbstraction class to my fork by default, which was the key in the Silverlight port and is actually the key for WinRT as well.  Any app developer can create their own IFileAbstraction implementation and substitute it in the ctor of the create functions and be ready to read.  I actually did this in the test project to re-implement a LocalFileAbstraction for test purposes and used the System.IO.File classes to achieve that, which are available when running VS unit tests.


What started out as a frustrating exercise turned out to be helpful for me to better understand PCLs and hopefully add value to those who have been asking for this.  As mentioned, this isn’t fully tested and still a ways to go, so if you use it please log bugs (and help fix them) to complete the implementation.  I won’t be working on this full time of course, but do hope to get the test suite ported over as well.  Here are some relevant links:

Hope this helps!

| Comments

In talking with a friend about some Windows Phone 7 and Silverlight stuff recently.  He was giving me some great feedback about a few things (all of which I’ve passed along).  One of the things was what I felt was a common task that might exist in the mobile space but admittedly isn’t as clear if you are just coming to WP7 development.  The scenario is that of downloading media files and storing them for later playback.

WP7 does not have a storage mechanism like SQLLite on the device, but since it is Silverlight, you do have Isolated Storage you can use leveraging the same .NET class libraries from the full framework.  Here’s a sample app that demonstrates downloading an MP3 and storing for later playback on Windows Phone 7.

The Scenario

First, to make the scenario clear, say you are building a specific app for your brand’s media archives (audio and/or video).  You want to enable the user to selectively (or automatically) download the media to their device.  You want the user to be able to playback the downloaded media while offline later.  The media in this scenario is an MP3 file, a common audio format.

The Pieces

In order to do this we’ll assume the following:

  • You have an absolute URI to the MP3 file.
  • You have some type of UI to display to the user a list of media for your application
  • Some type of UI to control the playback (play/stop/etc.)

For demonstration purposes my UI will not be very ‘user friendly’ as it is meant to be diagnostic in explaining the task.  I will have a ListBox that I’ll use to bind to the list of downloaded items, a TextBox to give an option to download an MP3 file, and three (3) buttons: Download (to fetch the file and store for later), Play and Stop.  Here is the XAML I’ve used starting with the blank Windows Phone Application using the Windows Phone Developer Tools available to developers.

   1: <!--LayoutRoot is the root grid where all page content is placed-->
   2: <Grid x:Name="LayoutRoot" Background="Transparent">
   3:     <Grid.RowDefinitions>
   4:         <RowDefinition Height="Auto"/>
   5:         <RowDefinition Height="*"/>
   6:     </Grid.RowDefinitions>
   8:     <!--TitlePanel contains the name of the application and page title-->
   9:     <StackPanel x:Name="TitlePanel" Grid.Row="0" Margin="12,9,0,40">
  10:         <TextBlock x:Name="ApplicationTitle" Text="AUDIO SAMPLES" Style="{StaticResource PhoneTextNormalStyle}"/>
  11:         <TextBlock x:Name="PageTitle" Text="MP3" Margin="9,-8,0,0" Style="{StaticResource PhoneTextTitle1Style}"/>
  12:     </StackPanel>
  14:     <!--ContentPanel - place additional content here-->
  15:     <Grid x:Name="ContentPanel" Grid.Row="1" Margin="12,0,12,0">
  16:         <TextBlock Height="67" HorizontalAlignment="Left" Margin="6,6,0,0" Name="textBlock1" Text="Showing how to store and playback MP3 files on the device." VerticalAlignment="Top" TextWrapping="Wrap" />
  17:         <ListBox Height="150" HorizontalAlignment="Left" Margin="6,79,0,0" Name="listBox1" VerticalAlignment="Top" Width="444" />
  18:         <ProgressBar x:Name="DownloadProgress" IsIndeterminate="False" Style="{StaticResource PerformanceProgressBar}" Margin="6,-60,0,0"/>
  19:         <TextBlock Height="30" HorizontalAlignment="Left" Margin="12,298,0,0" Name="textBlock2" Text="URL to download" VerticalAlignment="Top" Width="175" />
  20:         <TextBox Height="72" HorizontalAlignment="Left" Margin="0,334,0,0" Name="mpsUri" VerticalAlignment="Top" Width="460" Text="http://files.sparklingclient.com/099_2010.07.09_WP7_Phones_In_The_Wild.mp3" />
  21:         <Button Content="DOWNLOAD" Height="72" HorizontalAlignment="Left" Margin="12,399,0,0" Name="button1" VerticalAlignment="Top" Width="438" Click="button1_Click" />
  22:         <Button Content="PLAY" Height="72" HorizontalAlignment="Left" Margin="12,477,0,0" Name="button2" VerticalAlignment="Top" Width="220" Click="button2_Click" />
  23:         <MediaElement Height="43" HorizontalAlignment="Left" Margin="6,555,0,0" Name="mediaPlayback" VerticalAlignment="Top" Width="438" />
  24:         <Button Content="STOP" Height="72" HorizontalAlignment="Left" Margin="230,477,0,0" Name="button3" VerticalAlignment="Top" Width="220" Click="button3_Click" />
  25:     </Grid>
  26: </Grid>

You can see that I’m not using any advanced pattern development here for two reasons: 1) it doesn’t need it and 2) I want to focus on specific tasks to isolate this learning.  Therefore if you don’t like Click handlers in your Button XAML, then you can flog me later :-).

Downloading the media

The first step in my sample is to download some media.  I’ve chosen a favorite podcast, the Sparkling Client as my sample file.  I hope they don’t mind.  In fact you may want to pick a small MP3 file to test on one of your servers so you don’t have to wait a while for the download to complete.

SIDE NOTE: I like how Sparkling Client has become more of short review of the times with regard to Silverlight.  Good to see some perspectives (and rumors).  It’s like a live version of Dave Campbell’s Silverlight Cream.

Because typing long things in the emulator is, well, annoying, I’ve prepopulated the URI TextBox (mpsUrl) with a specific URL.  Feel free to change this or enter a new MP3 URI in the emulator.  The next step is to execute the download.  I’m choosing to use WebClient.OpenReadAsync in this regard.  It’s simple and gets the job done.  Admittedly I wish that there was an OpenReadDownloadProgress argument available (as there is for DownloadStringAsync) but there is not. 

Because there is no actual progress value provided for me, I do want to provide the user some feedback that something is happening.  To do this in my sample I am using Jeff Wilcox’s “High Performance ProgressBar” for Windows Phone 7.  I’m not going to go into the details of why and how to hook it up – read his post.  I followed the exact same instructions. 

NOTE: When using the progress bar, don’t (only) set Visibility to collapsed if that is one of your mechanisms for displaying/hiding it, but be sure to set IsIndeterminate to false when not using it.

When I start the download, I start the ProgressBar to show the user some type of feedback that something is happening.  Here’s the code for the function:

   1: private void button1_Click(object sender, RoutedEventArgs e)
   2: {
   3:     string fileName = System.IO.Path.GetFileName(mpsUri.Text);
   5:     // start the download of the MP3
   6:     WebClient wc = new WebClient();
   8:     wc.OpenReadCompleted += ((s, args) =>
   9:         {
  10:             DownloadProgress.IsIndeterminate = false;
  12:             // once get the streams, put in isolated storage
  14:         });
  16:     wc.OpenReadAsync(new System.Uri(mpsUri.Text, System.UriKind.RelativeOrAbsolute));
  17:     DownloadProgress.IsIndeterminate = true;
  18: }

And here’s a quick screenshot of what it looks like running:

Windows Phone 7 progress bar

Simple enough…let’s store the downloaded bits now.

Storing the media to IsolatedStorage

The result of OpenReadAsync is a Stream.  Using IsolatedStorage and specifically IsolatedStorageFileStream, I can write out those bits to a file that is stored in my device’s storage location.  Normally in the browser Silverlight world I would have to calculate the amount of storage needed, see if it is available and, if not, request a quota increase to the user.  I don’t have to do that in the phone world.  I can just begin to write out the data.  Ideally, however, I should check for available space since it is entirely possible the user has used all their storage.  This sample does not accommodate that logic.

In my OpenReadCompleted event I add the following logic:

   1: // snipped
   2: wc.OpenReadCompleted += ((s, args) =>
   3: {
   4:     DownloadProgress.IsIndeterminate = false;
   6:     // once get the streams, put in isolated storage
   7:     using (var store = IsolatedStorageFile.GetUserStoreForApplication())
   8:     {
   9:         if (store.FileExists(fileName))
  10:         {
  11:             store.DeleteFile(fileName);
  12:         }
  14:         using (var fs = new IsolatedStorageFileStream(fileName, FileMode.Create, store))
  15:         {
  16:             byte[] bytesInStream = new byte[args.Result.Length];
  17:             args.Result.Read(bytesInStream, 0, (int)bytesInStream.Length);
  18:             fs.Write(bytesInStream, 0, bytesInStream.Length);
  19:             fs.Flush();
  20:         }
  21:     }
  23:     RefreshIsoFiles();
  25: });
  26: // snipped

You can see that I’m writing out the file stream to a file using the same name as the MP3 file literally (which may not make sense to the user, so again this is one of those ‘polish’ areas you’d want to make better and perhaps organize the files in IsolatedStorage better).

The last step you see is a call to RefreshIsoFiles.  This is a function that I also call when the first user interface page is loaded.  It traverses the IsolatedStorage for the app to display the already stored media in the ListBox in our XAML:

   1: private void RefreshIsoFiles()
   2: {
   3:     string[] fileList;
   5:     using (var store = IsolatedStorageFile.GetUserStoreForApplication())
   6:     {
   7:         fileList = store.GetFileNames();
   8:     }
   9:     listBox1.ItemsSource = fileList;
  10: }

Now we have our data downloaded and ready for playback.

Playing back the stored media

So great, now you have a stored MP3 file and you want to play it back.  Remember the XAML above and that I have a MediaElement there.  MediaElement is so simple at playing back media files from a URI.  Simply set the source of MediaElement and you can then call Play() and other functions. 

UPDATE: Well, you learn something new always.  Corrado pointed out below in comments that in WP7 you can SetSource directly to an IsolatedStorageFileStream...so while the following is interesting, it doesn't appear to be required in WP7 :-).

The challenge is that we now have our media in IsolatedStorage and there isn’t a URI scheme for IsolatedStorage that is predictable to the developer.  What we are left with is opening the media as a Stream and feeding that to the MediaElement.  This introduces MediaStreamSource.  If you aren’t familiar with this API, you’re probably not alone.  This is the API that enables a few scenarios, namely Smooth Streaming playback for Silverlight.  It is an extensible API so that you could wrap your own decode logic, etc. as needed.  Now given that MP3 is a common format you’d think it would be simple to do this…but there isn’t one built-in method for these various different codecs.

When MediaStreamSource was introduced, the program manager on that feature had written some helper files as code samples for developers to use.  One of them was an MP3 MediaStreamSource helper.  I wrote about them and where you can get them: MediaStreamSource for Silverlight.  Here’s where some awesome code re-use comes in to play.

I downloaded the ManagedMediaHelpers project and built the Mp3MediaStreamSource project (which also builds MediaHelpers).  In my WP7 project I simply added a reference to these in my project.  I was able to use these Silverlight binaries directly in my WP7 project! (I’ve included the compiled binaries in this sample for convenience but you can also see the source link in the article above.)

Now I need to read the media from IsolatedStorage as a stream, feed that Stream into my MediaStreamSource, and set that as the source for my MediaElement.  Here’s the relevant code on the Play button on my sample:

   1: private void button2_Click(object sender, RoutedEventArgs e)
   2: {
   3:     if (listBox1.SelectedItem == null)
   4:     {
   5:         MessageBox.Show("choose an item to play back!");
   6:     }
   7:     else
   8:     {
   9:         using (var store = IsolatedStorageFile.GetUserStoreForApplication())
  10:         {
  11:             audio = store.OpenFile(listBox1.SelectedItem.ToString(), FileMode.Open);
  12:             // play it back as a MSS
  13:             mss = new Media.Mp3MediaStreamSource(audio);
  14:             mediaPlayback.SetSource(mss);
  15:         }
  16:     }
  17: }

The media now plays back on the phone.  My Stop button basically closes the stream (audio and mss are member variables of the project) and nulls out references.


This is I think what might be a common application scenario for WP7 (downloading media for playback later).  Hopefully over time our platform will improve to make some of this better (i.e., progress indication), but for now this should help those get started on this task.  The meat of the solution is in the MediaStreamSource implementation.  If you are working with MP3 format, then the sample code will help you greatly as it’s mostly done!  There are other implementations floating around for WAV and other things as well if you need them.

Hopefully this might nudge Chris along the right path and be a helpful tip to others as well.  If you have any feedback on the implementation or a better way of doing this, please share!  Here’s the solution bits to my sample in full: Mp3StoreandPlayback.zip.

Hope this helps!

This work is licensed under a Creative Commons Attribution By license.

| Comments

I’m working on a little sample application for music management in Silverlight using WCF RIA Services and some other new Silverlight 4 features.  One thing that I wanted to add to the application was the ability to drag an audio file and either lookup the data and/or add a new album/artist/song to the library automatically.

Audio formats have a ‘tag’ format known as ID3.  It’s a format used for audio file metadata that is used in Windows Media Player, iTunes, and various hardware devices as well.  Over the years there has been an evolution of this format, with the older ID3v1 format basically taking up a header space with fixed character spaces for various things like Album, Artist, Title, Year, Track.  Over time though the ID3v2 format has been adapted more as it is more flexible for things like album art, and longer titles, etc.  There are various implementations of ID3 libraries for .NET that developers can choose from.  All of these implementations don’t take into account Silverlight unfortunately.

Silverlight can only reference Silverlight-compatible libraries.  Most of these libraries were targeted for the full .NET Framework and thus I can’t binary reference them.  Luckily most of them (except one) are Open Source so I could tinker.  I took the step of simply copying the files to a Silverlight project and recompiling.  This did not work 100% in a single task.  Most of the libraries had some form of Serialization attributes/constructors and almost all used some form of ASCII encoding for various string manipulation of byte arrays.

I settled on TagLib# as the library that was the easiest to modify for me.  I had to make the same changes I mentioned above to this library as well.  I created a new Silverlight 4 class library and compiled it as such.  One thing that TagLib# didn’t have was a stream input implementation.  Most of the libraries, in fact, assumed a local file path.  Luckily the library was written using a generic ‘File’ interface, so I just had to create my own StreamFileAbstraction.  I chose to do this within my project rather than the base library.  It was easy since the LocalFileAbstraction actually perfomed an Open on the file as it’s first task and set some public variables.  My abstraction basically just hands the stream already and ready to go.

Now, using the Silverlight 4 drop target feature, I created just a simple test harness to test my theory.  My XAML basically is this (pretty rudimentary just to test my theory):

   1: <Grid x:Name="LayoutRoot" Background="White">
   2:         <StackPanel>
   3:             <Border x:Name="DropZone" Width="700" Height="300" Background="Silver" CornerRadius="8" AllowDrop="True" Drop="DropZone_Drop">
   4:                 <TextBlock TextWrapping="Wrap" Text="drop here" FontSize="64" HorizontalAlignment="Center" VerticalAlignment="Center" Foreground="Gray"/>
   5:             </Border>
   6:             <Grid Height="255" Width="700">
   7:                 <Grid.ColumnDefinitions>
   8:                     <ColumnDefinition Width="111"/>
   9:                     <ColumnDefinition Width="*"/>
  10:                 </Grid.ColumnDefinitions>
  11:                 <Grid.RowDefinitions>
  12:                     <RowDefinition Height="Auto"/>
  13:                     <RowDefinition Height="Auto"/>
  14:                     <RowDefinition Height="Auto"/>
  15:                     <RowDefinition Height="Auto" />
  16:                     <RowDefinition Height="50*" />
  17:                 </Grid.RowDefinitions>
  18:                 <dataInput:Label Content="Artist" HorizontalAlignment="Right" VerticalAlignment="Top" FontWeight="Bold" Margin="4" />
  19:                 <dataInput:Label Content="Album" Grid.Row="1" HorizontalAlignment="Right" VerticalAlignment="Top" FontWeight="Bold" Margin="4" />
  20:                 <dataInput:Label Content="Title" Grid.Row="2" HorizontalAlignment="Right" VerticalAlignment="Top" FontWeight="Bold" Margin="4" />
  21:                 <dataInput:Label Grid.Column="1" HorizontalAlignment="Left" Name="Artist" Margin="4" />
  22:                 <dataInput:Label Grid.Column="1" Grid.Row="1" HorizontalAlignment="Left" Name="Album" Margin="4" />
  23:                 <dataInput:Label Grid.Column="1" Grid.Row="2" HorizontalAlignment="Left" VerticalAlignment="Top" Name="Title" Margin="4" />
  24:                 <Image Grid.Column="1" Grid.Row="4" Height="118" HorizontalAlignment="Left" Margin="4,2,0,0" Name="AlbumArt" Stretch="Fill" VerticalAlignment="Top" Width="118" />
  25:             </Grid>
  26:         </StackPanel>
  27:     </Grid>

Notice on the Border the AllowDrop=”True” attribute.  This tells Silverlight that the element can be used as a drop target (for a file from the file system).  The rendered UI looks like this:

Sample MP3 test UI

You may also notice the Drop attribute on the Border element that maps to the event handler DropZone_Drop.  This event handler basically gives us an event argument that represents the dropped objects on the surface (yes you can drop more than one).  The initial stub of this function looks like this:

   1: private void DropZone_Drop(object sender, DragEventArgs e)
   2: {
   3:     IDataObject drop = e.Data as IDataObject;
   5:     object data = drop.GetData(DataFormats.FileDrop);
   7:     FileInfo[] files = data as FileInfo[];
   9:     FileInfo file = files[0];
  10: }

I’m being a little verbose in the code to show each of the steps.  As you can see you get a FileInfo array and can pull items out of that.  For my sample I’m just assuming one item was dropped.  In the next steps I just need to get the Stream from the file and use my library.  Here is the full function (with a quick check to make sure it is a supported audio file):

   1: private void DropZone_Drop(object sender, DragEventArgs e)
   2: {
   3:     IDataObject drop = e.Data as IDataObject;
   5:     object data = drop.GetData(DataFormats.FileDrop);
   7:     FileInfo[] files = data as FileInfo[];
   9:     FileInfo file = files[0];
  11:     if (file.Extension.ToLower() != ".mp3" && file.Extension.ToLower() != ".wma")
  12:     {
  13:         MessageBox.Show("Must be an MP3 file");
  14:     }
  15:     else
  16:     {
  17:         Stream fileStream = file.OpenRead();
  19:         TagLib.File.IFileAbstraction fileAbstraction = new StreamFileAbstraction(fileStream, file.Name);
  21:         TagLib.File tagFile = TagLib.File.Create(fileAbstraction);
  23:         if (tagFile.Tag.TagTypes.HasFlag(TagLib.TagTypes.Id3v2))
  24:         {
  25:             Artist.Content = tagFile.Tag.FirstAlbumArtist;
  26:             Album.Content = string.IsNullOrEmpty(tagFile.Tag.Album) ? "NO ALBUM NAME" : tagFile.Tag.Album;
  27:             Title.Content = tagFile.Tag.Title;
  28:             if (tagFile.Tag.Pictures.Length > 0)
  29:             {
  30:                 IPicture pic = tagFile.Tag.Pictures[0];
  31:                 MemoryStream img = new MemoryStream(pic.Data.Data);
  32:                 BitmapImage bmp = new BitmapImage();
  33:                 bmp.SetSource(img);
  35:                 AlbumArt.Source = bmp;
  36:             }
  37:         }
  38:         else
  39:         {
  40:             MessageBox.Show("no id3v2 tag");
  41:         }
  42:     }

Once all the pieces are together you drag an audio file on the drop surface and the items will populate.  Here’s a quick video showing how it all works together.

Get Microsoft Silverlight

So this is just a start – and I’ve got only the tag reading working…didn’t bother looking at the other parts of the library so I know it isn’t fully ported for Silverlight.

What do you think?  Found a better implementation of ID3 tag reading?

| Comments

You may be reading the title and wondering what is MediaStreamSourceMediaStreamSource is a piece of the Silverlight runtime that removes a the influence of a media file's container, giving developers direct access to APIs for manipulating encoded elementary audio and video streams.


Basically it can enable you as the developer to implement file parsers/etc. in managed code instead of Silverlight, enabling support beyond the native built-in formats for media.  There hasn’t been much information about these types of topics, and I’d agree that for the mainstream, they may be a bit more advanced media scenarios for when the default containers and formats aren’t enough for your use.  Outside of the MSDN documentation there hasn’t really been any good samples of this use either.

Until now.

One of the program managers on the Silverlight media team, Larry Olson, has just provided a detailed public sample of the MediaStreamSource in action on the MSDN Code Gallery site.  He calls the effort ManagedMediaHelpers.  The project contains:

    • Silverlight class library (MediaParsers) which has helper classes for working with MP3 files, including being able to find the right point in an MP3 to begin playback.
    • Silverlight test project for NUnit
    • Silverlight class library (Mp3MediaStreamSource) which has logic for using MP3 file streams
    • Silverlight Application Demo which shows the interaction between a MediaElement and a MediaStreamSource

Why would you want this?  Larry outlines in the project:

“For one thing, having access to elementary streams means that developers can now implement scenarios that other solutions haven't necessarily provided thus far. One example of this is adaptive streaming or multi-bitrate support as was seen during the 2008 Olympics.

For another reason, having access to elementary streams allows developers to implement scenarios that the Silverlight runtime hasn't had a chance to implement yet or that the runtime might not be able to implement in the same timeframe that a developer wants it. Examples of this could be, RTSP:T protocol support, SHOUTcast protocol support, seamless audio looping, ID3 v1 and ID3 v2 metadata support, and many other scenarios.”

This is a great sample and source for those working with media within Silverlight.  Right now it is audio only, MP3 support, but gives you an idea of the MediaStreamSource API and functionality you could implement.

| Comments

while preparing for the code trip, we have several 'on-board' needs.  one of which is a quick method to get our content encoded for consumption by devices and frameworks (i.e., silverlight).  we want a smooth method so that we aren't boggled down with multiple tools opening and changing settings, cutting and pasting, etc.

so, inspired by my colleagues post about using workflow to automate, i set about the task.  we're going to be making several assumptions along our production, one of which is we know we'll have multiple video/audio sources and that there *will* be some post-edit being done.  but once we have that post edit completed, we'll want all videos to feel similar (size, quality, etc.) and encoded for multiple uses.  i've pretty much decided that there really are 3 formats that would suffice the world: WMV, MP4, MP3.  if we get all of these, we can accommodate most.  here's how i justify that:

    • WMV: we'll have standard (4:3) and widescreen (16:9) format for viewing offline as well as online via silverlight (we'll be using the widescreen online most likely).  we'll also have a Zune formatted version for quick updating.
    • MP4: itunes, ipod, mac viewing in both standard and widescreen formats
    • MP3: audio format beloved by all

so the first step is to tackle the how.  if you don't know, expression encoder has a command-line interface.  this is especially helpful for a few things, namely our batch processing (you can also save job files and send in a job to the command-line interface quickly).  so the first thing i did was wrap the input parameters into a windows workflow foundation activity.  luckily michael did a lot of this for me in his webcast :-).  it basically abstracts all the possible input parameters and enables you to optionally send them into the activity:

for the mp4 encoding i'm using a piece of software that also has an command-line interface.  this one isn't as flexible so given my two known encoding types i'm pretty much hard-wiring in some of the settings and only enabling the size parameters for alteration.

mp3 version -- let's get to that later.

now that i have my activities (one for WMV, one for MP4) in an activity library, i'm ready for a client tool.  for our purposes, we don't need a fancy GUI tool, so i settled for a command-line interface.  in my client i added a sequential workflow and then added my activities.  i re-used the WMV activity 3 times to alter the different settings and then use the output of them to feed into my MP4 activity (used twice).  the resulting visual workflow looks like this:

the input to the command line looks for a source WMV file, title, description, author, album.  these input parameters are sent as named parameters to the workflow activity:

var namedArguments = new Dictionary<string, object>();

namedArguments.Add("SourceFilename", args[0]);
namedArguments.Add("Title", args[1]);
namedArguments.Add("Description", args[2]);
namedArguments.Add("Author", args[3]);
namedArguments.Add("Album", args[4]);

which are then mapped to properties of the activities:

the activity libraries also expose a few other properties that i'm passing in to my commands.  for example, i want each video to have a bumper intro and then an icon overlay in the right location.  i'm able to pass in these parameters which then map to expression encoder properties.  i'm also able to tell it to make sure to letterbox content that isn't native 16:9 aspect ratio for the standard format encodings.  the result of these activities is that i have three WMV files appropriate for my use.  expression encoder also generates thumbnail images of a frame in each video.  i've not much use for them, so i added the last workflow code to simply clean up the jpeg images generate (delete them) from the output directory.

one problem i had was that expression encoder exposes a lot of properties but not for metadata individually.  i wanted to embed the appropriate metadata for the WMV files for the title, description, etc.  luckily there is one input parameter for encoder that i can append to my other custom ones, and that is "/Preset" which enables me to provide certain presets that will be passed in and here is where it allows me to make metadata a part of that preset.  the input parameter looks for a literal xml file so i have to create one.  i added the template as a resource in my project:

<?xml version="1.0" encoding="utf-8"?>
            Value="{0}" />
            Value="{1}" />
            Value="2008, Microsoft" />
            Value="{2}" />
            Value="{3}" />
            Value="{4}" />
            Value="{4}" />
            Value="Podcast" />
            Value="{5}" />

and then when the user executes the command-line interface, i take their input, merge it with the xml here and output a temporary xml file that is then passed into the named parameter dictionary for the workflow activity.  when no longer needed it is cleaned up (on the workflow completed event handler).  now my WMV file is complete with formats and metadata.

for the MP4 format i chose to use the resulting output of the WMV file and do a single pass there.  the settings for the tool weren't ideal for adding overlays, etc. so using the resulting WMV file and same bitrates i'm just passing in the resulting WMV and creating two MP4 formats.  boom.  done.  the metadata actually *was* parameters i could send into this tool, so it was easy to ensure that metadata was in there.

now, on to the MP3 format.  sigh.  what i need is a tool that will enable me an WMV or MP4 input and extract the audio-only track into an MP3 file.  i found all sorts of tools that will do this, but none that can be automated from a command-line.  this is my last resulting automation problem for now.  if anyone has tips on how to do this, i'll send you a prize :-).

now on to decisions.  while we'll have several formats to offer viewers, we also want to have feeds with enclosures for readers.  that brings us to a decision.  podcast formats for enclosures only enable one enclosure.  so, dear reader, what do we choose?  i figure we offer a WMV, MP4 and MP3 feed uniquely...but then which format do we supply?  is this a lame question?  the widescreen will be the best quality, but will it render okay on all devices/readers?  what do you choose?

anyhow, a fun little project i finished (except for the MP3 -- prize awaiting) and thought i'd share how it is accomplished.  one little command "encodepipe.exe <file> <title> <desc> <author> <album>" and a few short minutes later i have all the formats i need.  the next step is to automate upload to a content delivery network so i don't have to pick and choose uploading!

we just posted the schedule for our trip, so if you want to subscribe to the feed to be notified when we'll start putting out some content, that would be cool.  real-time updates via twitter as well.  see you on the road!