Enclosure Download

Article
12/06/2006

A while ago I posted details about the RSS Platform Download Engine. That post focused on downloading of feeds, but did not include additional details on enclosure downloads.

Enclosures are, as most readers know, files that are "attached" to items in an RSS feed. Typically, a publisher will include a reference to a binary file, which an RSS aggregator can optionally download when the feed content is downloaded. The most common example of enclosure use in RSS feeds is for podcasting, where the attached (or "enclosed") files are audio files.

As with feed download, we designed the enclosure download with server and client bandwidth in mind since feed as well as enclosure downloads also happen in the background. Their impact on foreground applications should be limited. Similarly, the impact of large enclosure downloads on servers should be limited.

Let me sketch how the enclosure download process works:

Every time the feed download engine runs it processes feeds that have the "Automatically Download Enclosures" setting set to true it. If it comes across a new item with an enclosure it adds the URL of the enclosure to a FIFO queue.
Before the enclosure is added to the queue, the URL is checked with Attachment Execution Service API (AES) to assure the enclosure file type is one of the permitted types. If it's not, the enclosure download is failed (IFeedEnclosure.LastDownloadError = FDE_DOWNLOAD_BLOCKED).
The first 4 enclosures in the queue are then handed off to the Background Intelligent Transfer Service (BITS). BITS is a background download service that ships in Windows and which enables downloading of files in the background while limiting its affects on network usage. In particular, BITS uses HTTP RANGE requests to download files in chunks. BITS also monitors whether foreground applications (like email or browser) are using the network, and if so, it throttles back its own network usage to limit its impact on those applications.
Once BITS completes downloading an enclosure, the Download Engine uses AES to save the enclosure to the folder corresponding to the feed. Saving via AES associates zone information with the file. The zone information is used when the file is launched at a later time.
If there are more enclosures waiting to be downloaded and there are less than 4 enclosure downloads active, the next enclosure is handed off to BITS as in step #3.
If, however, the server of the enclosure does not support HTTP RANGE requests, the Platform Download Engine falls back to downloading the enclosure via a regular HTTP GET request. If this attempt fails as well, then the enclosure download is fails and will not be attempted again automatically.

Note that the enclosure fall-back download (HTTP GET) is size limited to 15MB to limit the impact of denial of service (DoS) attacks against the RSS Platform Download Engine. Since the RSS Platform Download Engine runs in the background, a malicious server could consume all of the client's download bandwidth without the user having any idea. Enclosure download via BITS (HTTP RANGE requests) is less impacted by such an attack and is consequently not size limited.

In other words, if you are an enclosure publisher that wants to serve enclosures larger then 15MB to IE7 users, then you should use HTTP servers that support HTTP RANGE requests. Most popular web servers support HTTP RANGE requests.

It's also worth noting, that when a server does not support HTTP RANGE requests, the RSS Platform Download Engine will issue two requests for each file (the first testing for HTTP RANGE support, and the second to download the file without range support).

For more details on the security measures used to protect applications and users from potentially malicious enclosures, see Miladin's enclosure security post.

I hope that this description of the enclosure download process explains the "multiple-requests" that some publishers have seen, as well the security restrictions associated with enclosure downloads.

-Walter vonKoch

Program Manager

Comments

Anonymous
December 06, 2006
I'm using IE7 and the feed list to manage the podcast I listen to and the auto-downloads work well. Except for trying to get WMP to autodiscover the new media. Right now I'm using a manual process.

Setup the feed to auto-download
Click the "view files" button and copy the folder to the clipboard
Go to WMP and add a new folder to monitor. Paste the path from the clipboard.
Setup a Auto-Playlist to include the new podcast
Setup the sync to my portable player (a Sprint PPC-6700) Any suggestions for simplifying this process? Are there events that an app could respond to notice that new feed is downloading attachments?

Anonymous
December 06, 2006
Is there a way to define the directory for downloads? It would be so much better for findnig the files later.
Anonymous
December 06, 2006
The comment has been removed
Anonymous
December 06, 2006
That will place a serious limitation on the Internet Explorer browser for feeds that contain video since it doesn't take much video to pass 15MB. This will adversly impact not only video bloggers, but all public media content producers. Just as more and more public broadcasting stations are adopting RSS feeds for sharing their local content, you're capping the file size in way that prevents most video and nearly any full-length audio program. There has to be a better alternative than HTTP-Range checking. The point of RSS is to make this kind of content syndication simple. RSS is an XML format and should be totally independant of the server.
Anonymous
December 07, 2006
The comment has been removed
Anonymous
December 09, 2006
It's intresting what you did with the security model, but I think you should allow "infinte" sized simple HTTP GET downloads for enclosures that come from the same server who supplied the feed.
Anonymous
December 26, 2006
So if I am reading all of this correctly. IE7 is really not an aggragator in that it does not actually download all of the shows and then place them with your other Rich Media. The real reason podcasting took off - was not that you could add an enclosure and have some one subscribe and download to it. Rather it went a crucial step further (or at least Adam Curry did) it was setup such that the files after being downloaded were placed with all of your other rich media files so the subscriber could easily find and consume the media. With out that crucial step - it is not podcasting. It is just downloading via RSS. At least if the file size is less than 15 Meg. Which by the way most podcasts are well over 15 Meg in size. So when is MS coming out with a truly intergrated Podcasting solution for the subscriber? Rob @ podCast411
Anonymous
January 04, 2007
How about a new year's resolution for the RSS team: Allow users to choose the maximum size of enclosure to download and let them decide whether or not to trust the server. If you are feeling really virtuous, how about another one: Allow users to set the download location for enclosures. I know that Microsoft is always being lambasted for not catering to the lowest common denonminator but these two things are pretty vital to achieving the functionality needed to make IE7 achieve its objectives.
Anonymous
January 22, 2007
How clever to aviod DoS attacks the size of the enclosure is limited to 15MB. Nobody would ever have a podcast bigger than that surely? Check out: http://msdn.microsoft.com/msdnmag/podcast/default.aspx 8 podcasts, 4 of which would not be automatically downloaded by IE7. What internet vandal would host such large files?
Anonymous
January 26, 2007
@Wilbur, You might have misunderstood the 15MB limit. It's a fallback limit which only applies if the webserver does not support HTTP RANGE requests. The enclosure server download.microsoft.com does support HTTP RANGE requests. So if you try it and turn on automatic enclosure download for MSDN podcast feed you will see all enclosures downloaded. I hope this clears up the confusion.

Walter [MSFT}

Anonymous
March 10, 2007
Does the feed manager recognise links with the micro format rel=enclosure as enclosures or does it look for the <enclosure> tag? What about Atom enclosures?
Anonymous
April 20, 2007
Please make it possible to set the download location of enclosures!
Anonymous
May 08, 2007
One more request for the ability to specify a download folder for each feed. Without this I am left with iTunes or manually downloading my podcasts so I can burn them to CD for listening in the car. Dann
Anonymous
May 31, 2007
Walter - You're not being misunderstood. You need to make it so that this does not bother users by not allowing them to download podcasts. Here's a better way of preventing bandwidth wastage: Monitor the feeds bandwidth usage pattern and inform the user of any abnormalities that appear. This will actually turn a stupid restriction into a useful feature.
Anonymous
June 01, 2007
I agree with Dann, being able to specify (different) download folders for different auto downloaded feeds would be a great advantage. Also feeds that are different auto downloaded go to Temporary Internet Files and so could easily be deleted forgetting that there are feeds downloaded.
Anonymous
July 05, 2007
The comment has been removed
Anonymous
July 10, 2007
It's intresting what you did with the security model, but I think you should allow "infinte" sized simple HTTP GET downloads for enclosures that come from the same server who supplied the feed.
Anonymous
September 05, 2007
Thanks Walter, Good to know. 15mb limit would be useless for our guys. I see what you've done now, it's starting to make sense. Thanks again.
Anonymous
January 18, 2008
The comment has been removed
Anonymous
January 18, 2008
How about a new year's resolution for the RSS team: Allow users to choose the maximum size of enclosure to download and let them decide whether or not to trust the server. If you are feeling really virtuous, how about another one: Allow users to set the download location for enclosures. I know that Microsoft is always being lambasted for not catering to the lowest common denonminator but these two things are pretty vital to achieving the functionality needed to make IE7 achieve its objectives.
Anonymous
January 18, 2008
That will place a serious limitation on the Internet Explorer browser for feeds that contain video since it doesn't take much video to pass 15MB. This will adversly impact not only video bloggers, but all public media content producers. Just as more and more public broadcasting stations are adopting RSS feeds for sharing their local content, you're capping the file size in way that prevents most video and nearly any full-length audio program. There has to be a better alternative than HTTP-Range checking. The point of RSS is to make this kind of content syndication simple. RSS is an XML format and should be totally independant of the server.
Anonymous
January 18, 2008
I'm using IE7 and the feed list to manage the podcast I listen to and the auto-downloads work well. Except for trying to get WMP to autodiscover the new media. Right now I'm using a manual process.

Setup the feed to auto-download
Click the "view files" button and copy the folder to the clipboard
Go to WMP and add a new folder to monitor. Paste the path from the clipboard.
Setup a Auto-Playlist to include the new podcast
Setup the sync to my portable player (a Sprint PPC-6700) Any suggestions for simplifying this process? Are there events that an app could respond to notice that new feed is downloading attachments?

Anonymous
March 20, 2008
How do I change the default directory for downloads?
Anonymous
March 26, 2008
The comment has been removed
Anonymous
April 15, 2008
Thanks for very interesting article. btw. I really enjoyed reading all of your posts. It’s interesting to read.
Anonymous
April 20, 2008
it is really good article, I found the reason for my problem!! Thanks
Anonymous
June 01, 2008
Thanks for the article. I really enjoyed reading all of your posts.
Anonymous
June 07, 2008
Thanks for very interesting article.
Anonymous
June 27, 2008
It's easy to overcome the 15 MB limit - just set the registry value HKCUSoftwareMicrosoftFeedsMaxEnclosureSize (DWORD) to a suitable value (in kB). There is a similar value for the maximum XML size (also 15 MB by default). If you are writing a podcast app that uses the common feed list, this is easy enough to adjust. More details are on MSDN.
Anonymous
June 27, 2008
The comment has been removed
Anonymous
June 30, 2008
now I get the multiple-requests subject. thanks! ;-)
Anonymous
July 08, 2008
Great article. It`s realy worth reading. I wish you further successes
Anonymous
July 24, 2008
It's intresting what you did with the security model, but I think you should allow "infinte" sized simple HTTP GET downloads for enclosures that come from the same server who supplied the feed.
Anonymous
August 04, 2008
Very interesting article. Good work!
Anonymous
August 23, 2008
The comment has been removed
Anonymous
October 01, 2008
15 MB limit - just set the registry value HKCUSoftwareMicrosoftFeedsMaxEnclosureSize (DWORD) to a suitable value (in kB). There is a similar value for the maximum XML size (also 15 MB by default) anyway thanks
Anonymous
November 13, 2008
<a href= SADDDDDDDDDDDDDDDDDDDD >DDDDDDDDDDDDDDDDDDD</a>, %-DDD,
Anonymous
November 18, 2008
The comment has been removed
Anonymous
October 28, 2009
The comment has been removed
Anonymous
April 14, 2010
Nice and very useful article. Thanks!
Anonymous
July 05, 2010
The comment has been removed
Anonymous
July 12, 2011
It would be very helpful for finding files later if there was a method of defining the directory for downloads.

Share via

Enclosure Download

Comments

Additional resources