Share via


Channel9 Spider

Don't try this at home folks, but I had to. Actually mine is still running. And I don't know if I have enough storage for all of it anyway, but I have my eye on an external 2TB eSATA enclosure to make sure...

When I visit my friends in South Africa and Zambia the one issue that always come up is the lack of or caps on bandwidth. For example, in South Africa the typical broadband connection (ADSL) is 512kbps or 1Mbps with a 3GB montly limit. In Zambia, rural users of satellite connections spend in excess of $1400 PER MONTH for less bandwidth than that and frequent outages is just part of life.

Thus "sneakernet" or the postal service still wins most bandwidth competitions. In my case, I take Channel9 videos to my programmer friends in Lusaka whenever I visit. The first time it was just a few gigabytes or so of content; this time I'm hoping to give them a browsable list of content and as much as I can fit on whatever hard disks I can get donated. From that idea to a Channel9 spider.

Attached is the little C# program I used to walk from one forum page to another to harvest all the URLs for forum posts containing videos or screencasts. Then, using that list, it grabs all the forum posts and extract all the media URLs. Then I use Free Download Manager for the heavy lifting of actually downloading all the content.

It is a purpose-written spider for a one-off job, so read the comments at the top and don't expect error recovery, progress reports, etc. You will need to modify the code or download PowerCollections.

Channel9Spider.zip