Hadoop for .NET Developers: Setting Up a Desktop Development Environment
NOTE This post is one in a series on Hadoop for .NET Developers.
If you are a .NET developer, you will want to setup a desktop development environment with the following components:
- Visual Studio 2010 or 2012
- NuGet Package Installer for Visual Studio
- A Local, Single Node Hadoop “Cluster”
Having these components installed on your desktop will allow you to develop against Hadoop locally as well as against a remote cluster (whether on-premise on in the cloud). You might be able to get away with not installing Hadoop locally, but most of the .NET-oriented documentation I’ve found assumes this is your setup.
I will assume you are comfortable installing Visual Studio on your own and the NuGet site provides simple enough installation options. I do recommend installing Visual Studio and NuGet first and then making sure your system is up-to-date with patches before proceeding with the Hadoop installation.
The Hadoop installation is very straightforward. That said, I once experienced an installation failure that prevented subsequent re-installation attempts. To fix this, I navigated to the <System>:Users<user name>AppDataLocalMicrosoftWeb Platform InstallerInstallers folder and deleted the installer associated with the Hadoop installation. After that, I was able to successfully re-install Hadoop.
Finally, I have found installing these components to a virtual machine running on my Windows 8 desktop (using Hyper-V) to provide me a reliable environment from within which to work and keeps my base desktop environment clean and stable. Whether you install to a VM or to your base desktop, please note that if your IP address changes, you will need to restart Hadoop as some components tend to hold onto the IP address with which the cluster was started. While I am told you could selectively restart components to clear the old IP address, I do not know which ones and have found that simply rebooting my development environment does the trick quickly enough :-)
Comments
Anonymous
August 20, 2013
What about VS 2012 ?Anonymous
August 20, 2013
Sorry for the oversight. I meant to say 2012 in the post. Its updated now.Anonymous
August 21, 2013
It is not <System>:UsersuserLocalMicrosoftWeb Platform InstallerInstallers but <System>:UsersuserAppDataLocalMicrosoftWeb Platform InstallerInstallers (missing the AppData Fragment)Anonymous
August 22, 2013
Thanks for catching that. I've updated the postAnonymous
September 16, 2013
private-repo-1.hortonworks.com/.../hdp-1.0.1-ms9.winpkg.msi is returning a 404 when the Web Platform Installer tries to download it.Anonymous
September 16, 2013
@Sean - Sorry I don't have an answer for this one. This would indicate there's a communication issue between the system to which you are installing the solution and the server you are downloading it from. You might contact Hortonworks on this one: hortonworks.com/.../contact-usAnonymous
February 07, 2014
Is there a way to set HDInsight emulator that it will bind to full machine name, not to localhost? Thanks Voffka