Azure HDInsight
An Azure managed cluster service for open-source analytics.
213 questions
Run C# mapreduce job
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(89.60000000000001, 86%, 18%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ELS%3C/text%3E%3C/svg%3E)
Lilukshi Silva
1
Reputation point
I am a beginner to Hadoop MapReduce.
I have implemented a MapReduce job in visual C# and want to run it locally. As I understood, the HDInsight emulator hasn't been updated for a long time. What else options I have, to run the job locally?
{count} votes
-
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(134.4, 9%, 23%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EKM%3C/text%3E%3C/svg%3E)
KranthiPakala-MSFT 46,512 Reputation points Microsoft Employee2022-02-22T22:04:48.873+00:00 Hello @Lilukshi Silva ,
Thanks for the question and using MS Q&A platform.
We have reached out to product team regarding this requirement and awaiting response from them. Will keep you posted as soon as we hear back from the team.
Thank you for your patience!
-
KranthiPakala-MSFT 46,512 Reputation points Microsoft Employee
2022-02-24T01:11:34.477+00:00 Hello @Lilukshi Silva ,
In general Hadoop Mapreduce job can be run in below three modes:
- MR Job can be run in standalone mode - where everything runs in a single JVM or Process.
mapreduce.framework.namehas to be set tolocal
https://blogs.justenougharchitecture.com/apache-hadoop/ - One Node Hadoop cluster can be setup on the Work Machine and the job can be submitted into it.
Related article - https://www.edureka.co/blog/install-hadoop-single-node-hadoop-cluster - The job can be submitted from Work Machine into the Remote Running Hadoop Cluster.
Related doc - https://learn.microsoft.com/en-us/azure/hdinsight/hadoop/apache-hadoop-dotnet-csharp-mapreduce-streaming
Do let us know if you have further questions or please elaborate your requirement so that we can assist accordingly.
- MR Job can be run in standalone mode - where everything runs in a single JVM or Process.
-
Lilukshi Silva 1 Reputation point
2022-02-25T20:30:24.82+00:00 Thank you very much replying. Actually, I setup a single-node Hadoop cluster but as I understood you need to write your MapReduce job in Java to run on it.
But I implemented in Visual C# and installed Microsoft .NET MapReduce API for Hadoop in Visual studio as described in https://www.youtube.com/watch?v=uyi41nrhlhw.However to run it, I cannot install windows HDInsight emulator. Therefore, need to know other options available to run my MapReduce job written in C#.
Sign in to comment
Sign in to answer