I have a project where I have a large C(100,20) number of combinations with minor work being done for each combination set.
I am using Spark .NET with visual studio as my technology (see setup below): https://learn.microsoft.com/en-us/dotnet/spark/tutorials/get-started
Spark .NET has a dataframe with SQL type commands. I am assuming I need to do a SQL type command to create the N choose K combinations with a user defined worker function to process the combinations.
The question is what does the code look like using Spark .NET with a DataFrame? If a DataFrame doesn't support an N choose K option, are there other options to keep the generation of the combinations distributed?