Library Guide

Run Spark on Windows 10 OS

Run Spark on Windows 10 OS

In order for spark to run on Windows 10, we need to download the winutils.exe from https://github.com/srccodes/hadoop-common-2.2.0-bin and put it in the hadoop.bin directory.

To do this download the zip from https://github.com/srccodes/hadoop-common-2.2.0-bin and then put the files in the zip directory into the hadoop/bin directory (full path on my computer is C:hadoop in).

In your spark code, put this before invoking the JavaSparkContext:

        SparkConf conf = new SparkConf().setAppName("LeftJoin2");

        String home = System.getProperty("hadoop.home.dir");

        if (home == null) {
            System.setProperty("hadoop.home.dir", "C:hadoop");
        }

        conf.setIfMissing("spark.master", "local[2]");

        JavaSparkContext context = new JavaSparkContext(conf);

download file now