Spark on Azure HDInsight

View Code

An example Pulumi component that deploys a Spark cluster on Azure HDInsight.

Running the App

1. Create a new stack:

   pulumi stack init dev
   

   Copy Copy

2. Login to Azure CLI (you will be prompted to do this during deployment if you forget this step):

   az login
   

   Copy Copy

3. Restore NPM dependencies:

   npm install
   

   Copy Copy

4. Configure target Azure environment:

   pulumi config set azure:location <location>
   pulumi config set azure:subscriptionId <YOUR_SUBSCRIPTION_ID>
   pulumi config set username <value>
   pulumi config set password --secret <value>
   

   Copy Copy

5. Run pulumi up to preview and deploy changes:

   $ pulumi up
   Previewing changes:
   ...
   
   Performing changes:
   ...
   info: 5 changes performed:
       + 5 resources created
   Update duration: 15m6s
   

   Copy Copy

6. Check the deployed Spark endpoint:

   $ pulumi stack output endpoint
   https://myspark1234abcd.azurehdinsight.net/
   
   # For instance, Jupyter notebooks are available at https://myspark1234abcd.azurehdinsight.net/jupyter/
   # Follow https://docs.microsoft.com/en-us/azure/hdinsight/spark/apache-spark-load-data-run-query to test it out
   

   Copy Copy