How to deploy a production ready data platform centered around Databricks and Azure services.
This post is an introduction and overview of what we want to set up. A production ready Data Platform consisting of Infrastructure and Framework, using Azure services and Databricks
The goal is to deploy and configure a production-ready data platform consisting of Azure Databricks Workspace at the heart, Azure Storage account as the data lake for the medallion architecture (bronze, silver, and gold data layers), Azure Key Vault for secrets, and securing everything with Vnet, Subnets, and Network Security Group in Azure. Beyond the infrastructure I will also cover the deployment of a Framework such that Databricks is configured completely and ready to work with as a Data Engineer or Data Scientist.