With an increasing number of organisations migrating their data platforms to the cloud, there is also a demand for cloud technologies that allow utilising the existing skillsets in the organisation while also ensuring successful migration.
ETL developers often form a sizeable part of data teams in many organisations. These developers are well versed in the use of GUI based ETL tools as well as complex SQL and also have or are beginning to develop programming skills in languages such as Python.
In this 3 part article series, I will share an overview of
- a scalable data lake architecture for storage…
The first article in this series provided an overview of a data lake solution architecture using Data Fusion for data integration and Cloud Composer for orchestration.
In this article, I will provide an overview of the detailed solution design based on that architecture. This article assumes you have some basic understanding of GCP Data Fusion and Composer. If you are new to GCP, you can start by reading the previous article in this series to get an understanding of the different services used in the architecture before proceeding here.
Design Approach
The solution design described here provides a framework to ingest a…
Neha Joshi
Neha is a Cloud Consultant in Google Cloud’s Professional Services group. She specialises in Data and Advanced Analytics, helping large enterprises adopt GCP.
