Institutional reporting data arrived as zipped CSV archives dropped into SharePoint. There was no automated path from that drop zone into a governed analytics layer — which meant every reporting cycle involved manual downloads, manual unzipping, and manual loads, with no schema enforcement and no way to reconcile a bad load.
I built an end-to-end pipeline that authenticates to SharePoint through Microsoft Graph using a service principal, resolves the latest archive by matching a date pattern in the filename, moves it into the Lakehouse with a Data Factory copy activity in binary mode, then hands off to a PySpark notebook that extracts the archive into a curated landing zone and loads 12 CSVs into Delta tables. Load behavior is schema-aware per table: full overwrite where the source is a snapshot, append where it is an event log, and merge (upsert) where records can be restated.