Consultant - PySpark / Databricks Developer (Intermediate - Senior)

About the Role

We are looking for both permeant and contract for this role.

We're looking for a developer to help build and maintain data pipelines on Databricks for a financial services / wealth management data platform. The platform ingests data from multiple source systems, transforms it through a medallion architecture (bronze → silver → gold) using PySpark, and publishes curated data to downstream consumers via file exports and Kafka. You'll work alongside a senior engineer, writing transform code, fixing pipeline issues, and helping keep the environment healthy across dev/qa/uat.

This is a good fit for someone with solid Python fundamentals and some Spark/SQL exposure who wants to grow into a data engineering specialist. You won't be expected to know Databricks internals on day one — you will be expected to learn fast, ask good questions, and write clean,

Writing and maintaining PySpark transforms (hand-written, not a generic framework) for silver and gold layer tables — things like customer, account, and transaction data models.

Working with Lakeflow Declarative Pipelines (Databricks' current pipeline framework — successor to DLT) and Auto CDC / SCD Type 2 patterns for change-data-capture merges.

Querying and validating data in Unity Catalog across environments (dev/qa/uat) using SQL warehouses.

Debugging failed pipeline runs: reading pipeline event logs, tracing bad records, fixing schema drift or data quality issues.

Maintaining reference/lookup tables and small utility scripts (PowerShell/Python) used to operate the platform.

Writing and updating unit/integration tests for transform logic.

Participating in code review, using Git feature branches and merge requests (GitLab).

Keeping documentation current when you change how something works.

Python — comfortable writing clean, readable code; understands functions, modules, basic OOP.

Databricks

Some exposure to Apache Spark / PySpark, or strong SQL skills plus a willingness to learn Spark quickly.

Working knowledge of SQL (joins, aggregations, window functions).

Basic Git workflow: branches, commits, pull/merge requests.

Comfortable reading other people's code and stack traces, and debugging methodically (not guess-and-check).

Direct experience with Databricks (notebooks, jobs, clusters, or SQL warehouses).

Familiarity with Delta Lake, medallion architecture (bronze/silver/gold), or CDC/SCD concepts.

Exposure to Azure (this environment runs on Azure Databricks with Azure AD service principal auth).

Experience with Kafka or other streaming/event systems.

Experience with CI/CD pipelines (this project uses GitLab CI).

Prior work in financial services or a regulated data environment.

Can independently pick up a small transform bug or enhancement ticket, make the change, test it, and open a merge request.

Comfortable running existing operational scripts to check pipeline status, query tables, and diagnose failures without hand-holding.

Starting to take ownership of the Databricks environments and help write new transforms