Senior Databricks Engineer | AWS Databricks
We are seeking a Senior Databricks Engineer for an initial three-month contract with a leading Australian financial services organisation.
The successful candidate will help mature an existing Databricks platform into an enterprise-grade capability. This is a hands-on role focused on platform scalability, automation, governance and the reduction of technical debt.
Engagement details
- Three-month contract.
- Senior, hands-on engineering engagement.
- Immediate or near-term availability preferred.
Key responsibilities
- Design and establish a governed, domain-aligned Unity Catalog structure incorporating Bronze, Silver, Gold and Reporting layers.
- Support the migration from the organisation’s existing flat catalog structure to the new target-state architecture.
- Re-platform existing ingestion and transformation frameworks to align with the new catalog model.
- Develop metadata-driven, reusable and scalable data pipelines.
- Implement RBAC and ABAC controls across all catalog layers, ensuring least-privilege and domain-based access.
- Refresh the underlying AWS storage and security foundations, including S3 buckets, bucket policies, IAM roles and access controls.
- Maintain and extend Terraform-based Infrastructure as Code across the Databricks and AWS environments.
- Automate Databricks cluster provisioning and SQL Warehouse deployments, minimising manual configuration.
- Support a phased, low-risk migration that protects critical regulatory and business reporting workloads.
- Essential experience
Essential experience
- Strong commercial experience engineering and supporting enterprise Databricks platforms.
- Hands-on Databricks experience within AWS environments.
- Demonstrated experience designing and implementing Unity Catalog architectures.
- Strong understanding of lakehouse and medallion architecture, including Bronze, Silver and Gold data layers.
- Experience developing metadata-driven ingestion and transformation frameworks.
- Advanced Spark and PySpark engineering skills.
- Strong knowledge of Databricks governance, RBAC, ABAC and least-privilege security models.
- Practical AWS experience covering S3, bucket policies, IAM roles and secure cross-service access.
Highly desirable
- Experience within banking, payments, superannuation or another regulated financial-services environment.
- Knowledge of CPS 230 or other APRA-regulated reporting requirements.
- Experience supporting financial risk, regulatory or client SLA reporting.
If you are interested, please apply now. For a confidential discussion, please reach out to Josh.costigan@talentinternational.com