Data Engineer // Data Lake & Databricks

  • Australia
  • Sydney
  • Contract
  • AU$800 - AU$850 per day

Our client is seeking a skilled and motivated Data Engineer to join their dynamic team. The ideal candidate will possess extensive experience with Microsoft Azure, Databricks, real-time integrations, and data streaming (Kafka). You will be responsible for designing, building, and maintaining their data infrastructure to support their data analytics and business intelligence needs. Your expertise in data lakes, SQL procedures, and CICD pipelines will be critical to ensuring efficient and reliable data processes.

Responsibilities:

  1. Azure Data Solutions:

    • Design, implement, and manage data solutions using Azure Blob Storage, Azure Kubernetes Service (AKS), and Azure Data Factory (ADF).
    • Ensure the scalability and reliability of our Azure-based data infrastructure.

  2. Databricks Development:

    • Develop and maintain data pipelines using Databricks, with a focus on PySpark and Python.
    • Optimize data workflows for performance and cost-efficiency within the Databricks environment.

  3. Real-time Data Integration:

    • Design and implement real-time data integration solutions using data streaming technologies such as Kafka, Azure Functions, ADF, and Flink.
    • Develop and maintain CICD pipelines to automate deployment and monitoring of data streaming processes.

  4. Data Lake Construction:

    • Build and maintain data lakes on Databricks to support scalable and flexible data storage and analytics.
    • Ensure data quality, consistency, and security within the data lake environment.

  5. SQL Development:

    • Write, debug, and optimize complex PL/SQL and T-SQL procedures.
    • Collaborate with data analysts and other stakeholders to meet their data querying and reporting needs.

Requirements:

  • 6-10 years of experience in data engineering or a related role.
  • Strong expertise in Microsoft Azure services, including Azure Blob Storage, AKS, and ADF – A MUST
  • Proficiency in Databricks with a focus on PySpark and Python.
  • Hands-on experience with real-time data integration and streaming technologies (Kafka, Azure Functions, ADF, Flink).
  • Proven experience building and maintaining data lakes on Databricks – A MUST
  • Strong knowledge of PL/SQL and T-SQL, with hands-on experience in writing and debugging SQL procedures.
  • Excellent problem-solving skills and the ability to work in a fast-paced, collaborative environment.
  • Strong communication skills and the ability to work effectively with cross-functional teams.

Preferred Qualifications:

  • Master’s degree in Computer Science, Information Technology, or a related field.
  • Certifications in Microsoft Azure and/or Databricks.
  • Experience with other data integration and ETL tools.
  • Familiarity with additional programming languages and data processing frameworks.

If you think you have above skills and experiences, click ‘Apply’ button or send your resume to alex.nguyen@talentinternational.com

Apply now

Submit your details and attach your resume below. Hint: make sure all relevant experience is included in your CV and keep your message to the hiring team short and sweet - 2000 characters or less is perfect.