Databricks Data Engineer

We are seeking an experienced Databricks Data Engineer to design and develop scalable data pipelines and cloud-based data solutions using Databricks, Spark, PySpark, and SQL. The role will focus on building high-performance data platforms, optimizing data workflows, and ensuring data quality and reliability.

The ideal candidate will collaborate with data architects, analysts, and business stakeholders to deliver scalable, efficient, and business-driven data solutions.

1. Design and develop scalable data pipelines and ETL/ELT workflows using Databricks.

2. Build data processing solutions with Apache Spark, PySpark, and SQL.

3. Develop and optimize batch and near-real-time data pipelines.

4. Implement solutions using Delta Lake, Delta Live Tables (DLT), and Databricks Workflows.

5. Develop robust data ingestion, transformation, cleansing, and validation processes.

6. Integrate data from databases, APIs, files, and cloud storage platforms.

7. Optimize pipelines and Spark workloads for performance, scalability, and cost efficiency.

8. Implement data quality, governance, security, monitoring, and validation practices.

9. Leverage Azure, AWS, or GCP data services to build cloud-native solutions.

10. Collaborate with architects, analysts, and engineering teams to deliver business-focused data solutions.

11. Troubleshoot pipeline, data quality, and performance issues.

12. Follow best practices for Git, CI/CD, testing, and deployment.