Senior Data Engineer -Databricks
Coimbatore · 2+ years experience · Full-time · Hybrid · Posted Oct 16, 2026
At a glance
- Experience
- 2+ years experience
- Employment
- Full-time
- Workplace
- Hybrid
- Location
- Coimbatore
About the role
- ShyftLabs is seeking a Senior Data Engineer to build scalable data pipelines and solutions on the Databricks Lakehouse Platform. The role involves working with Fortune 500 clients to deliver innovative digital data products.
Responsibilities
- Design, develop, and maintain scalable ETL/ELT pipelines using Databricks, PySpark, and SQL.
- Integrate data from multiple sources, including databases, Amazon S3, files, and REST APIs.
- Build data pipelines with Databricks Unity Catalog.
- Implement business logic, data transformations, and dimensional data models.
- Create, schedule, monitor, and optimize Databricks Jobs and Workflows.
- Design and manage Delta Lake tables using Medallion Architecture (Bronze, Silver, Gold).
- Ensure data quality through validations, error handling, logging, and monitoring.
- Optimize Spark workloads for performance, scalability, and reliability.
- Collaborate with cross-functional teams to deliver production-ready data solutions.
Requirements
- 4+ years of experience in Data Engineering.
- 2+ years of hands-on Databricks experience.
- Strong expertise in Python, PySpark, and Advanced SQL.
- Hands-on experience with the Databricks Lakehouse Platform.
- Understanding of Unity Catalog, Delta Lake, Databricks Workflows/Jobs, Clusters, Notebooks, Repos, and Medallion Architecture.
- Experience integrating with REST APIs for data ingestion and data export.
- Strong knowledge of ETL/ELT development, batch processing, incremental loading, and data transformation.
Preferred qualifications
- Data Governance experience
- Experience with data modeling (Star Schema, Snowflake Schema, Fact & Dimension tables, SCD concepts)
- Understanding of data warehousing concepts and best practices
- Experience working with structured and semi-structured data (CSV, JSON, Parquet, Delta)
- Knowledge of partitioning, file optimization, Spark performance tuning, and query optimization
- Experience with Git and CI/CD best practices
- Experience with Auto Loader, Spark Declarative pipelines, Kafka, Airflow, or dbt
- Databricks certification
Benefits
- Competitive salary
- Strong insurance package
- Extensive learning and development resources
Skills
- Python
- PySpark
- SQL
- Databricks
- AWS
- REST API
- Unity Catalog
- Delta Lake
- ETL/ELT
- Git
- CI/CD
- Kafka
Show all 14 skills
- Airflow
- Dbt
