Senior Data Engineer
Boston Hybrid · 3-5 years experience · Full-time · Hybrid · Posted Oct 12, 2026
At a glance
- Experience
- 3-5 years experience
- Employment
- Full-time
- Workplace
- Hybrid
- Location
- Boston Hybrid
About the role
- The Senior Data Engineer is a member of the Data Platform team, reporting to the Director of Machine Learning Engineering and Data.
- The Data Platform team is responsible for executing the data strategy for the organization, ensuring high-quality external data is ingested into the Lakehouse and realized in powerful insights for internal and external customers.
Responsibilities
- Contribute to the vision, architecture, and roadmap for the CodaMetrix data platform, ensuring scalability, reliability, regulatory alignment, and operational excellence.
- Participate in key technology decisions, disaster recovery design, architecture reviews, and data engineering standards across teams.
- Design, build, and maintain scalable streaming and batch data platforms Databricks using PySpark, Unity Catalog, Delta Lake, and Structured Streaming.
- Own Terraform-based infrastructure across environments, including jobs, catalogs, schemas, permissions, compute policies, volumes, and external locations.
- Build Jenkins CI/CD workflows for automated testing, tagging, and deployments, while optimizing training pipelines, materialized views, retention policies, and production performance.
- Implement and evolve least-privilege access controls across Databricks using Unity Catalog grants, YAML-driven group policies, and schema-level restrictions.
- Ensure HIPAA and SOC 2 compliance through PHI masking, audit logging, environment-level data segmentation, user provisioning, compute policies, and cost attribution.
- Drive platform cost reduction through compute policy tuning, serverless optimization, reserved pools, materialized view improvements, and remediation of underutilized resources.
- Monitor Databricks/AWS spend (using tools like CloudZero and AWS Cost Explorer), own cost attribution and budgeting, resolve production incidents, and maintain runbooks to ensure platform SLAs for uptime and performance.
- Enable ML, BI, Analytics, DevOps, and customer success and implementation teams with training data pipelines, model-ready datasets, feature store architecture, optimized views, dashboards, Tableau refreshes, infrastructure changes, and tenant onboarding.
Requirements
- A degree in Computer Science or a related field (Bachelor's, Master's, or Ph.D.), or an equivalent combination of education and demonstrable professional experience
- 3-5+ years of data engineering experience with progressive responsibility
- 3-5+ years hands-on experience with the Databricks platform (Unity Catalog, Delta Lake, Structured Streaming, Spark SQL, cluster management, platform administration)
- Expert proficiency in PySpark and Python; working knowledge of Scala
- Expert experience with Terraform for infrastructure-as-code (state management, modular structures, multi-environment deployments, YAML-driven configuration)
- Expert experience with Apache Kafka / AWS MSK (streaming ingestion, SASL/IAM auth, topic management, cluster migrations)
- Deep understanding of medallion/lakehouse architecture patterns (bronze/silver/gold, SCD2, materialized views, slowly changing dimensions)
- Proven track record building and maintaining CI/CD pipelines (Jenkins, GitHub Actions) for data platform deployments
- Expert SQL skills — complex CTEs, window functions, performance optimization of large-scale queries across petabyte-scale datasets
- Strong AWS experience (S3, IAM, Secrets Manager, VPC/Private Link, cross-region replication)
Preferred qualifications
- Prior experience working with healthcare data, such as medical claims, electronic health records (EHR), or billing code systems (ICD-10, CPT)
- Familiarity with healthcare data standards like HL7 or FHIR is highly desirable
Benefits
- Health Insurance: We cover 80% of the cost of medical and dental insurance and offer vision insurance
- Retirement: We offer a 401(k) plan that eligible employees can contribute to one month after their first day
- Flexibility: We have a generous Paid Time Off policy, which is managed but not limited, so you can take the time you need to relax and rejuvenate
- Development: We provide annual performance evaluations and prioritize working with employees on what their individual growth looks like
- Recognition: We recognize the outstanding achievements of our team through annual company awards where employees have the opportunity to nominate their peers
- Office Location: A modern open plan workspace located in the bustling Back Bay neighborhood of Boston
- Additional Employer Paid Benefits: We offer employer-paid life insurance and short-term and long-term disability insurance
Workplace
- Boston Hybrid
Skills
- Databricks
- PySpark
- Python
- Scala
- Unity Catalog
- Delta Lake
- Structured Streaming
- Spark SQL
- Terraform
- Apache Kafka
- AWS MSK
- Jenkins
Show all 15 skills
- GitHub Actions
- SQL
- AWS
