Reference Data Engineer
New York · USD 135,000-170,000 · 4-7 years experience · Posted Sep 28, 2026
At a glance
- Experience
- 4-7 years experience
- Location
- New York
- Salary
- USD 135,000-170,000 · Base salary
About the role
- The Operations Systems team is seeking an Assistant Vice President (AVP), Software Engineering who blends senior individual contribution with emerging technical leadership.
- This position involves designing and delivering robust backend services and data pipelines while guiding the technical direction of the team.
Responsibilities
- Design, build, and own scalable backend services and data pipelines primarily in Python.
- Develop and optimize large-scale data processing workflows using Apache Spark for batch and near-real-time use cases.
- Lead data modeling and schema design decisions that support reliable analytics and downstream consumption.
- Architect and maintain robust ETL/ELT workflows with strong data quality, lineage, and observability.
- Design and implement data distribution patterns to serve data efficiently across internal systems and consumers.
- Provision and manage cloud infrastructure using Terraform and infrastructure-as-code best practices.
- Own delivery of significant workstreams end-to-end, from technical design through production and operations.
- Drive engineering quality through design reviews, code reviews, and clear technical standards.
- Mentor and support the growth of analysts and junior engineers on the team.
- Partner with product, data, and platform leaders to shape roadmap and translate requirements into scalable solutions.
Requirements
- Bachelor's degree in Computer Science, Engineering, or a related field, or equivalent practical experience.
- 4–7 years of professional software engineering experience with a strong backend focus.
- Expert-level proficiency in Python for building production services and data pipelines.
- Strong hands-on experience with Apache Spark for distributed data processing at scale.
- Proficiency with Terraform for infrastructure-as-code and cloud resource management.
- Deep data engineering fundamentals, including data modeling, ETL/ELT design, and data distribution patterns.
- Solid command of relational and non-relational data stores and SQL.
- Experience with version control (Git), CI/CD pipelines, and modern software development practices.
- Demonstrated ability to own complex workstreams and mentor other engineers.
- Strong communication skills and the ability to collaborate across technical and business teams.
Preferred qualifications
- Experience with cloud platforms (AWS, Azure, or GCP) and their data services.
- Familiarity with workflow orchestration tools (e.g., Airflow, Dagster).
- Experience with streaming technologies (e.g., Kafka, Spark Structured Streaming).
- Exposure to containerization and orchestration (Docker, Kubernetes).
- Prior experience in financial services or another data-intensive, regulated industry.
- Experience with Palantir Foundry or similar data integration and analytics platforms.
Benefits
- Competitive compensation
- Comprehensive benefits
- Significant opportunities for professional growth, technical leadership, and impact
Skills
- Python
- Spark
- Terraform
- SQL
- Git
- CI/CD
- AWS
- Azure
- GCP
- Airflow
- Dagster
- Kafka
Show all 15 skills
- Docker
- Kubernetes
- Palantir Foundry
