eNGINE builds Technical Teams. We are a Solutions and Placement firm shaped by decades of interaction with Technical professionals. Our inspiration is continuous learning and engagement with the markets we serve, the talent we represent, and the teams we build.
Our Consulting Workforce is encouraged to enjoy career fulfillment in the form of challenging projects, schedule flexibility, and paid training/certifications. Successful outcomes start and finish with eNGINE.eNGINE is seeking a Senior Data Engineer to build, operate, and own the enterprise data warehouse (EDW) and core data pipelines on our Databricks platform.
As a senior practitioner on the Data Engineering team, you'll set the technical bar for pipeline quality, documentation, and reliability, while mentoring engineers as the team grows.
What You'll Do:
- Design, build, and operate EDW pipelines and data models across the Databricks medallion architecture (Bronze, Silver, Gold), including Silver conformance logic and Gold harmonization/survivorship rules
- Take named ownership of assigned EDW domains as production systems — data model integrity, pipeline reliability, ingestion SLAs, and incident response
- Lead structured documentation efforts for assigned domains, capturing architecture, transformation logic, and feed SQLDrive data model simplification, consolidating legacy transformation layers into the target architecture with validated output parity
- Develop and maintain production Delta Lake pipelines, evolving ingestion from batch/watermark patterns to event-driven architectures using Zerobus Ingestion and Spark Declarative Pipelines for high-SLA source systems
- Implement row-level reconciliation, validation checkpoints, and quality gates across ingestion, transformation, and delivery — operationalizing governance-defined data quality standards in partnership with QABuild Gold-layer tables and promotion logic supporting Unity Catalog metric views, lineage capture, and access controls, in partnership with Analytics Engineering and Data Governance
- Optimize pipeline performance, latency, and compute cost across all layers of the platform
- Document architecture decisions, data models, pipeline logic, and operational runbooks to team standards, ensuring no critical capability depends on a single point of failure
- Operate pipeline observability tooling — monitor anomaly detection, triage data reliability incidents, and drive root-cause remediation for assigned domains
- Implement data contract validation (ODCS or equivalent) for pipelines supporting external partner feeds, ensuring contract failures halt delivery before reaching consumers
- Mentor Data Engineers on Databricks development patterns, SQL/PySpark craft, and documentation discipline; review pull requests in Azure DevOps & Git to maintain code quality
- Partner with Data Governance on lineage and metadata standards, with QA on Tier 1/Tier 2 pipeline test suites, and with Data Services on feed engineering
- Work from structured requirements and acceptance criteria via the Informatics intake process, flagging requirements gaps before build work begins
Required Skills
- 6+ years of progressive data engineering experience, including production ownership of an enterprise data warehouse or large-scale transformation pipelines with defined SLAsDeep, hands-on expertise with Databricks in production: Delta Lake, medallion architecture, Unity Catalog, and pipeline performance optimization
- Strong data warehousing depth — dimensional and harmonized data modeling, conformance and survivorship logic, and running a warehouse as a production system
- Advanced SQL and strong PySpark/Python skills, sufficient to build, review, and optimize complex transformation logic independently
- Hands-on experience with Azure data services: Azure Databricks, Azure Data Lake Storage, Azure Data Factory, and CI/CD in Azure DevOpsProven ability to absorb complex, under-documented systems through structured knowledge transfer, and turn that knowledge into durable, transferable documentation
- Strong communication skills — comfortable working directly with QA, Governance, Informatics, and business teams without an intermediary
Preferred Skills
- Experience in healthcare, specialty pharmacy, or another regulated industry, with exposure to clinical or operational data
- Production experience with event-driven ingestion (Azure Event Hubs, Kafka, or equivalent) consumed via Structured Streaming
- Familiarity with data contract standards (ODCS or equivalent) and observability platforms like Monte Carlo
- Familiarity with enterprise data catalog tools (Atlan, Collibra, or equivalent) and designing pipelines as governed, catalogued assets
- Experience leading through influence — mentoring engineers and driving adoption without formal management authority
- Bachelor's degree in Computer Science, Data Engineering, Information Systems, or related field, or equivalent experience
- No C2C, relocation, referral, or sponsorship candidates for this role.
- For finer details on how eNGINE can impact your career, apply today!
