Sr. Data Engineer(PA locals only )

Job Title: Sr. Data Engineer

Location: Hybrid-2 days a week onsite in downtown Philadelphia, PA- locals only within one drive distance

Duration: 6 month contract to hire

Visa: only

Exp Level: 10+ years

Must have

These roles are focused on building real time streaming data solutions that enable data-driven decisions on live devices and telemetry data.

While the Senior Data Engineer will support batch data processes, the primary focus is on real-time streaming use cases, data services for APIs and microservices, and pipelines that feed both operational applications and ML/AI model development in a medical device IoT environment.

Role Description

As a Senior Data Engineer, you will play a key role in driving the design and architecture of data pipelines that consume real-time streaming data from connected medical devices. You will collaborate cross-functionally with business analysts, software engineers, machine learning engineers, and business users to implement technical approaches and infrastructure that support data consumption across the organization.

This is a great opportunity for someone who is passionate about data engineering, thrives in environments where you can take ownership and recommend solutions, and is a self-starter who is open to learning new tools as needed.

Qualifications

Required Experience

• 8+ years of experience designing, building, and operating big data and real-time streaming pipelines across cloud and on-premises environments
• 5+ years applying DevOps and CI/CD practices to data and analytics workloads
• Production experience building data services, APIs, or microservices for downstream data consumption

Required Technical Skills

• Strong, hands-on experience designing and building production-grade data pipelines on Databricks
• Demonstrated experience with Spark optimization and tuning in Databricks, including performance analysis, partitioning strategies, caching, shuffle optimization, and cost-aware pipeline design
• Strong experience with Azure cloud services for data engineering and streaming workloads
• Experience building data services, APIs, or microservices for data consumption.
• Data quality, validation, and privacy-aware handling for regulated or sensitive data

Preferred Technical Skills

• Azure Event Hubs and Azure Stream Analytics for IoT and telemetry ingestion and stream processing
• Additional Azure data and integration services supporting batch and streaming architectures
• Collaboration with ML teams: feature pipelines, training data, or streaming data for model development

Key Responsibilities

• Design and build batch and streaming data pipelines on Azure and Databricks, with primary focus on real-time IoT and telemetry use cases within a Medallion architecture.
• Develop ETL/ELT workflows to ingest, transform, and validate large volumes of structured and unstructured data.
• Build and maintain data services, APIs, and microservices for application, analytics, and ML/AI teams.
• Implement real-time streaming solutions using Azure Event Hubs, Azure Stream Analytics, and related Azure integration patterns, with cost-effective throughput, partitioning, and downstream delivery to Databricks.
• Optimize production Databricks pipelines using PySpark, Spark SQL, and Delta Lake, including Spark tuning for performance, reliability, and cost.
• Troubleshoot and resolve complex pipeline issues across Databricks, Azure, and on-premises systems, including root cause analysis and corrective action.
• Partner with data analysts, software engineers, ML engineers, and business stakeholders to translate requirements into technical designs and delivery priorities.
• Apply data quality, validation, and privacy-first practices, and deliver reliable pipelines through software engineering standards, documentation, testing, and CI/CD.

Outcomes

• Onboard to the Azure Databricks environment and contribute to troubleshooting, stabilization, and optimization of existing batch and streaming data pipelines.
• Stand up Azure streaming ingestion for telemetry data and deliver production-ready pipelines integrated with Databricks to support API, ML, and downstream analytics consumption within the Medallion architecture.
• Design and deliver data services or consumption patterns that enable business and ML teams to access near-real-time telemetry data reliably, securely, and on a scale.

Work Hours and Travel Requirements

Candidates must be open to assisting in troubleshooting and analysis in the event of off-hours production problems, as needed. The IT Team works in a hybrid environment that requires a minimum of two days per week in their downtown Philadelphia office.