Senior Data Engineer - AI & Analytics Platform - Hybrid

What you do

The Senior Data Engineer – AI & Analytics Platform will be a key role in driving Allianz Life’s AI and Analytics capabilities on the Azure/Databricks platform. You’ll leverage your hands-on data engineering and Databricks expertise to build and maintain data pipelines and automation that power advanced analytics and AI solutions.

You’ll have a hand in defining and executing data ingestion and data architecture with a focus on operational excellence, security, and performance. Collaboration will be key, as you partner with IT, data science, analytics engineering, and vendor teams to move and manage data on a robust platform that accelerates innovation and business value.

Data Engineering & Operations (60%)

• Develop, deploy, and maintain platform components, including: data connections, data pipelines, Databricks Lakeflow jobs, related GitHub Actions and Databricks Declarative Automation Bundle (DABs) pipelines, and various monitoring frameworks.
• Create and maintain data engineering templated project repositories enabled and simplified for the internal platform team with custom CLI. Create/manage Databricks DAB Infrastructure as Code repositories as needed.
• Ensure high availability, performance, and observability of data pipelines and data quality & cost monitoring through proactive monitoring and incident response.
• Maintain team documentation for the internal platform and for external business users of the platform (i.e. developers).

Collaboration & Governance (30%)

• Mentor junior data engineers within the internal team, including reviewing PRs and ensuring code quality are standardized and rigorously reviewed on a platform level.
• Partner with other members of the internal team, IT, data science, and analytics engineering teams to support end-to-end solution delivery, including building and promoting data pipelines, provisioning the related Databricks DAB infrastructure for data ingestion & transformation, and supporting DevOps pipelines (GitHub Actions) and processes.
• Provide support and guidance across the SDLC, including CI/CD, release management, and environment promotion.
• Provide ad hoc support for application deployment for Databricks Apps space and Azure App Services. Kubernetes application experience preferred.

Innovation & Enablement (10%)

• Help evaluate emerging Azure and Databricks services and AI/ML capabilities to drive innovation and continuous improvement.
• Contribute to data engineering platform governance, including data ethics, compliance, and security best practices.
• Mentor technical teams and promote architectural standards and reusable patterns.
• Utilization of artificial intelligence tools and resources (e.g. generative AI)

What you bring

• 6+ years of experience in a combination of Data Engineering, DevOps, and App Development roles.
• Experience configuring and maintaining Databricks Workspaces, ACLs, and other Workspace artifacts
• Experience working with Databricks (Lakehouse architecture, Spark, Unity Catalog, Delta tables).
• Experience with DevOps, and SDLC practices in a regulated enterprise environment.
• Experience creating GitHub Actions Workflows for code promotion across environments.
• Experience creating data ingestion and data transformation pipelines in Pyspark, preferably within Databricks.
• You must be legally authorized to work in the U.S. without requiring immigration sponsorship now or in the future. This includes holders of H-1B, H4, L-1, L-2, TN, OPT, CPT, and other nonimmigrant visas.

100603 | Data & AI | Professional | Non-Executive | Allianz US Life | Full-Time | Permanent