Senior Data Engineer & Technical Delivery Lead (Python / Databricks) | London, UK

Senior Data Engineer & Technical Delivery Lead (Python / Databricks)

Location: London (Hybrid – 2-3 days onsite)

Contract: Initial 6 months

We're currently supporting a major Financial Services organisation that's expanding its Data Engineering function and is looking to hire multiple contractors across Senior Data Engineer and Technical Delivery Lead level.

Whether you're a highly technical Senior Data Engineer looking to remain hands-on, or an experienced Technical Lead who enjoys combining delivery leadership with coding, we'd be keen to speak with you.

You'll be joining a large-scale cloud data transformation programme, helping design, build and optimise enterprise data platforms using Python, Databricks and Spark within a modern Lakehouse environment.

Key Responsibilities

• Design, develop and maintain scalable data pipelines using Python and Databricks.
• Build, optimise and support enterprise ETL/ELT workflows using Apache Spark and Delta Lake.
• Design and develop robust data models and Lakehouse architectures.
• Implement and manage data workflows within the Databricks ecosystem.
• Ensure high levels of data quality, governance and pipeline reliability.
• Optimise performance across large-scale distributed data processing platforms.
• Collaborate with architects, product owners, analysts and engineering teams to deliver high-quality data solutions.
• Implement monitoring, logging and alerting across critical data platforms.
• Drive engineering best practice through code reviews, mentoring and technical collaboration.
• For Technical Delivery Lead opportunities, provide technical leadership, architectural oversight and end-to-end delivery of complex data engineering initiatives.

Required Skills & Experience

• Strong commercial experience developing data engineering solutions using Python .
• Extensive hands-on experience with Databricks , including Workflows, Notebooks and Delta Lake.
• Strong experience with Apache Spark / PySpark .
• Proven experience building scalable ETL/ELT pipelines.
• Strong SQL and data modelling skills.
• Experience designing modern Lakehouse architectures.
• Experience working with cloud platforms including AWS, Azure or GCP.
• Experience with modern data warehousing technologies such as Snowflake, Redshift or BigQuery.
• Experience with orchestration tools such as Airflow or Databricks Workflows.
• Previous experience working within Agile delivery environments.

Desirable Experience

• Databricks certifications.
• Kafka or Structured Streaming.
• CI/CD and DevOps practices.
• Docker and Kubernetes.
• Exposure to Machine Learning pipelines.
• Experience within Financial Services or other highly regulated environments.

About You

You'll be an experienced Data Engineer or Technical Lead with a passion for building modern, scalable data platforms. Comfortable working in a collaborative Agile environment, you'll enjoy solving complex technical problems and delivering high-quality engineering solutions.

If you're applying at Technical Delivery Lead level, you'll also bring strong leadership, stakeholder management and mentoring experience whilst remaining hands-on technically.