Back to jobs
New

Senior Data Engineer

London, Greater London, England, United Kingdom

Pantheon has been at the forefront of private markets investing for more than 40 years, earning a reputation for providing innovative solutions covering the full lifecycle of investments, from primary fund commitments to co-investments and secondary purchases, across private equity, real assets and private credit.

We have partnered with more than 650 clients, including institutional investors of all sizes as well as a growing number of private wealth advisers and investors, with approximately $65bn in discretionary assets under management (as of December 31, 2023).

Leveraging our specialized experience and global team of professionals across Europe, the Americas and Asia, we invest with purpose and lead with expertise to build secure financial futures.

For further details please visit www.pantheon.com

Purpose of the Position

Pantheon is undergoing a multi-year program to build out a new best in class Data Platform using cloud native technologies hosted in Azure. We require an experienced and passionate hands-on Senior Data Engineer to design and implement new data pipelines for adaptation to business and/or technology changes. This role will be integral to the success of this program and establishing Pantheon as a data-centric organisation. 

You will be working with a modern Azure tech stack and proven experience of ingesting and transforming data from a variety of internal and external systems is core to the role.

You will be part of a small and highly skilled team, and you will need to be passionate about providing best in class solutions to our global user base.

Key Responsibilities

Pantheon requires a highly capable, enthusiastic and experienced individual to join our team who will design and implement development activities to build out a new best in class Data Warehouse in cloud native Azure technology stack, powered by Databricks. The Data Engineer will design and execute development activities across projects ensuring that the software produced is written in a standard, maintainable and efficient manner.

To be successful in the role you will be expected to:

  • Design, build, and maintain scalable, secure, and high-performance data pipelines on Azure, primarily using Azure Databricks, Azure Data Factory, and Azure Functions.
  • Develop and optimise batch and streaming data processing solutions using PySpark and SQL to support analytics, reporting, and downstream data products.
  • Implement robust data transformation layers using dbt, ensuring well-structured, tested, and documented analytical models.
  • Collaborate closely with business analysts, QA teams, and business stakeholders to translate data requirements into reliable technical solutions.
  • Ensure data quality, reliability, and observability through automated testing, monitoring, logging, and alerting.
  • Lead on performance tuning, cost optimisation, and capacity planning across Databricks and associated Azure services.
  • Implement and maintain CI/CD pipelines using Azure DevOps, promoting best practices for version control, automated testing, and deployment.
  • Enforce data governance, security, and compliance standards, including access controls, data lineage, and auditability.
  • Contribute to architectural decisions and provide technical leadership, mentoring junior engineers and setting engineering standards.
  • Produce clear technical documentation and contribute to knowledge sharing across the data engineering function.

Knowledge & Experience Required

Essential Technical Skills

  • Azure Databricks (Spark, Delta Lake, performance tuning).
  • Python and PySpark for large-scale data processing.
  • SQL (advanced querying, optimisation, and data modelling).
  • Azure Data Factory (pipeline orchestration and integration).
  • dbt (analytics engineering best practices).
  • Azure DevOps (Git, CI/CD pipelines, release management).
  • Azure Functions / serverless data processing patterns.
  • Data modelling (star schemas, data vault, or lakehouse-aligned approaches).
  • Data quality, testing frameworks, and monitoring/observability.
  • Strong problem-solving ability and a pragmatic, engineering-led mindset.
  • Experience in Agile SW development environment
  • Excellent communication skills, with the ability to explain complex technical concepts to both technical and non-technical stakeholders.
  • Leadership and mentoring capability, with a focus on raising engineering standards and best practices.

Essential Experience

  • Significant commercial experience (typically 5+ years) in data engineering roles, with demonstrable experience designing and operating production-grade data platforms.
  • Strong hands-on experience with Azure Databricks, including cluster configuration, job orchestration, and performance optimisation.
  • Proven experience building data pipelines with Databricks and Azure Data Factory; integrating with Azure-native services (e.g. Data Lake Storage Gen2, Azure Functions).
  • Advanced experience with Python for data engineering, including PySpark for distributed data processing.
  • Strong SQL expertise, with experience designing and optimising complex analytical queries and data models.
  • Practical experience using dbt in a production environment, including model design, testing, documentation, and deployment.
  • Experience implementing CI/CD pipelines using Azure DevOps or equivalent tooling.
  • Solid understanding of data warehousing and lakehouse architectures, including dimensional modelling and modern analytics patterns.
  • Experience working in agile delivery environments and collaborating with cross-functional teams.
  • Exposure to cloud security, data governance, and compliance concepts within Azure.

Desired Experience

  • Power BI and DAX
  • Business Objects Reporting

 

#LI-SB1

This job description is not to be construed as an exhaustive statement of duties, responsibilities, or requirements. You may be required to perform other job-related duties as reasonably requested by your manager.

Pantheon is an Equal Opportunities employer, we are committed to building a diverse and inclusive workforce so if you're excited about this role but your past experience doesn't perfectly align we'd still encourage you to apply.

 

 

Create a Job Alert

Interested in building your career at Pantheon Ventures Careers? Get future opportunities sent straight to your email.

Apply for this job

*

indicates a required field

Resume/CV*

Accepted file types: pdf, doc, docx, txt, rtf

Education

Select...
Select...
Select...

Including last and expected bonus, and any other important to mention benefits

Select...

Voluntary Self-Identification

For government reporting purposes, we ask candidates to respond to the below self-identification survey. Completion of the form is entirely voluntary. Whatever your decision, it will not be considered in the hiring process or thereafter. Any information that you do provide will be recorded and maintained in a confidential file.

As set forth in Pantheon Ventures Careers’s Equal Employment Opportunity policy, we do not discriminate on the basis of any protected group status under any applicable law.

Select...
Select...
Race & Ethnicity Definitions

If you believe you belong to any of the categories of protected veterans listed below, please indicate by making the appropriate selection. As a government contractor subject to the Vietnam Era Veterans Readjustment Assistance Act (VEVRAA), we request this information in order to measure the effectiveness of the outreach and positive recruitment efforts we undertake pursuant to VEVRAA. Classification of protected categories is as follows:

A "disabled veteran" is one of the following: a veteran of the U.S. military, ground, naval or air service who is entitled to compensation (or who but for the receipt of military retired pay would be entitled to compensation) under laws administered by the Secretary of Veterans Affairs; or a person who was discharged or released from active duty because of a service-connected disability.

A "recently separated veteran" means any veteran during the three-year period beginning on the date of such veteran's discharge or release from active duty in the U.S. military, ground, naval, or air service.

An "active duty wartime or campaign badge veteran" means a veteran who served on active duty in the U.S. military, ground, naval or air service during a war, or in a campaign or expedition for which a campaign badge has been authorized under the laws administered by the Department of Defense.

An "Armed forces service medal veteran" means a veteran who, while serving on active duty in the U.S. military, ground, naval or air service, participated in a United States military operation for which an Armed Forces service medal was awarded pursuant to Executive Order 12985.

Select...

Voluntary Self-Identification of Disability

Form CC-305
Page 1 of 1
OMB Control Number 1250-0005
Expires 04/30/2026

Why are you being asked to complete this form?

We are a federal contractor or subcontractor. The law requires us to provide equal employment opportunity to qualified people with disabilities. We have a goal of having at least 7% of our workers as people with disabilities. The law says we must measure our progress towards this goal. To do this, we must ask applicants and employees if they have a disability or have ever had one. People can become disabled, so we need to ask this question at least every five years.

Completing this form is voluntary, and we hope that you will choose to do so. Your answer is confidential. No one who makes hiring decisions will see it. Your decision to complete the form and your answer will not harm you in any way. If you want to learn more about the law or this form, visit the U.S. Department of Labor’s Office of Federal Contract Compliance Programs (OFCCP) website at www.dol.gov/ofccp.

How do you know if you have a disability?

A disability is a condition that substantially limits one or more of your “major life activities.” If you have or have ever had such a condition, you are a person with a disability. Disabilities include, but are not limited to:

  • Alcohol or other substance use disorder (not currently using drugs illegally)
  • Autoimmune disorder, for example, lupus, fibromyalgia, rheumatoid arthritis, HIV/AIDS
  • Blind or low vision
  • Cancer (past or present)
  • Cardiovascular or heart disease
  • Celiac disease
  • Cerebral palsy
  • Deaf or serious difficulty hearing
  • Diabetes
  • Disfigurement, for example, disfigurement caused by burns, wounds, accidents, or congenital disorders
  • Epilepsy or other seizure disorder
  • Gastrointestinal disorders, for example, Crohn's Disease, irritable bowel syndrome
  • Intellectual or developmental disability
  • Mental health conditions, for example, depression, bipolar disorder, anxiety disorder, schizophrenia, PTSD
  • Missing limbs or partially missing limbs
  • Mobility impairment, benefiting from the use of a wheelchair, scooter, walker, leg brace(s) and/or other supports
  • Nervous system condition, for example, migraine headaches, Parkinson’s disease, multiple sclerosis (MS)
  • Neurodivergence, for example, attention-deficit/hyperactivity disorder (ADHD), autism spectrum disorder, dyslexia, dyspraxia, other learning disabilities
  • Partial or complete paralysis (any cause)
  • Pulmonary or respiratory conditions, for example, tuberculosis, asthma, emphysema
  • Short stature (dwarfism)
  • Traumatic brain injury
Select...

PUBLIC BURDEN STATEMENT: According to the Paperwork Reduction Act of 1995 no persons are required to respond to a collection of information unless such collection displays a valid OMB control number. This survey should take about 5 minutes to complete.