Back to jobs
New

Senior Databricks Data Engineer

College Station, Texas

About Us
Capital Farm Credit is the largest rural lending cooperative in Texas, serving 192 counties through nearly 70 credit offices. With over $12 billion in assets and more than 600 team members, we provide essential financial services to farmers, ranchers, rural homeowners, and agribusinesses. As part of the nationwide Farm Credit System, we are dedicated to supporting rural communities and agriculture.

Why Join Us?
We seek motivated individuals who share our core values: commitment, trust, value, and family-like respect. As a customer-owned cooperative, we align employee success with member success, offering competitive pay, growth opportunities, and a supportive environment.

Our Benefits:

  • Incentive Program: Company-wide, goals-based rewards.
  • Accrued Time Off: Earn 13 days of annual leave and 15 days of sick leave per year, plus enjoy 10-12 paid holidays annually.
  • Retirement: 401(k) with up to 9% employer contribution/match.
  • Health Coverage: Affordable medical, dental, and vision plans.
  • Parental Leave: 8 weeks of paid parental leave.
  • Life & Disability Insurance: Employer-paid coverage.
  • Education & Wellness: Tuition reimbursement and up to $400 for wellness expenses.

 

At Capital Farm Credit, you’ll find more than a job—you’ll find purpose.

 

EDUCATION AND EXPERIENCE

  • Bachelor’s degree in computer science, Information Systems, Engineering, Data Analytics, or a related field, or an equivalent combination of education and experience.
  • Five (5) or more years of hands-on data engineering experience using Azure, Databricks, or comparable cloud data platforms, including two (2) or more years serving as a senior engineer, technical lead, or mentor on data engineering teams or projects.
  • Demonstrated experience in Databricks, Spark, PySpark, Delta Lake, Unity Catalog, SQL, Python, notebooks, jobs, clusters, and Lakehouse architecture is required.
  • Experience with Azure Data Lake Storage (ADLS), Azure SQL, Key Vault, managed identities, ETL/ELT processes, dimensional modeling, and cloud security fundamentals is required.
  • Experience designing data solutions that support analytics, machine learning, artificial intelligence, governed enterprise reporting, and secure data consumption use cases is also required, including knowledge of AI/ML data preparation practices such as feature engineering, model-ready data design, data quality, lineage, and secure access to sensitive data.
  • Databricks Certified Data Engineer Associate certification or an equivalent current Databricks data engineering certification is required;
  • Databricks Certified Data Engineer Professional certification is preferred.

 

JOB SUMMARY

  • The Senior Databricks Data Engineer designs, develops, and supports scalable, secure, and governed cloud-based data solutions that enable enterprise analytics, business intelligence, regulatory reporting, artificial intelligence initiatives, and data products. This role serves as a senior technical resource for the enterprise data platform and provides day-to-day technical leadership to a team of data engineers, including work planning and assignment, design and code review, mentoring, and escalation support for complex production issues.
  • In addition to building, maintaining, and optimizing production data pipelines, this position leads the design and development of the organization’s enterprise data platform using Azure and Databricks. The position applies and establishes modern Lakehouse engineering practices and technologies, including Databricks, Apache Spark, PySpark, Delta Lake, Unity Catalog, Azure Data Lake Storage, SQL, and Python, with an emphasis on data governance, performance, automation, security, and production reliability. The Senior Databricks Data Engineer also drives data solution architecture by establishing and applying Lakehouse and Medallion architecture patterns, developing governed and AI-ready data assets, creating reusable engineering frameworks, and supporting secure and consistent data consumption across the organization.

ESSENTIAL FUNCTIONS

  • Provide day-to-day technical leadership and direction to a team of data engineers, including assigning and prioritizing work, estimating effort, tracking delivery, and removing technical obstacles in partnership with the manager.  
  • Mentor and develop engineers through design guidance, pair programming, knowledge sharing, and structured onboarding of new team members, contractors, and partners. 
  • Lead technical design sessions and establish engineering standards, patterns, and best practices for ingestion, transformation, governance, deployment, observability, and data consumption.   
  • Own technical quality across the team by conducting design and code reviews and enforcing testing, documentation, and release standards.
  • Partner with data, analytics, application, infrastructure, security, and AI teams to architect the overall enterprise data solution using Azure and Databricks.
  • Design and maintain batch and near-real-time pipelines using Databricks, ADF, Spark/PySpark, SQL, REST APIs, and ADLS.
  • Build governed Bronze, Silver, and Gold data layers using Delta Lake, Medallion Architecture, schema enforcement, incremental processing, and data quality controls.
  • Implement Unity Catalog, RBAC, lineage, classification, and secure data access/sharing across Databricks environments.
  • Deliver reusable data products supporting Power BI, enterprise and regulatory reporting, analytics, and AI/ML.
  • Design AI-ready data pipelines and curated datasets that support machine learning, generative AI, feature engineering, retrieval-augmented generation, vector search, and advanced analytics use cases. 
  • Support Databricks AI and ML capabilities, including MLflow, feature engineering patterns, model-ready datasets, model lifecycle support, model monitoring, and responsible AI governance controls.   
  • Manage orchestration, scheduling, monitoring, alerting, error handling, testing, and production support, and serve as a senior escalation point for complex production incidents and root cause analysis.   
  • Optimize Spark, SQL, clusters, storage, and compute for performance, reliability, and cost efficiency.
  • Implement solutions using Azure DevOps, Git, CI/CD, automated testing, and environment promotion.
  • Collaborate with analytics, application, security, infrastructure, and business teams to deliver trusted enterprise data. 
  • Contribute to reference architectures, reusable frameworks, platform standards, and implementation patterns for ingestion, transformation, governance, deployment, observability, and data consumption. 
  • Support platform planning activities by providing roadmap input, level-of-effort estimates, resource considerations, and assessments of technical risk.

REQUIRED SKILLS

  • Ability to lead, mentor, and develop technical staff, and to guide the work of other engineers without direct supervisory authority.
  • Ability to lead technical design discussions, evaluate alternatives, build consensus, and make sound architecture and implementation decisions.
  • Ability to plan, prioritize, and coordinate work across multiple engineers, concurrent projects, and competing deadlines.
  • Ability to communicate technical concepts clearly and collaborate with data analysts, BI developers, data scientists, IT teams, business stakeholders, and other technical partners.
  • Advanced proficiency with Databricks, Delta Lake, Unity Catalog, Medallion architecture, and Lakehouse design principles.
  • Advanced skills developing, troubleshooting, and optimizing Spark and PySpark workloads.
  • Strong programming and query-development skills using SQL and Python for data engineering, transformation, and automation.
  • Hands-on knowledge of data pipelines in Databricks, Azure Data Lake Storage (ADLS), Azure SQL, Key Vault, and managed identities.
  • Ability to design, build, maintain, and optimize scalable ETL/ELT pipelines and production data workflows.
  • Knowledge of dimensional modeling, data integration, data layers, and reusable enterprise data architecture patterns.
  • Understanding of data governance, lineage, access controls, data quality, sensitive data protection, and cloud security practices.
  • Ability to identify and resolve performance issues involving Spark workloads, SQL queries, clusters, and data pipelines.
  • Working knowledge of Azure DevOps, Git, source control, automated testing, CI/CD pipelines, release management, and environment promotion.
  • Knowledge of preparing governed, high-quality, model-ready datasets for machine learning and AI applications.
  • Ability to diagnose data pipeline failures, identify root causes, implement solutions, and maintain reliable production environments.
  • Ability to analyze complex technical and data issues and develop scalable, practical solutions.

DISCLAIMER

We are an Equal Employment/Affirmative Action employer. We do not discriminate in hiring on the basis of sex, race, color, religious creed, national origin, physical or mental disability, protected Veteran status, or any other characteristic protected by federal, state, or local law. If you need a reasonable accommodation for any part of the employment process, please contact us by email at careers@capitalfarmcredit.com and let us know the nature of your request and your contact information. Requests for accommodation will be considered on a case-by-case basis. Please note that only inquiries concerning a request for reasonable accommodation will be responded to from this e-mail address. For more information, view the EEO - Know Your Rights and Pay Transparency Statement.

Applicants should personally complete and submit their application materials. Submissions generated through automated tools or third-party mass application services may not be reviewed.

Equal Opportunity Statement
Capital Farm Credit is committed to creating a diverse and inclusive workplace. The position title and requirements may be adjusted based on the candidate's experience and qualifications. We welcome applicants of all backgrounds and do not discriminate based on race, color, gender, religion, national origin, disability, veteran status, or any other protected status. A full job description is available upon request. Candidates selected for hire will be required to complete a background check, including criminal history, education verification, and employment verification. A credit check will be required for roles that require NMLS registration.

Create a Job Alert

Interested in building your career at Capital Farm Credit? Get future opportunities sent straight to your email.

Apply for this job

*

indicates a required field

Resume/CV*

Accepted file types: pdf, doc, docx, txt, rtf

Cover Letter

Accepted file types: pdf, doc, docx, txt, rtf

Select...

Applicants may be required to provide proof of diploma, degree, transcripts, licenses, certifications, and registrations.

Select...
Select...
Select...
Select...

If yes, make sure to include the details on the employment section of your resume.

Select...
Select...

If yes, please explain. If you choose not to disclose, you may be required to provide additional information post-background check.

Select...
Select...
Select...
Select...
Select...
Select...
Select...

- I certify that all the information provided by me in connection with my application and resume is true and complete, and I understand that any misstatement, falsification, or omission of information will be grounds for refusal to hire or, if hired, termination.

- I understand that as a condition of employment, I will be required to provide legal proof or authorization to work in the United States. 

- I authorize Capital Farm Credit and/or their designees to perform background checks on me, to possibly include inquiries into criminal, education, or credit history in accordance with applicable status.

- I am willing and able to perform the essential functions of the job, with or without reasonable accommodation.

- I authorize investigation of all statements in this qualification record if I am considered for employment. The companies or persons named herein are authorized to give information regarding me whether or not such information is part of their records, and they are hereby released from all liability for issuing such information.

- I have read, understand, and agree to the above statements. 

Voluntary Self-Identification

For government reporting purposes, we ask candidates to respond to the below self-identification survey. Completion of the form is entirely voluntary. Whatever your decision, it will not be considered in the hiring process or thereafter. Any information that you do provide will be recorded and maintained in a confidential file.

As set forth in Capital Farm Credit’s Equal Employment Opportunity policy, we do not discriminate on the basis of any protected group status under any applicable law.

Select...
Select...
Race & Ethnicity Definitions

If you believe you belong to any of the categories of protected veterans listed below, please indicate by making the appropriate selection. As a government contractor subject to the Vietnam Era Veterans Readjustment Assistance Act (VEVRAA), we request this information in order to measure the effectiveness of the outreach and positive recruitment efforts we undertake pursuant to VEVRAA. Classification of protected categories is as follows:

A "disabled veteran" is one of the following: a veteran of the U.S. military, ground, naval or air service who is entitled to compensation (or who but for the receipt of military retired pay would be entitled to compensation) under laws administered by the Secretary of Veterans Affairs; or a person who was discharged or released from active duty because of a service-connected disability.

A "recently separated veteran" means any veteran during the three-year period beginning on the date of such veteran's discharge or release from active duty in the U.S. military, ground, naval, or air service.

An "active duty wartime or campaign badge veteran" means a veteran who served on active duty in the U.S. military, ground, naval or air service during a war, or in a campaign or expedition for which a campaign badge has been authorized under the laws administered by the Department of Defense.

An "Armed forces service medal veteran" means a veteran who, while serving on active duty in the U.S. military, ground, naval or air service, participated in a United States military operation for which an Armed Forces service medal was awarded pursuant to Executive Order 12985.

Select...

Voluntary Self-Identification of Disability

Form CC-305
Page 1 of 1
OMB Control Number 1250-0005
Expires 07/31/2029

Why are you being asked to complete this form?

We are a federal contractor or subcontractor. The law requires us to provide equal employment opportunity to qualified people with disabilities. We have a goal of having at least 7% of our workers as people with disabilities. The law says we must measure our progress towards this goal. To do this, we must ask applicants and employees if they have a disability or have ever had one. People can become disabled, so we need to ask this question at least every five years.

Completing this form is voluntary, and we hope that you will choose to do so. Your answer is confidential. No one who makes hiring decisions will see it. Your decision to complete the form and your answer will not harm you in any way. If you want to learn more about the law or this form, visit the U.S. Department of Labor’s Office of Federal Contract Compliance Programs (OFCCP) website at www.dol.gov/ofccp.

How do you know if you have a disability?

A disability is a condition that substantially limits one or more of your “major life activities.” If you have or have ever had such a condition, you are a person with a disability. Disabilities include, but are not limited to:

  • Alcohol or other substance use disorder (not currently using drugs illegally)
  • Autoimmune disorder, for example, lupus, fibromyalgia, rheumatoid arthritis, HIV/AIDS
  • Blind or low vision
  • Cancer (past or present)
  • Cardiovascular or heart disease
  • Celiac disease
  • Cerebral palsy
  • Deaf or serious difficulty hearing
  • Diabetes
  • Disfigurement, for example, disfigurement caused by burns, wounds, accidents, or congenital disorders
  • Epilepsy or other seizure disorder
  • Gastrointestinal disorders, for example, Crohn's Disease, irritable bowel syndrome
  • Intellectual or developmental disability
  • Mental health conditions, for example, depression, bipolar disorder, anxiety disorder, schizophrenia, PTSD
  • Missing limbs or partially missing limbs
  • Mobility impairment, benefiting from the use of a wheelchair, scooter, walker, leg brace(s) and/or other supports
  • Nervous system condition, for example, migraine headaches, Parkinson’s disease, multiple sclerosis (MS)
  • Neurodivergence, for example, attention-deficit/hyperactivity disorder (ADHD), autism spectrum disorder, dyslexia, dyspraxia, other learning disabilities
  • Partial or complete paralysis (any cause)
  • Pulmonary or respiratory conditions, for example, tuberculosis, asthma, emphysema
  • Short stature (dwarfism)
  • Traumatic brain injury
Select...

PUBLIC BURDEN STATEMENT: According to the Paperwork Reduction Act of 1995 no persons are required to respond to a collection of information unless such collection displays a valid OMB control number. This survey should take about 5 minutes to complete.