HPC Engineer - Biohub SF

The Chan Zuckerberg Biohub San Francisco (CZ Biohub SF) (https://www.czbiohub.org/sf/) is an independent nonprofit research institute that brings together three powerhouse universities - Stanford, UC Berkeley, and UC San Francisco - into a single collaborative technology and discovery engine. CZ Biohub SF itself supports some of the brightest, boldest engineers, data scientists, and biomedical researchers to investigate the fundamental mechanisms underlying disease and develop new technologies that will lead to actionable diagnostics and effective therapies. We are guided by our values of scholarly excellence; disruptive innovation; hands-on engineering/hacking/building; partnership and collaboration; open communication and respect; inclusiveness; and opportunity for all.

Our Vision

• We pursue large scientific challenges that cannot be pursued in conventional environments
• We enable individual investigators to pursue their riskiest and most innovative ideas
• The technologies developed at CZ Biohub San Francisco facilitate research by scientists and clinicians at our home institutions and beyond

Diversity of thought, ideas, and perspectives are at the heart of CZ Biohub Network and enable disruptive innovation and scholarly excellence. We are committed to cultivating an organization where all colleagues feel inspired and know their work makes an important contribution.

The Opportunity

The Chan Zuckerberg Biohub San Francisco has an immediate opening for an experienced Principal High Performance Computing (HPC) Engineer who will work to develop, support and optimize the CZ Biohub HPC systems. We are seeking a talented engineer to strengthen and expand the scientific computational capacity to further the Biohub’s expanding global scientific leadership.  In this position, you will support day-to-day operations of the HPC systems including systems administration, monitoring and parallel filesystem performance up to and including network components.  You will manage cluster-level services via the SLURM scheduler as well as user facing services such as Open OnDemand and NoMachine.  The ideal candidate will also install, configure and optimize applications and provide user support. This position will also work closely with many different science teams simultaneously to translate experimental descriptions into software and hardware requirements and across all phases of the scientific lifecycle, including data ingest, analysis, management and storage, computation, authentication, tool development and many other computational needs expressed by scientific projects. 

You will

• Manage cluster-level services via the SLURM scheduler as well as user facing services such as Open OnDemand and NoMachine
• Install, configure and optimize applications and provide user support
• Work closely with many different science teams simultaneously to translate experimental descriptions into software and hardware requirements and across all phases of the scientific lifecycle, including data ingest, analysis, management and storage, computation, authentication, tool development and many other computational needs expressed by scientific projects

You have

Essential –

• Bachelor’s Degree in Computer Science, Mathematics, Systems Engineering or a related field or equivalent training/experience also acceptable
• A minimum of 7 years of experience with progressively increasing responsibility in HPC computing environments or complex Linux environments
• Experience building on-prem HPC infrastructure and capacity planning
• Experience and expertise working on complex issues where analysis of situations or data requires an in-depth evaluation of variable factors
• Experience supporting scientific facilities, and prior knowledge of scientific user needs, program management, data management planning or lab-bench IT needs
• Experience with HPC and cloud computing environments
• Ability to interact with a variety of technical and scientific personnel with varied academic backgrounds
• Strong written and verbal communication skills to present and disseminate scientific software developments at group meetings
• Demonstrated ability to reason clearly about load, latency, bandwidth, performance, reliability, and cost and make sound engineering decisions balancing them
• Demonstrated ability to quickly and creatively implement novel solutions and ideas

Technical requirements -

• Proven ability to analyze, troubleshoot, and resolve complex problems that arise in the HPC production storage hardware, software systems, storage networks and systems
• Configuring and administering parallel, network attached storage (Lustre, NFS, ESS, Ceph) and storage subsystems (e.g. IBM, NetApp, DataDirect Network, LSI, etc.)
• Installing, configuring, and maintaining job management tools (such as SLURM, Moab, TORQUE, PBS, etc.)
• Red Hat Enterprise Linux, CentOS, or derivatives and Linux services and technologies like dnsmasq, systemd, LDAP, PAM, sssd, OpenSSH, cgroups
• Scripting languages (including Bash, Python, or Perl)
• Virtualization (ESXi or KVM/libvirt), containerization (Docker or Singularity), configuration management and automation (tools like xCAT, Puppet, kickstart) and orchestration (Kubernetes, docker-compose, CloudFormation, Terraform.)
• High performance networking technologies (Ethernet and Infiniband) and hardware (Mellanox and Juniper)
• Configuring, installing, tuning and maintaining scientific application software
• Familiarity with source control tools (Git or SVN)

Nice to have -

• Understand and translate researchers' scientific challenges into computational solutions
• Scientific background, research experience, and/or experience in a University or a research setting

Compensation

The San Francisco, CA base pay range for this role is Senior Staff Engineer = $192,000.00 - $264,000.00, and Principal Engineer = $216,000.00 - $297,000.00 and will be adjusted based on candidate’s geographic location. New hires are typically hired into the lower portion of the range, enabling employee growth in the range over time. Actual placement in range is based on job-related skills and experience, as evaluated throughout the interview process.

What We Provide

• Resources to disrupt and innovate at the frontiers of our knowledge of biology and disease
• A collegial and collaborative environment consisting of diverse expertise
• Access to collaborators, resources and facilities at our three partner universities (Stanford University, UC Berkeley, and UC San Francisco) and at partner organizations in the SF Bay Area and beyond
• Competitive compensation and benefits commensurate with the experience

If you’re interested in a role but your previous experience doesn’t perfectly align with each qualification in the job description, we still encourage you to apply as you may be the perfect fit for this or another role.

Benefits

We offer a robust benefits program that enables the important work Biohubbers do everyday. Our benefits include healthcare coverage, life and disability insurance, commuter subsidies, family planning services with fertility care, childcare stipend, 401(k) match, flexible time off and a generous parental leave policy. In addition, we honor our commitment to career development and our value of scholarly excellence through regular onsite opportunities to learn from the world's leading scientists.

The CZ Biohub Network is an equal opportunity employer committed to diversity of thought, ideas and perspectives. We are committed to cultivating an inclusive organization where all Biohubbers feel inspired and know their work makes an important contribution. Therefore, we provide employment opportunities without regard to age, race, color, ancestry, national origin, religion, disability, sex, gender identity or expression, sexual orientation, or any other protected status in accordance with applicable law.

Pursuant to the California Fair Chance Act, we will consider for employment qualified applicants with arrest and conviction records.

Headhunters and recruitment agencies may not submit resumes/CVs through this website or directly to managers. The CZ Biohub Network does not accept unsolicited headhunter and agency resumes. The CZ Biohub Network will not pay fees to any third-party agency or company that does not have a signed agreement with the CZ Biohub Network.