Senior Staff Engineer - Observability

Salary Range: PLN 303,800 - 411,000

Subject to alignment to the responsibilities and duties of the role.

About Graphcore

At Graphcore, we’re building the future of AI compute.We’re a team of semiconductor, software and AI experts, with deep experience in creating the complete AI compute stack - from silicon and software to infrastructure at datacenter scale.As part of the SoftBank Group, backed by significant long-term investment, we are delivering key technology into the fast-growing SoftBank AI ecosystem.To meet the vast and exciting AI opportunity, Graphcore is expanding its teams around the world.We are bringing together the brightest minds to solve the toughest problems, in a place where everyone has the opportunity to make an impact on the company, our products and the future of artificial intelligence.

Job Summary 

As a Senior Staff Engineer you will contribute to the architecture, implementation and deployment of highly scalable management solutions for AI infrastructure built using our next-generation products.  Covering monitoring, observability, control, and data centre infrastructure management, you will work with software, cloud and customer-facing teams, to establish first-hand knowledge of these solutions, creating proof-of-concepts, reference designs and integrations with third-party tooling. 

Your team will work with product, architecture and other delivery teams to ensure that functionally complete, simple to deploy, and easy to use solutions are deployed internally to support engineering efforts and supply reference designs to our customers. 

Responsibilities and Duties  

• Contribute to all phases of overall product development, from product definition, architecture, and design, through implementation, debugging, testing and early customer support.
• Contribute to the architecture of the solution, implement multi-component integrations based on Graphcore and third-party technology stacks, from data ingestion to decision making, ensuring seamless management, monitoring and UI. 
• Create reference designs including documentation, configuration files, scripts, source code, etc.
• Deploy solution(s) internally for use by our engineering teams to aid debugging, performance analysis, benchmarking, test/QA, etc. at all scales.
• Ensure solutions are properly tested by working with the development and QA teams to enhance unit testing and ensure a comprehensive test plan is in place.
• Contribute to the design and implementation of fault-remediation solutions at scale.
• Mentor and guide junior engineers, fostering a culture of continuous learning and improvement. 

Skills and Experience 

• BSc or MSc degree in Computer Engineering, Computer Science, or related degree or equivalent experience.
• Demonstrated success in architecting and implementing scalable, performant, reliable cluster management systems including telemetry collection & analysis engines.
• Understanding of platform architecture, including CPU, AI accelerators, and other server component interactions.
• Expertise of in-band and out-of-band management architectures and working knowledge of associated tools.
• Familiarity with data centre management standards (BMC Redfish, IPMI, Open Compute, DMTF).
• Experience developing hardware metrics collectors (in-band and out-of-band) and integrating with telemetry standards.
• Experience maintaining and scaling modern observability stacks using Prometheus, Grafana, OTEL, ClickHouse, Kafka, Superset or Elastic Stack
• Understanding of secure telemetry practices and data exposure controls.
• Experience with large-scale telemetry datasets, time series databases, down-sampling techniques, and creating actionable dashboards.
• Strong knowledge of the design and implementation of versioned, RESTful APIs.
• Good understanding of automation technologies, such as Ansible.
• Excellent written and verbal communication skills.
• Strong and demonstrable skill in at least one of C/C++/Go and Python.
• Experience programming and debugging skills for server platforms.
• Experience with containerisation (e.g. Docker). 
• Experience managing or developing in Linux environments.  

Desirable: 

• 10+ years of relevant post-degree experience.
• Previous experience working with system software for accelerators such as GPUs, DPUs, or FPGAs.
• Knowledge and experience of working with Redfish APIs.
• Familiarity with DMTF standards.
• Familiarity with Open Compute (OCP). 
• Demonstrated ability to prototype ideas and objectively evaluate their value.
• Knowledge of cloud-native development and deployment methodologies (SaaS/PaaS/IaaS).
• Knowledge of data center networking and monitoring best practices.
• Contribution to open-source communities.  
• Working knowledge of commercial observability solutions like Datadog, Dynatrace and Splunk.
• Knowledge of monitoring, observability and management solutions in use by hyperscalers and offered to their customers.
• Knowledge of declarative management systems. 

Benefits

In addition to a competitive salary, annual leave policy, medical and dental health plans, a gym card and employee pension (matched up to 4%). We review our benefits on a yearly basis to ensure we offer a valuable and rewarding benefits programme to our employees.  We welcome people of different backgrounds and experiences; we’re committed to building an inclusive work environment that makes Graphcore a great home for everyone. We offer an equal opportunity process and understand that there are visible and invisible differences in all of us. We can provide a flexible approach to interview and encourage you to chat to us if you require any reasonable adjustments.