Senior Principal Site Reliability Engineer (Chaos Engineering)
About Us
-
Chaos Engineering Platform Architecture & Development (50%)
-
Design and build an enterprise-grade chaos engineering platform supporting multi-cluster (K8s + EC2 hybrid), multi-region, and multi-environment (testnet/mainnet) deployments
-
Core capability development:
-
Fault Injection Engine: Pod-level / Node-level / AZ-level fault simulation, network latency / packet loss / partition, dependency timeout / error injection
-
Production Safety Assurance: Blast radius control, one-click Kill Switch, automatic rollback, real-time impact monitoring
-
Traffic Isolation: Experiment traffic tagging and isolation to ensure fault injection does not impact real users
-
Fault Isolation: Precise impact scoping at service / cluster / AZ granularity
-
-
Design experiment orchestration capabilities supporting complex fault scenario composition (e.g., simultaneous network latency + downstream timeout + cache invalidation)
-
Deep integration with existing monitoring, alerting, and SLO systems to achieve an automated closed loop: inject fault → observe impact → determine pass/fail
-
Production Resilience Validation Framework (30%)
-
Define safety standards and approval workflows for mainnet fault injection
-
Design and drive routine chaos experiments:
-
Daily patrol-level experiments: Low-risk experiments executed automatically on a daily/weekly basis
-
Periodic validation experiments: Monthly/quarterly resilience verification of critical paths
-
Large-scale drills: Cross-AZ / cross-region disaster recovery failover validation
-
Establish a resilience scoring system to quantify system health based on experiment results
-
Deliver improvement recommendations and drive business teams to remediate identified weaknesses
-
Technology Selection & Team Enablement (20%)
-
Evaluate and select the technology foundation (Chaos Mesh / Litmus / custom components — hybrid strategy)
-
Develop chaos engineering best practices and playbooks to enable SRE teams and application developers
-
Mentor and grow the team (2–3 engineers) in chaos engineering capabilities
-
Stay current with industry developments and introduce cutting-edge practices (e.g., AI-driven fault scenario discovery)
-
8+ years of backend / infrastructure engineering experience, with 3+ years dedicated to chaos engineering or stability engineering
-
Hands-on experience with large-scale fault injection in production environments (not just test environments), with deep understanding of production safety constraints
-
Expert-level proficiency in Kubernetes fault injection (Chaos Mesh / Litmus / custom solutions), familiar with CRD / Operator development
-
Proficient in at least one backend language (Go preferred), with platform-level system architecture design capability
-
Deep understanding of distributed system failure modes (network partitions, split-brain, cascading failures, data inconsistency, etc.)
-
Familiarity with observability tech stack (Prometheus / Grafana / Thanos / OpenTelemetry)
-
Excellent technical documentation and solution design skills
-
Experience in financial / trading system stability (understanding of transaction consistency and fund safety constraints)
-
Experience building SLO / Error Budget frameworks
-
Experience building automated fault recovery (self-healing) systems
-
Familiarity with AWS infrastructure (EC2 / EKS / Multi-AZ / Multi-Region)
-
Knowledge of Netflix Chaos Engineering / AWS Fault Injection Simulator / Gremlin
-
Open-source community contributions (Chaos Mesh / Litmus or similar projects)
-
Ability to balance "safety" and "validation depth" — not afraid of production injection, while maintaining strict risk control
-
Strong cross-team collaboration and influence — chaos engineering requires buy-in from business teams; this role demands persuasion skills
-
Self-driven, capable of independently planning and executing in ambiguous situations
-
混沌工程平台架构设计与建设(50%)
-
设计并构建企业级混沌工程平台,支持多集群(K8s + EC2混合)、多区域、多环境(testnet/mainnet)
-
核心能力建设:
-
故障注入引擎: Pod级/Node级/AZ级故障模拟,网络延迟/丢包/分区,依赖超时/错误注入
-
生产环境安全保障: 爆炸半径控制、一键停止(Kill Switch)、自动回滚、影响范围实时监控
-
流量隔离: 实验流量标记与隔离,确保故障注入不影响真实用户
-
故障隔离: 按服务/集群/AZ粒度精准控制影响范围
-
-
设计实验编排能力:支持复杂故障场景的组合编排(如:同时注入网络延迟+下游超时+缓存失效)
-
与现有监控、告警、SLO体系深度集成,实现"注入故障→观测影响→判定通过/失败"的自动化闭环
-
生产环境韧性验证体系(30%)
-
制定主网故障注入的安全规范和审批流程
-
设计和推动常态化混沌实验:
-
日常巡检级实验(每日/每周自动执行低风险实验)
-
定期验证级实验(月度/季度核心链路韧性验证)
-
大规模演练(跨AZ/跨Region容灾切换验证)
-
-
建立韧性评分体系:基于实验结果量化系统健康度
-
输出改进建议并推动业务方修复发现的弱点
-
技术方案选型与团队赋能(20%)
-
评估并选择技术底座(Chaos Mesh / Litmus / 自研组件的组合策略)
-
制定混沌工程最佳实践和Playbook,赋能SRE团队和业务研发
-
培养团队(2-3人)的混沌工程能力
-
与业界保持同步,引入前沿实践(如:AI驱动的故障场景发现)
-
8年+后端/基础设施开发经验,3年+混沌工程或稳定性工程专项经验
-
有生产环境大规模故障注入的实操经验(非仅测试环境),深刻理解生产安全约束
-
精通Kubernetes故障注入(Chaos Mesh/Litmus/自研方案),熟悉CRD/Operator开发
-
精通至少一种后端语言(Go优先),具备平台级系统架构设计能力
-
深入理解分布式系统故障模式(网络分区、脑裂、级联故障、数据不一致等)
-
熟悉可观测性技术栈(Prometheus/Grafana/Thanos/OpenTelemetry)
-
优秀的技术文档和方案设计能力
-
有金融/交易系统稳定性保障经验(理解交易一致性、资金安全约束)
-
有SLO/Error Budget体系建设经验
-
有自动化故障恢复(Self-healing)系统建设经验
-
熟悉AWS基础设施(EC2/EKS/Multi-AZ/Multi-Region)
-
了解Netflix Chaos Engineering / AWS Fault Injection Simulator / Gremlin
-
有开源社区贡献经验(Chaos Mesh/Litmus等项目)
-
能在"安全"和"验证深度"之间做好平衡——既不畏惧生产注入,也能严格控制风险
-
有推动跨团队协作的能力——混沌工程需要业务方配合,这个岗位需要说服力
-
自驱型,能在方向模糊时独立规划和推进
Why Join Us
At Bybit, we are committed to fostering a supportive and enriching work environment.
Our benefits include:
- Study Growth Fund: We support your professional development and continuous learning.
- Internal Events: Participate in regular team-building activities, workshops, and events designed to promote collaboration and innovation.
- Global Collaboration: Be part of a diverse, international team, working alongside colleagues from around the world.
- Career Advancement: Access opportunities for growth and advancement within a rapidly expanding global company.
- Internal Mobility: Grow with us- Your long-term development is important to us. We offer internal job opportunities to help build your career path.
Apply for this job
*
indicates a required field

