08.Data Engineer
About PayPay India
Why India ?
To build our Payment services, we got technical cooperation from Paytm (A large payment service company in India). And based on their customer-first technologies , we created and expanded the smartphone payment service in Japan. Therefore, we have decided to establish a development base in India, because it is a major IT country with many talented engineers, as evidenced by the fact that cutting-edge mobile payments can continue to be generated.
OUR VISION IS UNLIMITED
Job Description
PayPay's growth is driving a rapid expansion of PayPay product teams and the need for a robust Data Engineering Platform to support our growing business needs is more critical than ever. The DaaS team’s responsibility is to design, implement, and operate this platform using cutting edge technologies such as Spark, Hudi, Delta Lake, Scala, and AWS suite of data tools.
We are looking for talented Data Engineers to join our team and help us scale our platform across the organizations.
Main Responsibilities
- Design, develop, and maintain scalable data ingestion pipelines using Databricks, Airflow, Kafka, AWS Lambda, and Terraform.
- Optimize and manage large scale data pipelines to ensure high performance, reliability, and efficiency.
- Implement data processing workflows using Delta Lake, Databricks, Python, and Scala.
- Design and maintain Databricks Unity Catalog for effective data management and discovery.
- Collaborate with cross-functional teams to ensure seamless data flow and integration across the organization.
- Implement best practices for observability, data governance, security, and compliance.
Qualifications
- 4+ years experience as a Data Engineer or in a similar role.
- Hands-on experience with Delta Lake, Hudi, Spark, and Scala.
- Experience designing, building, and operating a DataLake or Data Warehouse.
- Knowledge of Data Orchestration tools such as Airflow, Dagster, Prefect.
- Strong expertise in AWS services, including Glue, Step Functions, Lambda, and EMR.
- Familiarity with change data capture tools like Canal, Debezium, and Maxwell.
- Experience with data warehousing tools like Databricks, Snowflake, and BigQuery.
- Experience in at least one primary language (e.g. Scala, Python) and SQL (any variant).
- Experience with data cataloging and metadata management using Databricks Unity Catalog, AWS Glue Data Catalog, or Lakeformation.
- Proficiency in Terraform for infrastructure as code (IaC).
- Strong problem-solving skills and ability to troubleshoot complex data issues.
- Excellent communication and collaboration skills.
- Ability to work in a fast-paced, dynamic environment and manage multiple tasks simultaneously.
Apply for this job
*
indicates a required field