Mastercard
Senior Software Engineer - Big Data / Streaming Platforms (Spark / Kafka / NiFi)
Be an Early Applicant
This role entails testing and validating big data platforms focusing on real-time and batch processing, ensuring data correctness and system reliability.
Our Purpose
Mastercard powers economies and empowers people in 200+ countries and territories worldwide. Together with our customers, we're helping build a sustainable economy where everyone can prosper. We support a wide range of digital payments choices, making transactions secure, simple, smart and accessible. Our technology and innovation, partnerships and networks combine to deliver a unique set of products and services that help people, businesses and governments realize their greatest potential.
Title and Summary
Senior Software Engineer - Big Data / Streaming Platforms (Spark / Kafka / NiFi)
Job Posting Title: Senior Software Engineer - Big Data / Streaming Platforms (Spark / Kafka / NiFi)Who is Mastercard?
Mastercard is a global technology company in the payments industry. Our mission is to connect and power an inclusive, digital economy that benefits everyone, everywhere by making transactions safe, simple, smart, and accessible. Using secure data and networks, partnerships and passion, our innovations and solutions help individuals, financial institutions, governments, and businesses realize their greatest potential. Our culture is guided by the Mastercard Way-own it, simplify it, sense of urgency, thoughtful risk taking, unlock potential, and be inclusive.Job Summary:
We are looking for a highly skilled Senior QA Engineer / SDET with deep expertise in testing large scale batch and real time data platforms built on Apache Spark, Kafka, and Apache NiFi.
This role focuses on validating streaming heavy architectures, ensuring data correctness, latency, fault tolerance, and reliability across complex distributed systems leveraging object storage platforms such as Apache Ozone and Ceph.
The ideal candidate is passionate about data quality, test automation, observability, and production readiness for mission critical big data systems.
# Key Responsibilities
- Analyze business, regulatory, and technical requirements to design risk based and audit compliant test scenarios.
- Develop, maintain, and execute test plans, test cases, and test data with appropriate documentation and traceability to requirements.
- Perform functional, regression, and system integration testing in accordance with enterprise quality standards and release controls.
- Log, track, and manage defects in JIRA, ensuring clear reproduction steps, impact assessment, and lifecycle traceability.
- Validate JSON payloads, cloud storage data, and Apache NiFi data flows to ensure data accuracy, completeness, and integrity.
- Collaborate with development teams to support timely defect resolution and adherence to quality gates.
- Participate in design reviews, test plan reviews, and technical documentation
- Participate in Agile ceremonies, providing clear and accurate test execution status, risk assessments, and quality metrics.
- Test Kafka based streaming pipelines, including topic partitioning, offsets, throughput, and ordering guarantees
- Verify event replay and reprocessing flows using Kafka offsets and Spark checkpoints
- Test Apache NiFi flows for ingestion, routing, back pressure handling, error paths, and retry semantics
- Execute performance, load, and soak testing for Spark batch and streaming workloads
- Collaborate with engineering teams to validate partitioning strategies, resource tuning, and scalability assumptions
- Build and maintain test automation frameworks for data validation, streaming verification, and regression testing
- Automate validation using SQL, PySpark, and Python based test harnesses
- Integrate automated tests into CI/CD pipelines for data applications
- Participate in production validation, go live readiness, and post deployment verification
- Support incident analysis through data validation, replay testing, and root cause analysis
- Validate observability metrics such as lag, throughput, latency, error rates, and data drift
# Required Qualifications
- Bachelor's degree in Computer Science, Engineering, or equivalent practical experience
- Strong hands on experience of 6+ years in testing big data platforms in production environments
- Solid understanding of Apache Spark (batch and streaming) from a testing and validation perspective
- Working knowledge of Apache NiFi, JSON, and cloud based data validation.
- Strong analytical, problem solving, and communication skills, with the ability to work effectively across cross functional teams.
- Strong SQL skills for data validation and reconciliation
- Proficiency with Linux, shell scripting, Git, and basic CI/CD concepts
- Proficiency in JIRA for defect tracking, reporting, and quality metrics
Corporate Security Responsibility
All activities involving access to Mastercard assets, information, and networks comes with an inherent risk to the organization and, therefore, it is expected that every person working for, or on behalf of, Mastercard is responsible for information security and must:
Mastercard powers economies and empowers people in 200+ countries and territories worldwide. Together with our customers, we're helping build a sustainable economy where everyone can prosper. We support a wide range of digital payments choices, making transactions secure, simple, smart and accessible. Our technology and innovation, partnerships and networks combine to deliver a unique set of products and services that help people, businesses and governments realize their greatest potential.
Title and Summary
Senior Software Engineer - Big Data / Streaming Platforms (Spark / Kafka / NiFi)
Job Posting Title: Senior Software Engineer - Big Data / Streaming Platforms (Spark / Kafka / NiFi)Who is Mastercard?
Mastercard is a global technology company in the payments industry. Our mission is to connect and power an inclusive, digital economy that benefits everyone, everywhere by making transactions safe, simple, smart, and accessible. Using secure data and networks, partnerships and passion, our innovations and solutions help individuals, financial institutions, governments, and businesses realize their greatest potential. Our culture is guided by the Mastercard Way-own it, simplify it, sense of urgency, thoughtful risk taking, unlock potential, and be inclusive.Job Summary:
We are looking for a highly skilled Senior QA Engineer / SDET with deep expertise in testing large scale batch and real time data platforms built on Apache Spark, Kafka, and Apache NiFi.
This role focuses on validating streaming heavy architectures, ensuring data correctness, latency, fault tolerance, and reliability across complex distributed systems leveraging object storage platforms such as Apache Ozone and Ceph.
The ideal candidate is passionate about data quality, test automation, observability, and production readiness for mission critical big data systems.
# Key Responsibilities
- Analyze business, regulatory, and technical requirements to design risk based and audit compliant test scenarios.
- Develop, maintain, and execute test plans, test cases, and test data with appropriate documentation and traceability to requirements.
- Perform functional, regression, and system integration testing in accordance with enterprise quality standards and release controls.
- Log, track, and manage defects in JIRA, ensuring clear reproduction steps, impact assessment, and lifecycle traceability.
- Validate JSON payloads, cloud storage data, and Apache NiFi data flows to ensure data accuracy, completeness, and integrity.
- Collaborate with development teams to support timely defect resolution and adherence to quality gates.
- Participate in design reviews, test plan reviews, and technical documentation
- Participate in Agile ceremonies, providing clear and accurate test execution status, risk assessments, and quality metrics.
- Test Kafka based streaming pipelines, including topic partitioning, offsets, throughput, and ordering guarantees
- Verify event replay and reprocessing flows using Kafka offsets and Spark checkpoints
- Test Apache NiFi flows for ingestion, routing, back pressure handling, error paths, and retry semantics
- Execute performance, load, and soak testing for Spark batch and streaming workloads
- Collaborate with engineering teams to validate partitioning strategies, resource tuning, and scalability assumptions
- Build and maintain test automation frameworks for data validation, streaming verification, and regression testing
- Automate validation using SQL, PySpark, and Python based test harnesses
- Integrate automated tests into CI/CD pipelines for data applications
- Participate in production validation, go live readiness, and post deployment verification
- Support incident analysis through data validation, replay testing, and root cause analysis
- Validate observability metrics such as lag, throughput, latency, error rates, and data drift
# Required Qualifications
- Bachelor's degree in Computer Science, Engineering, or equivalent practical experience
- Strong hands on experience of 6+ years in testing big data platforms in production environments
- Solid understanding of Apache Spark (batch and streaming) from a testing and validation perspective
- Working knowledge of Apache NiFi, JSON, and cloud based data validation.
- Strong analytical, problem solving, and communication skills, with the ability to work effectively across cross functional teams.
- Strong SQL skills for data validation and reconciliation
- Proficiency with Linux, shell scripting, Git, and basic CI/CD concepts
- Proficiency in JIRA for defect tracking, reporting, and quality metrics
Corporate Security Responsibility
All activities involving access to Mastercard assets, information, and networks comes with an inherent risk to the organization and, therefore, it is expected that every person working for, or on behalf of, Mastercard is responsible for information security and must:
- Abide by Mastercard's security policies and practices;
- Ensure the confidentiality and integrity of the information being accessed;
- Report any suspected information security violation or breach, and
- Complete all periodic mandatory security trainings in accordance with Mastercard's guidelines.
Similar Jobs at Mastercard
Blockchain • Fintech • Payments • Consulting • Cryptocurrency • Cybersecurity • Quantum Computing
The Senior Accountant manages expense recording and reconciles accounts, ensures compliance with regulations, and provides analysis and reporting support.
Top Skills:
BeelineBlacklineHyperionOracle
Blockchain • Fintech • Payments • Consulting • Cryptocurrency • Cybersecurity • Quantum Computing
The Manager Systems Analyst leads a team in producing specifications, use cases, and documentation, while managing processes and mentoring team members in an Agile environment.
Top Skills:
AgileMS OfficeScrumVisio
Blockchain • Fintech • Payments • Consulting • Cryptocurrency • Cybersecurity • Quantum Computing
As a Software Engineer II, you'll design, develop, and maintain software solutions while collaborating with project teams and enhancing code quality and processes.
Top Skills:
Software Engineering Concepts
What you need to know about the Kolkata Tech Scene
When considering the industries shaping India's tech scene, gaming might not immediately come to mind. However, in the last decade, increased internet usage and greater access to mobile devices have catapulted the industry to new heights, with Kolkata-based companies like Virtualinfocom, Red Apple Technologies and Digitoonz, at the forefront, driving the design and animation of new gaming titles for players.
