Similar Jobs
This role is for one of the Weekday's clients
Salary range: Rs 1500000 - Rs 2500000 (ie INR 15-25 LPA)
Experience: 7+ yrs
Location: Chennai, Coimbatore, Bangalore, Pune
Jobtype: full-time
We are looking for a highly skilled Lead Data Engineer to design, build, and optimize scalable cloud-based data platforms and processing pipelines for enterprise-grade applications. This role is ideal for someone who enjoys solving large-scale data engineering challenges and has deep expertise in Google Cloud Platform (GCP), Apache Beam/Dataflow, Java, and BigQuery.
As a Lead Data Engineer, you will play a critical role in building robust batch and real-time data processing systems that enable analytics, reporting, and data-driven decision-making across the organization. You will work closely with architects, analysts, DevOps teams, and business stakeholders to develop reliable, scalable, and high-performance data solutions.
The ideal candidate is passionate about modern cloud data architectures, distributed systems, and building efficient ETL/ELT pipelines capable of handling large-scale enterprise workloads. This role requires strong technical expertise, problem-solving ability, and the capability to drive engineering best practices across data platforms.
RequirementsKey Responsibilities
- Design and develop scalable ETL/ELT pipelines using Google Cloud Platform services
- Build and maintain real-time and batch data processing pipelines using Apache Beam and Google Dataflow
- Develop backend processing components and data transformation services using Java
- Work extensively with BigQuery for data warehousing, analytics, querying, and performance optimization
- Integrate data from multiple sources including APIs, relational databases, streaming systems, and cloud platforms
- Build reliable and scalable streaming data pipelines using technologies such as Kafka and cloud-native services
- Optimize pipeline performance, scalability, reliability, and cloud infrastructure costs
- Ensure high standards of data quality, governance, monitoring, security, and operational excellence
- Collaborate with cross-functional teams including Architects, Analysts, DevOps, and business stakeholders to deliver data-driven solutions
- Troubleshoot production issues, perform root-cause analysis, and implement long-term scalable fixes
- Implement CI/CD practices, version control workflows, and automated deployment pipelines for data engineering solutions
- Design and maintain scalable data models, schemas, and warehouse structures
- Participate in architecture discussions and contribute to improving overall cloud data platform capabilities
- Drive best practices in distributed data processing, pipeline optimization, and cloud-native engineering
- Strong hands-on experience with Google Cloud Platform (GCP) services
- Deep expertise in Apache Beam and Google Dataflow for large-scale data processing
- Strong programming skills in Java with experience building backend processing systems
- Hands-on experience with BigQuery and cloud-based data warehousing solutions
- Experience building and maintaining batch and real-time streaming data pipelines
- Strong understanding of SQL, data modeling, and distributed data processing concepts
- Experience working with Kafka or similar streaming technologies
- Familiarity with CI/CD pipelines, Git workflows, and Agile development methodologies
- Strong analytical, troubleshooting, and debugging capabilities
- Understanding of scalable cloud architecture, performance optimization, and reliability engineering
- Ability to work collaboratively with cross-functional engineering and business teams
- Experience handling enterprise-scale datasets and complex integration requirements
- Strong communication skills with the ability to translate technical solutions into business impact
- Passion for building modern cloud-native data platforms and solving large-scale engineering challenges
