As a Senior Data Engineer, you'll automate data workflows, troubleshoot complex data pipeline issues, optimize performance, and mentor junior engineers.
Embark on an exciting journey into the realm of software development with 3Pillar! We extend an invitation for you to join our team and gear up for a thrilling adventure. At 3Pillar, our focus is on leveraging cutting-edge technologies that revolutionize industries by enabling data driven decision making. As a Senior Data Engineer, you will hold a crucial position within our dynamic team, actively contributing to thrilling projects that reshape data analytics for our clients, providing them with a competitive advantage in their respective industries.
Remote Opening
Shift: Day Shift (9-hour shift, 5 days a week, weekends and holidays on rotation).
This role requires covering weekends and holidays on a rotational basis to ensure 7-day-a-week coverage.
The critical time window to be covered is 11:00 AM to 7:00 PM IST
Responsibilities:
- Team & Operational Management
- Team Management: Manage the team of 3-4 Data Engineers.
- Roster & Scheduling: Create, manage, and maintain team rosters and schedules to ensure full 8-hour coverage, 7 days a week, including rotational weekend and holiday coverage.
- Availability: Must be able to work a day shift and manage a team that provides 7-day-a-week coverage, including weekends and holidays on rotation.
- Coverage Assurance: Handle all team leave requests (planned and unplanned). You are responsible for ensuring coverage is always maintained, especially during medical or other unexpected absences.
- Performance & SLAs: Act as the primary operational point of contact, ensuring the team executes all tasks efficiently, makes sound decisions, and adheres to all agreed-upon SLAs.
- Technical Leadership & Engineering
- Final Escalation Point: Serve as the final technical escalation point for the team; personally handle the most critical job failures and system issues.
- Expert RCA: Lead all high-priority Root Cause Analysis (RCA) efforts, documenting solutions and implementing permanent fixes to prevent recurrence.
- Strategic Improvement: Drive the team's efforts in performance tuning and AWS Glue cost optimization.
- Automation Strategy: Define and lead the development of new automation jobs (using Glue, Lambda, and Step Functions) to reduce manual support tasks.
- Complex Data Management: Oversee all data integrity and synchronization tasks, including CDC reloads and write-backs to external databases like Teradata.
- Knowledge Transfer: Own the knowledge transfer process to internal resources and ensure the team maintains excellent documentation and runbooks.
Qualifications:
- Technical Expertise: In-depth, expert-level knowledge and extensive hands-on experience with AWS S3, Glue, Lambda, Step Functions and other related services.
- Demonstrated expertise with a minimum of 5-8 years of relevant experience.
- Deep understanding of Data engineering, ETL principles and data warehouse design.
- Optimization Experience: Demonstrable experience in system performance tuning and cloud cost optimization, specifically with AWS Glue.
- Automation: Proven ability to design and develop automation solutions from the ground up.
- Problem-Solving: Expert-level troubleshooting and RCA skills.
- Leadership: Ability to work independently, make critical decisions, and provide technical guidance.
- Availability: Must be able to work a day shift that includes coverage for weekends and holidays on a rotational basis.
- Database Skills: Strong experience with Change Data Capture (CDC), managing database synchronization, and working with Teradata.
- Preferred Skills: AWS Certified.
Top Skills
Aws Glue
Aws Lambda
Aws S3
Aws Step Functions
Teradata
Similar Jobs
.png)
Artificial Intelligence • Blockchain • Fintech • Financial Services • Cryptocurrency • NFT • Web3
As a Staff Software Engineer, you'll architect and develop platform solutions, mentor junior engineers, and solve technical challenges utilizing Golang and modern cloud technologies, within a high-performing team focusing on AI and customer experience.
Top Skills:
DockerDynamoDBGoLanguage ModelsMongoDBPostgres
Artificial Intelligence • Blockchain • Fintech • Financial Services • Cryptocurrency • NFT • Web3
Architect and develop payment solutions for Coinbase's backend, focusing on scalable systems, service-oriented architecture, and high-quality code. Collaborate with cross-functional teams and provide technical structure and strategic roadmaps.
Top Skills:
DockerDynamoDBGoJavaMongoDBPostgres
Artificial Intelligence • Blockchain • Fintech • Financial Services • Cryptocurrency • NFT • Web3
As a Staff Software Engineer at Coinbase, you will architect and develop platforms, mentor engineers, and ensure code quality, while collaborating on strategic roadmaps.
Top Skills:
SparkDockerGoGrpcHiveSQL
What you need to know about the Kolkata Tech Scene
When considering the industries shaping India's tech scene, gaming might not immediately come to mind. However, in the last decade, increased internet usage and greater access to mobile devices have catapulted the industry to new heights, with Kolkata-based companies like Virtualinfocom, Red Apple Technologies and Digitoonz, at the forefront, driving the design and animation of new gaming titles for players.
