Designs and optimizes scalable ETL/ELT pipelines using Databricks, PySpark, SQL, GCP, BigQuery, and Delta Lake. Builds agentic workflows for data validation, troubleshooting, issue identification, and operational automation. Responsibilities include developing reusable frameworks, implementing data quality and monitoring controls, optimizing workloads, and supporting testing, deployment, productionization, and ongoing enhancements. Collaborates with business, data engineering, and platform teams to identify automation opportunities.
We are looking for a Data Engineer with strong Databricks and GCP experience to develop scalable ETL/ELT pipelines and productionize agentic workflows for data engineering automation.
The mandatory requirements are 5+ years of hands-on experience with Databricks and PySpark, advanced SQL, GCP, Delta Lake, modern lakehouse architectures, and an upper-intermediate English level.
RECRUITMENT PROCESS
1. Application — Share a few details about your experience and background.
2. Coding Challenge — If applicable, complete it in the coding language you are most comfortable with.
3. Video Interview — Record a short video introduction in English.
4. Technical Interview or Hiring Manager Interview — Discuss your experience and fit for the role.
5. Offer — If it’s a match, you’ll receive an offer.
WHAT YOU’LL GAIN
• Remote work — Work from where you feel most productive.
• Local presence in India — Work in a structured and compliant environment aligned with Indian regulations.
• Competitive compensation in INR — Receive compensation in INR, plus support for learning, education, and wellness.
• Exciting projects — Work with modern technologies for global clients and fast-growing companies.
WHAT YOU WILL DO
• Design, develop, and optimize scalable ETL/ELT data pipelines using Databricks, PySpark, and SQL.
• Build and operationalize agentic workflows to automate data engineering and operational processes such as data validation, issue identification, troubleshooting, and workflow execution.
• Integrate agentic capabilities with existing Databricks, GCP, BigQuery, and Delta Lake environments.
• Develop data pipelines and reusable frameworks and components to support new business requirements and datasets across multiple use cases.
• Implement data quality checks, monitoring, validation, exception handling, and production controls.
• Optimize PySpark and SQL workloads for performance, reliability, and scalability.
• Support testing, deployment, productionization, troubleshooting, and ongoing enhancement of data and agentic solutions.
• Collaborate with business, data engineering, and platform teams to identify further automation opportunities.
MUST HAVES
• 5+ years of strong hands-on experience with Databricks and PySpark .
• Advanced SQL and data-processing skills.
• Hands-on experience with GCP , particularly BigQuery .
• Experience with Delta Lake and modern data lake/lakehouse architectures.
• Strong understanding of ETL/ELT, data pipeline design, performance optimization, and data quality.
• Experience building reliable, scalable, production-grade data solutions.
• Strong analytical and troubleshooting skills.
• Understanding of software engineering practices, including testing, version control, deployment, monitoring, and production support.
• Upper-intermediate English level.
NICE TO HAVES
• Experience developing or integrating AI/agentic workflows, AI agents, or workflow automation solutions.
• Experience applying AI to automate data engineering, validation, troubleshooting, or operational processes.
• Familiarity with orchestration and automation frameworks.
• Experience developing reusable data engineering frameworks and platform components.
• Exposure to productionizing AI-enabled solutions with appropriate validation, monitoring, and human oversight.
Similar Jobs
Artificial Intelligence • Healthtech • Machine Learning • Natural Language Processing • Biotech • Pharmaceutical
Supports Pfizer CentreOne’s inside sales and marketing activities by reviewing and routing marketing-qualified leads, executing outreach sequences, documenting activity in Salesforce, and assisting with campaigns, content, meetings, reporting, and marketing projects. The role develops commercial expertise across marketing, inside sales, and business development while progressing leads toward initial meetings and BD handoff.
Top Skills:
HubspotExcelMicrosoft OutlookMicrosoft PowerpointMicrosoft SharepointMicrosoft TeamsMicrosoft WordSalesforce
Fintech • Legal Tech • Software • Financial Services • Cybersecurity • Data Privacy
Supports payment-system workflows across client onboarding, product enhancements, production support, testing, and releases. Gathers and documents requirements, creates user stories and process flows, coordinates cross-functional stakeholders, supports UAT and defect resolution, investigates production issues, tracks risks and dependencies, and manages operational handovers. Requires flexibility for urgent production issues and planned releases.
Top Skills:
AgileAzure DevopsJIRA
Aerospace • Artificial Intelligence • Cloud • Machine Learning • Software • Cybersecurity • Defense
Leads the design, architecture, development, testing, modernization, and delivery of turbine design and analysis software tools for aerospace applications. Serves as the primary technical software lead, performs requirements analysis, establishes automated testing strategies, collaborates with global engineering teams, and mentors less-experienced engineers while ensuring software quality and process compliance.
Top Skills:
2D/3D GraphicsAPIsC++CadDllsFortranJavaJSONLinuxMatlabObject-Oriented Analysis And DesignObject-Oriented ProgrammingPythonQtWindowsXML
What you need to know about the Kolkata Tech Scene
When considering the industries shaping India's tech scene, gaming might not immediately come to mind. However, in the last decade, increased internet usage and greater access to mobile devices have catapulted the industry to new heights, with Kolkata-based companies like Virtualinfocom, Red Apple Technologies and Digitoonz, at the forefront, driving the design and animation of new gaming titles for players.
