Design, optimize, and productionize scalable ETL/ELT pipelines using Databricks, PySpark, SQL, GCP, BigQuery, and Delta Lake. Build agentic workflows for data validation, troubleshooting, issue identification, and workflow automation. Develop reusable frameworks, implement data quality and monitoring controls, optimize workloads, support deployment and production operations, and collaborate with business, data engineering, and platform teams.
We are looking for a Data Engineer with strong Databricks and GCP experience to build scalable ETL/ELT pipelines and productionize agentic workflows for data engineering automation.
The mandatory requirements are 5+ years of hands-on Databricks and PySpark experience, advanced SQL skills, strong GCP and BigQuery knowledge, and experience with Delta Lake and lakehouse architectures.
RECRUITMENT PROCESS
1. Application — Share a few details about your experience and background.
2. Coding Challenge — If applicable, complete it in the coding language you are most comfortable with.
3. Video Interview — Record a short video introduction in English.
4. Technical Interview or Hiring Manager Interview — Discuss your experience and fit for the role.
5. Offer — If it’s a match, you’ll receive an offer.
WHAT YOU’LL GAIN
- Remote work — Work from where you feel most productive.
- Local presence in India — Work in a structured and compliant environment aligned with Indian regulations.
- Competitive compensation in INR — Receive compensation in INR, plus support for learning, education, and wellness.
- Exciting projects — Work with modern technologies for global clients and fast-growing companies.
MUST HAVES
- 5+ years of strong hands-on experience with Databricks and PySpark.
- Advanced SQL and data-processing skills.
- Hands-on experience with GCP, particularly BigQuery.
- Experience with Delta Lake and modern data lake/lakehouse architectures.
- Strong understanding of ETL/ELT, data pipeline design, performance optimization, and data quality.
- Experience building reliable, scalable, production-grade data solutions.
- Strong analytical and troubleshooting skills.
- Understanding of software engineering practices, including testing, version control, deployment, monitoring, and production support.
- Upper-intermediate English level.
NICE TO HAVES
- Experience developing or integrating AI/agentic workflows, AI agents, or workflow automation solutions.
- Experience applying AI to automate data engineering, validation, troubleshooting, or operational processes.
- Familiarity with orchestration and automation frameworks.
- Experience developing reusable data engineering frameworks and platform components.
- Exposure to productionizing AI-enabled solutions with appropriate validation, monitoring, and human oversight.
WHAT YOU WILL DO
- Design, develop, and optimize scalable ETL/ELT pipelines using Databricks, PySpark, and SQL.
- Build and operationalize agentic workflows that automate data validation, issue identification, troubleshooting, and workflow execution.
- Integrate agentic capabilities with Databricks, GCP, BigQuery, and Delta Lake environments while supporting new business requirements and datasets.
- Build reusable frameworks and components that can support multiple data engineering and business use cases.
- Implement data quality checks, monitoring, validation, exception handling, and production controls.
- Optimize PySpark and SQL workloads and support testing, deployment, productionization, and ongoing enhancement of data and agentic solutions.
- Troubleshoot complex production issues and collaborate with business, data engineering, and platform teams to identify additional automation opportunities.
Similar Jobs
Fintech • Legal Tech • Software • Financial Services • Cybersecurity • Data Privacy
Oversee private equity fund accounting, including NAV calculations, capital accounts, waterfalls, carried interest, fees, expenses, transactions, valuations, and financial reporting. Review statements, investor packages, audits, tax-related information, and regulatory compliance. Lead and develop fund accounting teams, manage stakeholders, improve controls and processes, and support Investran or Geneva implementations.
Top Skills:
ExcelGenevaIfrsInvestranLux GaapUs Gaap
Cloud • Information Technology • Internet of Things • Machine Learning • Software • Cybersecurity • Infrastructure as a Service (IaaS)
Develops and integrates radiating components and passive radiation-pattern structures for multiband base-station antennas. Responsibilities include 3D and 2D electromagnetic simulation, prototype RF measurements, performance optimization, troubleshooting, technical documentation, and supporting the product lifecycle through mass production. The role collaborates with international cross-functional teams and requires relocation to Rosenheim, Germany.
Top Skills:
2D Rf Simulation3D Electromagnetic SimulationActive Antenna SystemsAnsys HfssAntenna DesignAwr Microwave OfficeCst Microwave StudioEmcKeysight AdsMachine LearningMatlabOptimization AlgorithmsPassive Multiband AntennasPimPythonRfRf Measurement
Cloud • Computer Vision • Information Technology • Sales • Security • Cybersecurity
Lead hands-on delivery of agentic AI solutions across Salesforce, Slack, and enterprise workflows. Design LLM-powered agents, multi-agent systems, RAG pipelines, semantic search, AI evaluation frameworks, and secure integrations. Build Salesforce components and Agentforce actions, implement CI/CD and observability, establish governance and fallback strategies, modernize legacy integrations, mentor engineers, evaluate AI platforms, and automate manual GTM processes.
Top Skills:
AgentcoreAgentforceApexAutogenAws BedrockCopadoCrewaiGithub ActionsJavaScriptJenkinsLangchainLanggraphLightning Web ComponentsLlamaindexMcpPythonRestRetrieval Augmented GenerationSalesforceSalesforce EinsteinSalesforce Platform EventsSemantic KernelSlackSoapTypescriptVector DatabasesVertex Ai
What you need to know about the Kolkata Tech Scene
When considering the industries shaping India's tech scene, gaming might not immediately come to mind. However, in the last decade, increased internet usage and greater access to mobile devices have catapulted the industry to new heights, with Kolkata-based companies like Virtualinfocom, Red Apple Technologies and Digitoonz, at the forefront, driving the design and animation of new gaming titles for players.
