Design, build, and optimize end-to-end data pipelines for large structured and unstructured data. Implement near-real-time ETL, data validation, monitoring, and performance optimization. Collaborate with stakeholders, document designs and workflows, and provide technical guidance to the team.
Location: Pune
Responsibilities include:- Design, implement, and optimize end-to-end data pipelines for ingesting, processing, and transforming large volumes of structured and unstructured data.
- Develop data pipelines to extract and transform data in near real-time using cloud-native technologies.
- Implement data validation and quality checks to ensure accuracy and consistency.
- Monitor system performance, troubleshoot issues, and implement optimizations to enhance reliability and efficiency.
- Collaborate with business users, analysts, and other stakeholders to understand data requirements and deliver tailored solutions.
- Document technical designs, workflows, and best practices to facilitate knowledge sharing and maintain system documentation.
- Provide technical guidance and support to team members and stakeholders as needed.
- 8+ years of work experience.
- Proficiency in writing complex SQL queries on MPP systems (Snowflake/Redshift).
- Experience in Databricks and Delta tables.
- Data engineering experience with Spark/Scala/Python.
- Experience in Microsoft Azure stack (Azure Storage Accounts, Data Factory, and Databricks).
- Experience in Azure DevOps and CI/CD pipelines.
- Working knowledge of Python.
- Comfortable participating in 2-week sprint development cycles.
Similar Jobs
Agency • Information Technology
Design, build, optimize, and maintain high-performance Spark-based data pipelines using Scala/Java and Hive on Hadoop/CDP. Own full project lifecycle, enforce coding best practices, troubleshoot Spark/Hive/YARN performance, and collaborate with stakeholders to deliver scalable data solutions.
Top Skills:
SparkCloudera Data Platform (Cdp)HadoopHiveJavaScalaYarn
Cloud • Information Technology • Productivity • Software • Automation
As a Product Support Sr. Engineer, you will troubleshoot complex technical issues, work with the Boomi AtomSphere Platform, and ensure customer success across various global regions.
Top Skills:
Boomi Atomsphere PlatformCharles ProxyEltETLGroovyHadoopHttp/SJavaJavaScriptKubernetesLinux OsNetSuiteOauth 2.0PostmanRancher DesktopReactRestSalesforceSftpSoapSsl/TlsTcp/IpWindows OsWiresharkWsdl
Cloud • Information Technology • Security • Software
Lead quality efforts for major product areas, define test strategies, write and maintain complex automated tests, enable developers to shift testing left, participate in incident reviews and root-cause analysis, mentor junior QEs, and ensure releases meet reliability, performance, and security standards while partnering with architects and development leads.
Top Skills:
Ci/CdJavaScriptPlaywrightPytestPythonTypescript
What you need to know about the Kolkata Tech Scene
When considering the industries shaping India's tech scene, gaming might not immediately come to mind. However, in the last decade, increased internet usage and greater access to mobile devices have catapulted the industry to new heights, with Kolkata-based companies like Virtualinfocom, Red Apple Technologies and Digitoonz, at the forefront, driving the design and animation of new gaming titles for players.
