Build, optimize, and monitor large-scale data pipelines using Databricks, Spark, and Delta Lake. Implement ingestion, transformation, and integration workflows, ensure data quality and performance, enable analytics with stakeholders, apply lakehouse and cloud best practices, and work in an Agile delivery model.
- Design, develop, and optimize data pipelines using Databricks, Apache Spark, and Delta Lake.
- Implement data ingestion, transformation, and integration workflows for large-scale datasets.
- Ensure data quality, reliability, and performance across all pipelines.
- Collaborate with data architects and analysts to enable analytics and reporting capabilities.
- Apply best practices for Lakehouse architecture and cloud-based data engineering.
- Monitor and troubleshoot pipeline performance, ensuring scalability and efficiency.
- Work within an Agile framework for sprint planning, backlog management, and continuous delivery.
- Design, develop, and optimize data pipelines using Databricks, Apache Spark, and Delta Lake.
- Implement data ingestion, transformation, and integration workflows for large-scale datasets.
- Ensure data quality, reliability, and performance across all pipelines.
- Collaborate with data architects and analysts to enable analytics and reporting capabilities.
- Apply best practices for Lakehouse architecture and cloud-based data engineering.
- Monitor and troubleshoot pipeline performance, ensuring scalability and efficiency.
- Work within an Agile framework for sprint planning, backlog management, and continuous delivery.
- 5+ years of experience in data engineering with strong focus on Databricks.
- Hands-on expertise in Apache Spark, Delta Lake, and Databricks Lakehouse architecture.
- Strong knowledge of Python, PySpark, and SQL.
- Experience with cloud platforms (Azure preferred) and data integration tools.
- Strong SQL skills (joins, window functions, optimization)
- Experience with cloud platforms (Azure / AWS / GCP)
- Knowledge of ETL/ELT, CDC, SCD (Type 1 & 2)
- Familiarity with CI/CD pipelines and DevOps practices.
- Understanding of data governance, metadata management, and performance tuning.
- Excellent problem-solving and communication skills.
Similar Jobs
Fintech • Professional Services • Consulting • Energy • Financial Services • Cybersecurity • Generative AI
Design, build, and maintain cloud-native data pipelines and ETL/ELT workflows for high-volume trade lifecycle data. Integrate DTCC sources (RTTM, Settlement Web) into a Unified Trade Record, process ISO 15022 messages, manage trade exception databases, optimize SQL Server performance, and ensure data quality, lineage, governance, and observability while collaborating with business stakeholders.
Top Skills:
AWSDockerDtcc RttmDtcc Settlement WebEksEltETLFixFxmlIbm MqIndexingIso 15022KafkaKubernetesRdsRest ApiS3SQLSQL ServerStored ProceduresUnified Trade Record (Utr)
Healthtech • Information Technology • Software • Consulting
Design, build, and maintain ETL pipelines and data platforms using Python, PySpark, SQL, and Airflow. Support cloud data platforms (AWS/Azure/GCP), maintain relational databases, convert unstructured data into vectors, collaborate with data/ML teams, troubleshoot pipeline issues, and document processes.
Top Skills:
AirflowAWSAzureGCPGitLangchainLlamaindexPostgresPysparkPythonRedshiftSQL
HR Tech • Legal Tech • Software • Consulting
Lead technical direction for data platforms, design and maintain data pipelines for reporting and AI (RAG, embeddings, vector stores), reduce architectural debt, mentor engineers, partner with product leadership, and represent Mitratech externally.
Top Skills:
AirbyteAnalytics PlatformsAWSBi ToolsCi/CdClaude CodeCloud StorageCopilotCursorDbtDbt Semantic LayerEmbeddingsEtl/EltFivetranGitInfrastructure-As-CodeMonitoringPostgresRag IngestionReactRuby On RailsSnowflakeSQLTerraformVector Stores
What you need to know about the Kolkata Tech Scene
When considering the industries shaping India's tech scene, gaming might not immediately come to mind. However, in the last decade, increased internet usage and greater access to mobile devices have catapulted the industry to new heights, with Kolkata-based companies like Virtualinfocom, Red Apple Technologies and Digitoonz, at the forefront, driving the design and animation of new gaming titles for players.
