Design and build scalable batch and real-time data pipelines using Databricks, PySpark, Structured Streaming, Delta Lake, and Medallion Architecture. Develop reusable ingestion and data quality frameworks, implement ABAC and RBAC security, and build agentic AI workflows using MCP and serverless computing. Create RAG pipelines with vector database integration, while managing governance, lineage, access controls, CI/CD, and LLM evaluation frameworks through Unity Catalog.
Senior Data engineer Key Responsibilities • 7 to 10 years of experience in data engineering and minimum of 3+ years of experience working in Databricks • 4+ years Working in leveraging Pyspark for building reusable Data ingestion and Data quality frameworks • Lakehouse & Data Engineering: Design scalable batch and real-time ETL pipelines using PySpark, Structured Streaming, and Delta Lake built on a Medallion Architecture (Bronze, Silver, Gold). • ABAC and RBAC Security - Experience in setting up ABAC and RBAC security for Agentic AI Journey use cases • Agentic AI & Orchestration: Build autonomous, multi-agent workflows and reasoning loops using MCP Protocol and Serverless compute . • RAG & Vector Integration: Develop end-to-end Retrieval-Augmented Generation (RAG) pipelines and integrate vector databases for semantic retrieval. • Governance & Operations: Manage data lineage, access controls, and security using Unity Catalog alongside CI/CD workflows and LLM evaluation frameworks
Similar Jobs
Information Technology • Consulting
Design and implement enterprise-scale data platforms, strategies, and modern cloud data architectures. Define architecture standards, build scalable and secure data platforms, and architect end-to-end solutions spanning data ingestion, transformation, storage, governance, analytics, and consumption. The role requires strong hands-on expertise with dbt and Snowflake, along with broad knowledge of the modern data ecosystem.
Top Skills:
Cloud-Native TechnologiesData LakesData WarehousingDbtSnowflake
Artificial Intelligence • Big Data • Logistics • Machine Learning • Software • Transportation
Build, deploy, monitor, and retrain machine learning models for ETA forecasting, status extraction, and logistics automation. Develop NLP and LLM pipelines, manage end-to-end production ML workflows, diagnose performance gaps, and translate model improvements into business value. Collaborate with product, engineering, and operations teams while mentoring data scientists and making architecture tradeoffs.
Top Skills:
Amazon Ec2Amazon RedshiftAmazon S3Apache AirflowApache KafkaAWSGrafanaLlmsNlpPandasPythonScikit-LearnSnowflakeSQL
Aerospace • Artificial Intelligence • Cloud • Machine Learning • Software • Cybersecurity • Defense
Develops and executes verification plans and test cases for embedded platform software, including boot software, hardware abstraction layers, I/O drivers, and compiler abstraction layers. Performs requirement- and scenario-based testing, change impact analysis, audits, issue resolution, and technical improvements. Collaborates with stakeholders, leads technical activities, guides engineers, and supports quality, cost, and schedule objectives.
Top Skills:
As9100BitbucketC++DeosDoors NgIcdJIRALauterbachLdraLinuxRtosVectorcastVectorcast/Cover
What you need to know about the Kolkata Tech Scene
When considering the industries shaping India's tech scene, gaming might not immediately come to mind. However, in the last decade, increased internet usage and greater access to mobile devices have catapulted the industry to new heights, with Kolkata-based companies like Virtualinfocom, Red Apple Technologies and Digitoonz, at the forefront, driving the design and animation of new gaming titles for players.
