Design, build, and optimize scalable data pipelines and data platforms using PySpark, Databricks, Azure Data Factory, SQL, and Delta Lake. Develop ingestion, orchestration, transformations, data models, and large-scale processing workflows. Ensure performance, reliability, data quality, consistency, and governance while collaborating with analysts, data scientists, and business stakeholders.
This is a remote position.
Job Description – Senior Data EngineerRole Overview
We are seeking a highly skilled Senior Data Engineer to design, build, and optimize scalable data pipelines and data platforms. The ideal candidate will have strong expertise in big data technologies, cloud-based data engineering, and modern data lakehouse architectures.
- Design, develop, and maintain scalable data pipelines using PySpark and Databricks
- Build and manage data ingestion and orchestration workflows using Azure Data Factory (ADF)
- Perform complex data transformations and optimization using SQL and PySpark
- Implement and manage Delta Lake for reliable, high-performance data storage
- Develop and maintain data models (fact and dimension tables) for analytics and reporting
- Handle large-scale data processing, ensuring performance, scalability, and reliability
- Collaborate with cross-functional teams including data analysts, data scientists, and business stakeholders
- Ensure data quality, consistency, and governance across pipelines
- Strong experience in PySpark and Databricks
- Hands-on experience with Azure Data Factory (ADF)
- Proficiency in SQL and data transformation techniques
- Experience with Delta Lake and Lakehouse architecture
- Good understanding of data modeling (fact/dimension design)
- Experience handling large-scale data processing
RequirementsPreferred Qualifications (Nice to Have)
- Experience with streaming frameworks (Kafka / Structured Streaming)
- Knowledge of Azure ecosystem (ADLS, Key Vault, Synapse)
- Familiarity with data governance and security practices
- Experience in performance tuning and optimization in Spark
Similar Jobs
Cloud • Information Technology • Productivity • Security • Software • App development • Automation
Lead and execute data engineering initiatives: define metrics, instrument logging, ingest and model data, build ETL/ELT pipelines, ensure data quality and governance, optimize processing performance, enable analytics/visualization, and lead projects that drive product and business strategy.
Top Skills:
Anomaly DetectionDashboardingData ModelingData QualityData VisualizationEltETLLoggingProduct AnalyticsPythonRelational DatabasesSQL
Artificial Intelligence • Blockchain • Information Technology • Internet of Things
Designs, builds, and maintains scalable cloud data platforms and production pipelines across Azure, Databricks, Snowflake, and SAP. Responsibilities include ingestion, ETL/ELT, Lakehouse and dimensional modeling, CI/CD with Databricks Asset Bundles and Azure DevOps, data quality and observability, performance optimization, documentation, and stakeholder collaboration. The role also includes technical architecture, client advisory, engineering leadership, mentoring, and delivery of production Generative AI systems.
Top Skills:
Adls Gen2Agentic Ai ArchitecturesApache AirflowApache KafkaSparkAzureAzure DatabricksAzure DevopsAzure Event HubsDatabricks Asset BundlesDatabricks Unity CatalogDatabricks WorkflowsDelta LakeDelta Live TablesDockerGenerative AiGitLarge Language ModelsPower BIPysparkPythonRetrieval-Augmented GenerationSAPSnowflakeSpark SqlSQLTableauTerraformYaml
Information Technology • Consulting
Design, maintain, and improve web scrapers for alternative datasets; clean, transform, validate, and load data into databases or warehouses; orchestrate pipelines with Airflow; and build quality checks, alerts, and data tools. Collaborate with analysts and cross-functional teams to define data requirements, investigate time-sensitive incidents, and support investment decision-making through reliable datasets and notifications.
Top Skills:
Apache AirflowAPIsAws LambdaAws RdsAws S3Aws SnsAws SqsCi/CdDockerFiddlerGithub ActionsHTMLJavaScriptJenkinsKubernetesPandasPostmanPythonScrapySeleniumSQLXpath
What you need to know about the Kolkata Tech Scene
When considering the industries shaping India's tech scene, gaming might not immediately come to mind. However, in the last decade, increased internet usage and greater access to mobile devices have catapulted the industry to new heights, with Kolkata-based companies like Virtualinfocom, Red Apple Technologies and Digitoonz, at the forefront, driving the design and animation of new gaming titles for players.
