Design, build, deploy, and maintain scalable GCP data platforms and pipelines using Python, SQL, and related technologies. Ensure data quality, governance, security, and efficient analytical use while optimizing performance, scalability, and cost. Manage workflow orchestration and infrastructure through Astronomer and Terraform, implement automation and CI/CD, collaborate with architects and cross-functional teams, translate business requirements into data solutions, and document processes.
- The ideal candidate will have strong experience in data engineering, cloud technologies, and software development, with a passion for building reliable, scalable, and secure data solutions.Required technical expertise includes:
- Strong proficiency in SQL, Python, and Java.
- Hands-on experience designing, developing, and deploying cloud-based data pipelines using Google Cloud Platform (GCP), including BigQuery, Dataflow, and Dataproc.
- Experience with relational databases such as PostgreSQL and MySQL, as well as NoSQL and columnar databases.
- Understanding of Service-Oriented Architecture (SOA) and microservices-based solutions.
- Knowledge of data governance, security controls, encryption, and data masking techniques.
- Experience implementing CI/CD pipelines and Infrastructure as Code (IaC) using tools such as Terraform and Tekton.
- Ability to monitor, troubleshoot, and optimize cloud workloads for performance, scalability, and cost efficiency.
- Strong analytical, problem-solving, and communication skills.
- Data Pipeline Architect & Builder: Spearhead the design, development, and maintenance of scalable data ingestion and curation pipelines from diverse sources. Ensure data is standardized, high-quality, and optimized for analytical use. Leverage cutting-edge tools and technologies, including Python, SQL, and DBT/Dataform, to build robust and efficient data pipelines.
- End-to-End Integration Expert: Utilize your full-stack skills to contribute to seamless end-to-end development, ensuring smooth and reliable data flow from source to insight.
- GCP Data Solutions Leader: Leverage your deep expertise in GCP services (BigQuery, Dataflow, Pub/Sub, Cloud Functions, etc.) to build and manage data platforms that not only meet but exceed business needs and expectations.
- Data Governance & Security Champion: Implement and manage robust data governance policies, access controls, and security best practices, fully utilizing GCP's native security features to protect sensitive data.
- Data Workflow Orchestrator: Employ Astronomer and Terraform for efficient data workflow management and cloud infrastructure provisioning, championing best practices in Infrastructure as Code (IaC).
- Performance Optimization Driver: Continuously monitor and improve the performance, scalability, and efficiency of data pipelines and storage solutions, ensuring optimal resource utilization and cost-effectiveness.
- Collaborative Innovator: Collaborate effectively with data architects, application architects, service owners, and cross-functional teams to define and promote best practices, design patterns, and frameworks for cloud data engineering.
- Automation & Reliability Advocate: Proactively automate data platform processes to enhance reliability, improve data quality, minimize manual intervention, and drive operational efficiency.
- Effective Communicator: Clearly and transparently communicate complex technical decisions to both technical and non-technical stakeholders, fostering understanding and alignment.
- Continuous Learner: Stay ahead of the curve by continuously learning about industry trends and emerging technologies, proactively identifying opportunities to improve our data platform and enhance our capabilities.
- Business Impact Translator: Translate complex business requirements into optimized data asset designs and efficient code, ensuring that our data solutions directly contribute to business goals.
- Documentation & Knowledge Sharer: Develop comprehensive documentation for data engineering processes, promoting knowledge sharing, facilitating collaboration, and ensuring long-term system maintainability.
Required
- Bachelor's degree in Computer Science, Information Technology, Information Systems, Data Analytics, Engineering, or a related field, or equivalent practical experience.
- 5 to 7 years of experience in Data Engineering or Software Engineering.
- Minimum 2 years of hands-on experience building and deploying cloud-based data platforms, preferably on Google Cloud Platform (GCP).
- Strong proficiency in SQL and Python.
- Experience with BigQuery, Dataflow, Dataproc, and cloud-based data processing technologies.
- Experience with Terraform, CI/CD pipelines, and automation frameworks.
- Knowledge of cloud security, data governance, and data quality best practices.
Preferred
- Experience with DBT, Dataform, or similar transformation frameworks.
- Experience with Apache Airflow or Astronomer.
- Experience designing microservices and API-based integrations.
- Exposure to FinOps practices and cloud cost optimization.
#LI-SKV
Similar Jobs
Cloud • Information Technology • Productivity • Security • Software • App development • Automation
Lead data engineering architecture and execution for large-scale, low-latency analytics. Own technical direction, ensure operational data quality, design streaming and batch pipelines, mentor engineers, coordinate cross-functional teams, and deliver scalable solutions using Spark, Airflow, and AWS or equivalent data platforms.
Top Skills:
AirflowAthenaColumn StoresDatabricksDatabricks ApisEmrFlinkHiveKafkaKappa ArchitectureLambda ArchitectureMaster Data Management (Mdm)Microservices ArchitectureRedshiftSparkSQLStreaming Pipelines
Cloud • Information Technology • Productivity • Security • Software • App development • Automation
Lead and execute data engineering initiatives: define metrics, instrument logging, ingest and model data, build ETL/ELT pipelines, ensure data quality and governance, optimize processing performance, enable analytics/visualization, and lead projects that drive product and business strategy.
Top Skills:
Anomaly DetectionDashboardingData ModelingData QualityData VisualizationEltETLLoggingProduct AnalyticsPythonRelational DatabasesSQL
Cloud • Information Technology • Professional Services • Software
Build production-grade agentic AI systems that reason, search, and execute workflows. Develop RAG pipelines, AI agents, semantic search layers, data processing solutions, dimensional data models, and scalable AI workloads. Work with structured and unstructured data, LLMs, vector databases, cloud infrastructure, CI/CD, and ML observability. Responsibilities include fine-tuning models, prompt engineering, optimizing context windows, and deploying reliable AI applications.
Top Skills:
Ai AgentsAmazon RedshiftAws BedrockAws LambdaCi/CdClaudeEmbedding ModelsGeminiGoogle IndexingGptLangchainLlamaindexMl ObservabilityPandasPgvectorPysparkPythonRagScalaSnowflakeSQLTransformer Architectures
What you need to know about the Kolkata Tech Scene
When considering the industries shaping India's tech scene, gaming might not immediately come to mind. However, in the last decade, increased internet usage and greater access to mobile devices have catapulted the industry to new heights, with Kolkata-based companies like Virtualinfocom, Red Apple Technologies and Digitoonz, at the forefront, driving the design and animation of new gaming titles for players.
