Lead Data Engineer

Key Responsibilities

Data Engineering & Platform Architecture

• Architect, design, and govern scalable batch and streaming data platforms using Python (strong OOP design), Apache Spark (PySpark, Spark SQL), and Azure Databricks.
• Define end-to-end reference architectures for real-time and near-real-time streaming solutions using Apache Kafka or Apache Flink, with Spark Structured Streaming or Apache Flink, ensuring:
  • Event-time correctness
  • Exactly-once processing guarantees
  • Stateful stream processing and checkpointing
  • High availability and fault tolerance
• Own the architectural design of Medallion architecture (Bronze, Silver, Gold) on ADLS Gen2 with Delta Lake, including data lifecycle, retention, and cost optimization strategies.
• Lead data governance architecture, defining standards for security, access control, lineage, and metadata using Databricks Unity Catalog.
• Design domain-oriented, scalable data products aligned with Data Mesh and cloud-native architecture principles.
• Define integration patterns for data services, streaming producers/consumers, and microservices-based architectures.
• Architect and oversee deployment of data and streaming workloads on AKS, including container strategy, scaling, resiliency, and networking.
• Define and enforce performance, scalability, and reliability standards for Spark and streaming workloads (partitioning, Z-ordering, state tuning, caching).
• Establish data quality, validation, and schema evolution standards for both batch and streaming pipelines.
• Design secure-by-default architectures using Azure IAM, RBAC, Key Vault, private endpoints, VNet integration, and network isolation.
• Lead CI/CD and DevOps architecture using GitLab, enabling automated testing, deployment, rollback, and environment promotion.
• Define Infrastructure as Code architecture using Terraform and Pulumi for repeatable, auditable deployments.
• Establish observability architecture (monitoring, logging, alerting) across Databricks, AKS, streaming platforms, and Azure services.
• Review and approve solution designs, ADRs, and technical proposals, ensuring alignment with enterprise standards.

Product Owner (PO) Responsibilities

• Act as Technical Product Owner for data platforms and streaming solutions, owning one or more data domains or products.
• Partner with business stakeholders, analysts, and architects to translate business objectives into architecture-aligned epics, user stories, and acceptance criteria.
• Own and prioritize the product and technical backlog, balancing feature delivery, technical debt, scalability, security, and cost efficiency.
• Define and track product KPIs (data latency, data quality, availability, adoption, platform cost).
• Drive roadmap definition for data and streaming platforms, aligned with enterprise strategy and architectural vision.
• Manage dependencies, risks, and cross-team coordination across engineering, analytics, DevOps, and security teams.
• Support release planning, stakeholder communication, and architectural decision-making.
• Act as a subject-matter expert and decision authority for the data platform.

Leadership & Engineering Excellence

• Provide architectural and technical leadership to data engineers and cross-functional teams.
• Conduct design reviews, code reviews, and architecture walkthroughs.
• Mentor engineers on distributed systems, streaming design, cloud-native patterns, and performance optimization.
• Establish and enforce coding standards, design patterns, and best practices.
• Champion continuous improvement, innovation, and engineering excellence.

Required Skills & Qualifications

• Bachelor’s or Master’s degree in Computer Science, Engineering, or a related field.
• 8–12 years of experience in Data Engineering, with demonstrated architecture ownership on Azure-based platforms.
• Strong proficiency in Python, with solid OOP, design patterns, and system design principles.
• Deep expertise in Apache Spark (PySpark, Spark SQL) and Azure Databricks.
• Strong hands-on and architectural experience with streaming platforms:
  • Apache Kafka OR Apache Flink
  • Spark Structured Streaming
• Proven experience designing microservices and event-driven architectures.
• Strong experience deploying and operating workloads on Azure Kubernetes Service (AKS).
• Deep understanding of Delta Lake and large-scale lakehouse architectures.
• Advanced SQL skills for analytics and optimization.
• Strong experience with Azure ADLS Gen2, Databricks, Azure Functions, Service Bus, Key Vault.
• Strong experience with Git, GitLab CI/CD, and release management.
• Experience with Terraform and Pulumi for enterprise-grade IaC.
• Strong knowledge of data modeling, distributed systems, and fault tolerance

Nice to Have

• Experience with both Kafka and Flink in large-scale production systems.
• Exposure to Kafka Schema Registry, CDC, and event versioning.
• Experience implementing Data Mesh or domain-driven data platforms.
• Exposure to DevSecOps and Zero Trust architectures.
• Experience with Generative AI / LLM-enabled data platforms (RAG, embeddings, vector databases).
• Background in regulated or large enterprise environments (banking, automotive, telecom).

Key Responsibilities

Data Engineering & Platform Architecture

• Architect, design, and govern scalable batch and streaming data platforms using Python (strong OOP design), Apache Spark (PySpark, Spark SQL), and Azure Databricks.
• Define end-to-end reference architectures for real-time and near-real-time streaming solutions using Apache Kafka or Apache Flink, with Spark Structured Streaming or Apache Flink, ensuring:
  • Event-time correctness
  • Exactly-once processing guarantees
  • Stateful stream processing and checkpointing
  • High availability and fault tolerance
• Own the architectural design of Medallion architecture (Bronze, Silver, Gold) on ADLS Gen2 with Delta Lake, including data lifecycle, retention, and cost optimization strategies.
• Lead data governance architecture, defining standards for security, access control, lineage, and metadata using Databricks Unity Catalog.
• Design domain-oriented, scalable data products aligned with Data Mesh and cloud-native architecture principles.
• Define integration patterns for data services, streaming producers/consumers, and microservices-based architectures.
• Architect and oversee deployment of data and streaming workloads on AKS, including container strategy, scaling, resiliency, and networking.
• Define and enforce performance, scalability, and reliability standards for Spark and streaming workloads (partitioning, Z-ordering, state tuning, caching).
• Establish data quality, validation, and schema evolution standards for both batch and streaming pipelines.
• Design secure-by-default architectures using Azure IAM, RBAC, Key Vault, private endpoints, VNet integration, and network isolation.
• Lead CI/CD and DevOps architecture using GitLab, enabling automated testing, deployment, rollback, and environment promotion.
• Define Infrastructure as Code architecture using Terraform and Pulumi for repeatable, auditable deployments.
• Establish observability architecture (monitoring, logging, alerting) across Databricks, AKS, streaming platforms, and Azure services.
• Review and approve solution designs, ADRs, and technical proposals, ensuring alignment with enterprise standards.

Product Owner (PO) Responsibilities

• Act as Technical Product Owner for data platforms and streaming solutions, owning one or more data domains or products.
• Partner with business stakeholders, analysts, and architects to translate business objectives into architecture-aligned epics, user stories, and acceptance criteria.
• Own and prioritize the product and technical backlog, balancing feature delivery, technical debt, scalability, security, and cost efficiency.
• Define and track product KPIs (data latency, data quality, availability, adoption, platform cost).
• Drive roadmap definition for data and streaming platforms, aligned with enterprise strategy and architectural vision.
• Manage dependencies, risks, and cross-team coordination across engineering, analytics, DevOps, and security teams.
• Support release planning, stakeholder communication, and architectural decision-making.
• Act as a subject-matter expert and decision authority for the data platform.

Leadership & Engineering Excellence

• Provide architectural and technical leadership to data engineers and cross-functional teams.
• Conduct design reviews, code reviews, and architecture walkthroughs.
• Mentor engineers on distributed systems, streaming design, cloud-native patterns, and performance optimization.
• Establish and enforce coding standards, design patterns, and best practices.
• Champion continuous improvement, innovation, and engineering excellence.

Required Skills & Qualifications

• Bachelor’s or Master’s degree in Computer Science, Engineering, or a related field.
• 8–12 years of experience in Data Engineering, with demonstrated architecture ownership on Azure-based platforms.
• Strong proficiency in Python, with solid OOP, design patterns, and system design principles.
• Deep expertise in Apache Spark (PySpark, Spark SQL) and Azure Databricks.
• Strong hands-on and architectural experience with streaming platforms:
  • Apache Kafka OR Apache Flink
  • Spark Structured Streaming
• Proven experience designing microservices and event-driven architectures.
• Strong experience deploying and operating workloads on Azure Kubernetes Service (AKS).
• Deep understanding of Delta Lake and large-scale lakehouse architectures.
• Advanced SQL skills for analytics and optimization.
• Strong experience with Azure ADLS Gen2, Databricks, Azure Functions, Service Bus, Key Vault.
• Strong experience with Git, GitLab CI/CD, and release management.
• Experience with Terraform and Pulumi for enterprise-grade IaC.
• Strong knowledge of data modeling, distributed systems, and fault tolerance.

Nice to Have

• Experience with both Kafka and Flink in large-scale production systems.
• Exposure to Kafka Schema Registry, CDC, and event versioning.
• Experience implementing Data Mesh or domain-driven data platforms.
• Exposure to DevSecOps and Zero Trust architectures.
• Experience with Generative AI / LLM-enabled data platforms (RAG, embeddings, vector databases).
• Background in regulated or large enterprise environments (banking, automotive, telecom).