LeadDataEngineer

Key Responsibilities Data Engineering & Platform Architecture Architect, design, and govern scalable batch and streaming data platforms using Python (strong OOP design) , Apache Spark (PySpark, Spark SQL) , and Azure Databricks . Define end-to-end reference architectures for real-time and near-real-time streaming solutions using Apache Kafka or Apache Flink , with Spark Structured Streaming or Apache Flink , ensuring: Event-time correctness Exactly-once processing guarantees Stateful stream processing and checkpointing High availability and fault tolerance Own the architectural design of Medallion architecture (Bronze, Silver, Gold) on ADLS Gen2 with Delta Lake , including data lifecycle, retention, and cost optimization strategies. Lead data governance architecture , defining standards for security, access control, lineage, and metadata using Databricks Unity Catalog . Design domain-oriented, scalable data products aligned with Data Mesh and cloud-native architecture principles . Define integration patterns for data services, streaming producers/consumers, and microservices-based architectures . Architect and oversee deployment of data and streaming workloads on AKS , including container strategy, scaling, resiliency, and networking. Define and enforce performance, scalability, and reliability standards for Spark and streaming workloads (partitioning, Z-ordering, state tuning, caching). Establish data quality, validation, and schema evolution standards for both batch and streaming pipelines. Design secure-by-default architectures using Azure IAM, RBAC, Key Vault, private endpoints, VNet integration, and network isolation . Lead CI/CD and DevOps architecture using GitLab , enabling automated testing, deployment, rollback, and environment promotion. Define Infrastructure as Code architecture using Terraform and Pulumi for repeatable, auditable deployments. Establish observability architecture (monitoring, logging, alerting) across Databricks, AKS, streaming platforms