SeniorExpert

We are seeking a highly experienced Azure Data Engineer with 10+ years of hands-on experience, specializing in Azure Data Factory and Azure Databricks, to design, build, and manage scalable, secure, and high-performance data platforms on Microsoft Azure. The ideal candidate will be deeply skilled in ETL/ELT engineering, distributed data processing, and cloud-native data architecture, and will play a key role in transforming complex data from multiple sources into reliable, analytics ready datasets that support business and operational decision-making. Key Responsibilities * Design, develop, and maintain robust ETL/ELT pipelines using Azure Data Factory and Azure Databricks. * Ingest and integrate data from on-premises systems, cloud platforms, APIs, and flat files into Azure-based data platforms. * Build and optimize data processing frameworks using Databricks (PySpark / Spark SQL) for large-scale data transformation. * Develop and manage data models and curated datasets in Azure Data Lake Storage Gen2, Azure SQL Database, and Azure Synapse Analytics. * Architect scalable, secure, and cost-efficient data storage solutions using Azure-native services. * Optimize Databricks clusters and ADF pipelines for performance, reliability, and cost efficiency. * Monitor, troubleshoot, and resolve pipeline and data quality issues using Azure Monitor, Log Analytics, and native service diagnostics. * Collaborate closely with business stakeholders and downstream consumers to understand data requirements and delivery expectations. * Translate business and analytical needs into technical data engineering solutions aligned with enterprise architecture standards. * Implement data security, encryption, and role-based access control (RBAC) across the data platform. * Ensure data pipelines and storage comply with organizational policies and regulatory requirements (e.g., GDPR, HIPAA where applicable). * Create and maintain technical documentation for data flows, pipeline logic, transforma