SeniorSoftwareEngineer(SeniorDataInfrastructureEngineer)

Senior Data Infrastructure Engineer Job Description * Manage and optimize Databricks infrastructure, including cluster lifecycle management, job scheduling, Unity Catalog administration, user permissions, and cost monitoring. * Manage Airflow clusters deployed on Kubernetes via Helm charts, including upgrades, configuration management, and integration with ArgoCD and Vault. * Write and maintain Terraform modules to provision and manage cloud infrastructure in a scalable, reliable, and secure manner. * Develop and maintain data pipelines and workflows using Python, PySpark, and dbt to support data processing and analytics use cases. * Use GitHub Copilot with custom skills and agents to improve team productivity, automate repetitive development tasks, and accelerate code reviews and documentation. * Communicate with stakeholders across the organization to understand their data needs and translate them into infrastructure and pipeline solutions. * Collaborate with engineering teams to ensure data quality and consistency across the organization. * Review and approve infrastructure and pipeline changes from team members, applying runbooks and automation components. * Participate in on-call rotation for data infrastructure and pipeline incidents. You will be expected to take all necessary actions to resolve issues and restore services to a normal state. * Review and maintain technical documentation, keeping it up to date with current system state and runbooks. ## Qualifications * 5+ years of professional software engineering experience. * 5+ years of experience with AWS and Kubernetes. * 3+ years managing Databricks infrastructure, including cluster management, job scheduling, user management, cost monitoring, and performance optimization. * 3+ years of Python development for data pipelines and workflows, including PySpark and dbt. * 2+ years managing Airflow infrastructure. * 2+ years writing and maintaining Terraform modules for cloud infrastructure. * Proficiency in us