Informations principales

Description et exigences de la tâche

For our customer in Bavaria we are looking from November for a DataOps Engineer (m/f/d) - 100% remote


 


Your tasks:

• Architect and develop automated data pipelines on Azure
• Implement data ingestion processes using Azure Data Factory
• Set up and maintain CI/CD pipelines for data workflows using Azure DevOps to automate the testing, integration, and deployment of data pipeline changes
• Optimize ETL/ELT processes for performance and scalability by implementing best practices such as partitioning, indexing, and parallel processing within Azure services
• Implement comprehensive monitoring solutions using New Relic and Azure Monitor to track the performance, health, and failures of data pipelines
• Implement data governance frameworks using Azure Purview to ensure compliance with regulations such as GDPR, CCPA, and HIPAA, and internal policies related to data privacy and protection
• Design and implement Data Vault 2.0 architectures on Azure SQL Database or Azure Synapse Analytics
• Implement and manage the Lighthouse architecture to ensure efficient data access, security, and governance on Azure
• Implement comprehensive data observability solutions using Azure Data Factory, New Relic, and Azure Purview to monitor data quality, lineage, and performance
• Document and communicate data workflows, architectures, and processes
• Provide training and advice project relevant members on data infrastructure and best practices


 


Your qualifications:

• Proven experience as a DataOps Engineer or similar role, with a focus on automating data pipelines and implementing Data Vault and Lighthouse architectures
• Several years of project relevant experience
• Strong proficiency in programming languages such as Python, SQL, and familiarity with data engineering tools and frameworks on Azure
• Experience with Azure services (Azure Data Factory, Azure Synapse Analytics, Azure Databricks, Azure Cosmos DB, Azure Monitor, Azure Purview, etc.)
• Experience with relational databases (e.g., Azure SQL Database, Azure Database for PostgreSQL), NoSQL databases (e.g., Azure Cosmos DB), and cloud storage (e.g., Azure Blob Storage)
• Solid understanding of data governance, security, and compliance best practices