Senior Data Engineer
Dicetek LLC
Date: 1 day ago
City: Abu Dhabi
Contract type: Contractor
Job Overview
The Data Engineering Lead is responsible for managing the development and support of internally built or supported ETL/ELT processes and Azure Data Services, including Azure Data Factory, Azure Databricks, Big Data, and streaming solutions. This role involves gathering business requirements, designing data models, and developing comprehensive data solutions.
JOB SCOPE
Data Engineering (ETL/ELT & Data Processing)
Essential:
The Data Engineering Lead is responsible for managing the development and support of internally built or supported ETL/ELT processes and Azure Data Services, including Azure Data Factory, Azure Databricks, Big Data, and streaming solutions. This role involves gathering business requirements, designing data models, and developing comprehensive data solutions.
JOB SCOPE
Data Engineering (ETL/ELT & Data Processing)
- Analyze, understand, and document business requirements to design data-driven solutions that align with organizational objectives.
- Expertise in ETL methodologies, Big Data, and Data Warehousing principles, including architecture design, data modeling, and performance optimization.
- Design, develop, test, optimize, and deploy ETL pipelines using Azure Data Factory (ADF), Azure Databricks, and Azure Data Lake Storage.
- Familiarity with leading ETL tools such as Talend, Informatica, and Pentaho for building scalable ETL/ELT workflows is a plus.
- Translate source-to-target mapping documents into efficient ETL processes to support data integration and transformation.
- Develop and maintain robust ETL code and stored procedures using SQL, Python, and Spark.
- Design, implement, and manage ETL processes on Microsoft Azure Cloud, with expertise in Azure Data Factory (ADF), Azure Databricks, Azure Synapse Analytics, and Azure Data Lake.
- Administer and maintain cloud-based data platforms, including Azure Cloud and Snowflake Enterprise Data Warehouse, ensuring data security, performance, and availability.
- Develop and manage Spark jobs within Azure Databricks for large-scale data processing.
- Operate in an Agile development environment, applying best practices for CI/CD using Azure DevOps, GitHub, and automation tools.
- Automate deployment workflows, manage version control, and ensure continuous integration and delivery of data solutions.
- Implement DevOps practices for monitoring, testing, and optimizing data pipelines to guarantee high system reliability and performance.
- Proactively monitor, troubleshoot, and maintain ETL components to ensure data quality, system stability, and timely data delivery.
- Optimize data workflows for performance, scalability, and cost-efficiency in cloud environments.
- Deploy machine learning (ML) models in the cloud using Azure Machine Learning and integrate them into data pipelines.
- Design and integrate APIs to enable seamless data flow across systems and applications.
- Lead, mentor, and support team members in data engineering activities, promoting knowledge sharing and best practices.
- Collaborate effectively with business stakeholders, data scientists, and IT teams to ensure that technical solutions align with business needs.
Essential:
- Bachelor’s Degree in Computer Science, Engineering, or a related field.
- 5+ years of experience in Data Engineering.
- 5+ years of ETL development experience using Talend or Azure Data Services.
- 5+ years of experience writing complex SQL queries, stored procedures, and views.
- 3+ years of hands-on experience in Python development.
- 3+ years of experience with Azure or other cloud-based solutions.
- Strong proficiency in Python and SQL for data engineering tasks.
- In-depth expertise in Azure Data Services, including:
- Azure Data Factory (ADF)
- Azure Synapse Analytics
- Advanced certifications in Azure Data Engineering (e.g., Azure Data Factory, Azure ML).
- Certification in Azure Databricks or any cloud data warehouse (e.g., Snowflake, Redshift).
- Data Science certification is an advantage.
- 3+ years of experience with Azure Databricks or similar cloud data warehousing tools.
- 2+ years of experience with Azure Data Factory (ADF).
- Experience with machine learning algorithms, leveraging Python, Azure Databricks, and Azure Machine Learning for advanced analytics.
- Strong hands-on experience with Databricks for end-to-end data processing and transformation.
- Expertise in data warehousing concepts with practical experience in ETL/ELT development, data processing, and optimization within Azure environments.
- Proven ability to deploy and manage machine learning models using Azure Machine Learning and related cloud-based solutions.
- Expertise in managing cloud-based data warehouses, particularly Snowflake and Azure Data Warehouses.
- Proficiency in designing, writing, and debugging complex SQL queries, stored procedures, views, triggers, and functions in SQL Server, Oracle, and data warehouse environments.
- Solid background in data modeling, using tools like Erwin to design efficient, scalable data architectures.
- Hands-on experience with Azure DevOps and GitHub for version control, CI/CD pipelines, and automated deployments.
- Strong understanding of Master Data Management (MDM) solutions and customer data profiling techniques.
- Excellent communication and collaboration skills with the ability to work effectively across cross-functional teams.
How to apply
To apply for this job you need to authorize on our website. If you don't have an account yet, please register.
Post a resumeSimilar jobs
Project Engr II
Honeywell,
Abu Dhabi
16 minutes ago
Job DescriptionJob Title: Project Lead Engineer IIExperience Required: 6–8 YearsJob PurposeTo provide technical leadership in the design and execution of industrial automation projects focused on fixed gas detection systems, Fire Alarm Control Panel (Addressable), Fire Extinguishing Release Panel and Control & Safety systems. The role involves end-to-end engineering responsibility from technical documentation to system design and FAT/SAT support, with emphasis...
Sales Executive - HSE Training
TÜV Rheinland Group,
Abu Dhabi
2 hours ago
Industrial Services & CybersecurityAt TÜV Rheinland, in the Industry Service & Cybersecurity business unit, we are looking for talents who are ready to help shape the future of the industry and accompany technological progress. Our focus is on digitalization and networking in the industry, as well as the further international expansion of our services in the field of renewable energy...
Inspector - Civil
AtkinsRéalis,
Abu Dhabi
8 hours ago
AtkinsRéalis is looking for an Inspector - Civil (Site Operation) in UAE.About AtkinsRéalisCreated by the integration of long-standing organizations dating back to 1911, AtkinsRéalis is a world-class engineering services and nuclear company dedicated to engineering a better future for our planet and its people. We create sustainable solutions that connect people, data and technology to transform the world's infrastructure and...
