Senior Data Engineer
Dicetek LLC
Date: 3 weeks ago
City: Abu Dhabi
Contract type: Contractor
Job Overview
The Data Engineering Lead is responsible for managing the development and support of internally built or supported ETL/ELT processes and Azure Data Services, including Azure Data Factory, Azure Databricks, Big Data, and streaming solutions. This role involves gathering business requirements, designing data models, and developing comprehensive data solutions.
JOB SCOPE
Data Engineering (ETL/ELT & Data Processing)
Essential:
The Data Engineering Lead is responsible for managing the development and support of internally built or supported ETL/ELT processes and Azure Data Services, including Azure Data Factory, Azure Databricks, Big Data, and streaming solutions. This role involves gathering business requirements, designing data models, and developing comprehensive data solutions.
JOB SCOPE
Data Engineering (ETL/ELT & Data Processing)
- Analyze, understand, and document business requirements to design data-driven solutions that align with organizational objectives.
- Expertise in ETL methodologies, Big Data, and Data Warehousing principles, including architecture design, data modeling, and performance optimization.
- Design, develop, test, optimize, and deploy ETL pipelines using Azure Data Factory (ADF), Azure Databricks, and Azure Data Lake Storage.
- Familiarity with leading ETL tools such as Talend, Informatica, and Pentaho for building scalable ETL/ELT workflows is a plus.
- Translate source-to-target mapping documents into efficient ETL processes to support data integration and transformation.
- Develop and maintain robust ETL code and stored procedures using SQL, Python, and Spark.
- Design, implement, and manage ETL processes on Microsoft Azure Cloud, with expertise in Azure Data Factory (ADF), Azure Databricks, Azure Synapse Analytics, and Azure Data Lake.
- Administer and maintain cloud-based data platforms, including Azure Cloud and Snowflake Enterprise Data Warehouse, ensuring data security, performance, and availability.
- Develop and manage Spark jobs within Azure Databricks for large-scale data processing.
- Operate in an Agile development environment, applying best practices for CI/CD using Azure DevOps, GitHub, and automation tools.
- Automate deployment workflows, manage version control, and ensure continuous integration and delivery of data solutions.
- Implement DevOps practices for monitoring, testing, and optimizing data pipelines to guarantee high system reliability and performance.
- Proactively monitor, troubleshoot, and maintain ETL components to ensure data quality, system stability, and timely data delivery.
- Optimize data workflows for performance, scalability, and cost-efficiency in cloud environments.
- Deploy machine learning (ML) models in the cloud using Azure Machine Learning and integrate them into data pipelines.
- Design and integrate APIs to enable seamless data flow across systems and applications.
- Lead, mentor, and support team members in data engineering activities, promoting knowledge sharing and best practices.
- Collaborate effectively with business stakeholders, data scientists, and IT teams to ensure that technical solutions align with business needs.
Essential:
- Bachelor’s Degree in Computer Science, Engineering, or a related field.
- 5+ years of experience in Data Engineering.
- 5+ years of ETL development experience using Talend or Azure Data Services.
- 5+ years of experience writing complex SQL queries, stored procedures, and views.
- 3+ years of hands-on experience in Python development.
- 3+ years of experience with Azure or other cloud-based solutions.
- Strong proficiency in Python and SQL for data engineering tasks.
- In-depth expertise in Azure Data Services, including:
- Azure Data Factory (ADF)
- Azure Synapse Analytics
- Advanced certifications in Azure Data Engineering (e.g., Azure Data Factory, Azure ML).
- Certification in Azure Databricks or any cloud data warehouse (e.g., Snowflake, Redshift).
- Data Science certification is an advantage.
- 3+ years of experience with Azure Databricks or similar cloud data warehousing tools.
- 2+ years of experience with Azure Data Factory (ADF).
- Experience with machine learning algorithms, leveraging Python, Azure Databricks, and Azure Machine Learning for advanced analytics.
- Strong hands-on experience with Databricks for end-to-end data processing and transformation.
- Expertise in data warehousing concepts with practical experience in ETL/ELT development, data processing, and optimization within Azure environments.
- Proven ability to deploy and manage machine learning models using Azure Machine Learning and related cloud-based solutions.
- Expertise in managing cloud-based data warehouses, particularly Snowflake and Azure Data Warehouses.
- Proficiency in designing, writing, and debugging complex SQL queries, stored procedures, views, triggers, and functions in SQL Server, Oracle, and data warehouse environments.
- Solid background in data modeling, using tools like Erwin to design efficient, scalable data architectures.
- Hands-on experience with Azure DevOps and GitHub for version control, CI/CD pipelines, and automated deployments.
- Strong understanding of Master Data Management (MDM) solutions and customer data profiling techniques.
- Excellent communication and collaboration skills with the ability to work effectively across cross-functional teams.
How to apply
To apply for this job you need to authorize on our website. If you don't have an account yet, please register.
Post a resumeSimilar jobs
Engineer, Crane Maintenance
ADNOC Group,
Abu Dhabi
15 hours ago
Job Purpose: Plans and Provide crane and lifting equipment maintenance and operation. Coordinates with Crane Management Contractor to ensure all operations and maintenance meet company requirements with regard to availability and reliability, whilst fully complying with all Company HSE Regulations, Standards and ProceduresKey Accountabilities: Job Specific AccountabilitiesActs as the Technical Authority for Maintenance and Operation of Lifting Equipment. Supports various...
Technician, Fire Services
ADNOC Group,
Abu Dhabi
18 hours ago
JOB PURPOSE:Site based tactical responder. Deputizes for Lead Fire fighters during their absence. Responds to any emergency such as anuncontrolled fire or release of potentially harmful substances, all rescues including road traffic accidents and industrialaccidents. Prevents and minimize loss of life and property damage as a result of fire, gas release incidents and rescuesituations. Carries out a range of testing,...
Front Office Manager
Dusit Thani Dubai,
Abu Dhabi
19 hours ago
Job DescriptionPRIMARY RESPONSIBILITIES Maximizes occupancy and average rate at all time. Closely coordinates room availability with the Director of Sales. Maintains a good relationship with travel agents and tour operators. Coordinates with in-house Engineering and Housekeeping to ensure that there is a maximum number of rooms available for sales at a given time. Answers inquiries pertaining to hotel policies and...
