Submitting more applications increases your chances of landing a job.
Here’s how busy the average job seeker was last month:
Opportunities viewed
Applications submitted
Keep exploring and applying to maximize your chances!
Looking for employers with a proven track record of hiring women?
Click here to explore opportunities now!You are invited to participate in a survey designed to help researchers understand how best to match workers to the types of jobs they are searching for
Would You Be Likely to Participate?
If selected, we will contact you via email with further instructions and details about your participation.
You will receive a $7 payout for answering the survey.
The Data Engineer is responsible for designing, building, and maintaining reliable, scalable data pipelines that support analytics and reporting use cases. The role requires consistent application of enterprise frameworks, data quality controls, and security standards while owning components end to end. Data Engineers collaborate closely with platform, analytics, and business teams and are expected to continuously optimize performance, reliability, and cost efficiency of data solutions. Understanding of the Lakehouse Architecture and medallion.
A. Data Pipeline Development (Primary Responsibility)
• Design, build, and maintain scalable, reliable data ingestion and transformation pipelines across curated, analytics, and reporting layers.
• Develop ELT workflows using standardized enterprise frameworks, ensuring consistency, reusability, and alignment with platform standards.
• Assemble, process, and optimize large and complex datasets from diverse internal and external data sources.
• Implement CDC, delta-based updates for incremental processing.
• Understanding of the different data format like Parquet, Delta etc.
• Implement automated validations, reconciliations, and error handling mechanisms to ensure data accuracy and reliability.
• Support pipeline deployment, scheduling, and operational stability across non production and production environments.
• Working knowledge of streaming pipelines using Event Hub / Kafka /Structure Streaming is strongly preferred.
B. Platform, Security & Governance Alignment
• Embed data quality rules, security controls, and documentation into pipelines by design.
• Ensure datasets adhere to access controls, RBAC policies, and PII handling standards.
• Strong understanding of Unity Catalog (Databricks)
• Apply approved ingestion and transformation patterns that meet enterprise governance and audit requirements.
• Identify and remediate data quality or performance issues, escalating risks where appropriate.
C. Collaboration & Continuous Improvement
Strong working knowledge of Python for building and maintaining data pipelines
Databases
• Proficient SQL skills including complex joins, aggregations, and performance aware queries
Cloud & Data Platforms
• Hands on experience with Azure Data Factory (ADF) for orchestration and scheduling
• Practical experience using Azure Databricks in production environments
• Strong understanding of Delta Lake (Delta tables, ACID transactions, Time Travel etc.)
• Proficient in using ADLS, Azure Event Hub etc,
ETL / ELT Engineering
• Ability to design and implement ELT workflows across multiple data sources
Data Modelling
• Working knowledge of conceptual and logical data modelling for analytics use cases
Frameworks
• Proficient use of Spark / PySpark / SparkSQL for distributed processing
DevOps
• Good working knowledge of Azure DevOps/GitHub actions, pipeline deployment automation
B. Desirable / Nice to Have Skills
• FinOps and cloud cost optimization concepts.
• Azure Analysis Services
You'll no longer be considered for this role and your application will be removed from the employer's inbox.