الوصف الوظيفي
You will be instrumental in designing, building, and maintaining high-performance data pipelines and infrastructure specifically tailored for AI and ML applications.
Your expertise will be critical in ensuring data quality, accessibility, and scalability for training, evaluating, and deploying advanced AI models, with a particular emphasis on the unique data requirements of LLMs and other complex AI techniques.
Client Details
A pivotal and leading government entity, dedicated to leveraging advanced technology and data to serve national interests and drive strategic objectives
Description
Design, build, and optimise scalable, robust, and efficient data pipelines for ingesting, transforming, and preparing large and complex datasets specifically for AI/ML models, including structured and unstructured data sources.Develop and manage data infrastructure components in cloud environments (e.g., Azure, AWS, GCP) ensuring security, compliance, and performance for AI workloads.Implement data governance best practices, ensuring data quality, lineage, privacy, and security for sensitive government data used in AI applications.Collaborate closely with AI Scientists and Machine Learning Engineers to understand their data needs, optimising data formats, storage, and access patterns for model training and inference, particularly for LLMs.Develop and maintain data versioning, feature stores, and model-serving data layers to support the AI/ML lifecycle.Troubleshoot and resolve data-related issues, ensuring data reliability and integrity for AI projects.
Profile
Bachelor's degree or equivalent practical experience in Computer Science, Data Engineering, Software Engineering, or a related quantitative field.Minimum of 6+ years of progressive experience in data engineering, with at least 3 years specifically focused on building data pipelines and infrastructure for AI/ML projects.Demonstrable strong experience with data engineering concepts and tools for AI, including data preparation for techniques like LLMs, natural language processing, and deep learning.Expert proficiency in at least one programming language commonly used in data engineering (e.g., Python, Scala, Java).Strong experience with big data technologies (e.g., Spark, Hadoop, Flink) and distributed data processing frameworks.Proven experience with cloud data platforms and services (e.g., Azure Data Factory, Azure Databricks, AWS Glue, Google Cloud Dataflow, BigQuery).
Job Offer
An exceptional opportunity to contribute to high-impact national initiatives within a leading government entity.
