Job description
Position: Web Data Engineer (Data Collection & Automation) About Our Client: Our client is a renowned tech company focused on leveraging data to drive insights and decision-making across various industries.
They are seeking a skilled Web Data Engineer to develop and automate data collection processes that support their data analytics initiatives.
This role is crucial for ensuring that accurate, high-quality data is readily available for analysis.
Key Responsibilities: Design, develop, and maintain scalable data collection architectures and workflows.
Automate data extraction processes from various web sources, APIs, and data feeds.
Implement web scraping strategies while ensuring compliance with legal and ethical standards.
Collaborate with data analysts and data scientists to understand data requirements and ensure that the data collected meets their needs.
Perform data cleansing, transformation, and validation to ensure data integrity and accuracy.
Monitor and troubleshoot data pipelines, addressing any issues or discrepancies in a timely manner.
Document processes, workflows, and data sources for visibility and reproducibility.
Stay updated on industry trends in data collection techniques, tools, and technologies.
Participate in code reviews and contribute to continuous improvement in data engineering practices.
Full Time, Freelance and Contract roles available Qualifications & Skills: Bachelor’s degree in Computer Science, Data Science, or a related field.
Proven experience as a Data Engineer, Data Scientist, or in a similar role focused on data collection and automation.
Strong programming skills in languages such as Python, Java, or Scala.
Experience with web scraping tools and technologies such as Beautiful Soup, Scrapy, Selenium, or similar frameworks.
Proficiency in SQL and experience with databases like MySQL, PostgreSQL, or NoSQL solutions.
Familiarity with cloud platforms (AWS, Google Cloud, or Azure) and data processing tools (Apache Spark, Apache Airflow, etc.
). Understanding of data governance, data quality, and data management principles.
Excellent analytical and problem-solving skills, with a keen attention to detail.
Strong communication skills to collaborate effectively with technical and non-technical teams.
Preferred Qualifications: Experience with ETL processes and data integration tools.
Knowledge of data visualization tools (Tableau, Power BI, etc.
) is a plus.
Familiarity with version control systems such as Git.
Experience working in Agile development environments.
This job post has been translated by AI and may contain minor differences or errors.
