Web Scraping and Data Engineer

Full Time
,
Chennai

Experience:1-2 Years

Job Description:

As a Web Scraping and Data Engineer, you will be responsible for developing and maintaining web scraping scripts, ensuring data quality, and supporting our data pipeline. You will work closely with our data scientists, analysts, and other stakeholders to provide clean and reliable data for various business needs.

Mandatory Skills : Python | Web scraping | Data base | Excellent Communication

Good to have: AWS / Azure | ETL Tools | containerization and orchestration tools

Key Responsibilities:

  1. Web Scraping:
    • Develop and maintain web scraping scripts using tools like BeautifulSoup, Scrapy, and Selenium.
    • Handle dynamic content and JavaScript-heavy websites using headless browsers or related techniques.
    • Implement anti-scraping bypass techniques such as rotating proxies and handling Captchas.
  2. Data Engineering:
    • Design and implement data pipelines to extract, transform, and load (ETL) web data into our data storage systems.
    • Ensure data quality by cleaning, validating, and preprocessing scraped data.
    • Maintain and optimize existing data pipelines for performance and reliability.
  3. Collaboration and Communication:
    • Work closely with data scientists and analysts to understand data requirements and deliver relevant datasets.
    • Collaborate with other engineers to integrate web scraping solutions into larger data architectures.
  4. Documentation and Reporting:
    • Document web scraping processes, data pipeline architectures, and any technical decisions made.
    • Generate reports and visualizations to communicate data insights and findings to non-technical stakeholders.

If this excites you, Please fill the Form to start the application process and we will be in touch