Share via

Scrapping and Automation In ASP Dot Net

Jason06 31 Reputation points
2022-09-01T10:03:41.077+00:00

Hey There!

Basically, I want to make a website that can scrape data from other websites without using an API in the same way that cURL in PHP scrapes data from other websites. I would like to know if it is possible to do this type of programming in ASP.net . My final year project will be a ASP dot net project that focuses on automation and that I wish to do in ASP dot net.

Thanks.

Developer technologies | ASP.NET | Other

Answer accepted by question author

  1. Bruce (SqlWork.com) 83,821 Reputation points
    2022-09-01T21:32:54.74+00:00

    1 person found this answer helpful.

4 additional answers

Sort by: Most helpful
Most helpful Newest Oldest
  1. Ahsan Khan 5 Reputation points
    2025-09-06T10:54:57.52+00:00

    Yes, it's totally possible! In fact, ASP.NET (whether you're using the older Web Forms or newer ASP.NET Core) allows you to make HTTP requests and parse HTML content, which is the basic requirement for web scraping.

    1 person found this answer helpful.
    0 comments
  2. Sam of Simple Samples 5,586 Reputation points
    2022-09-01T20:09:27.423+00:00

    There are two things that your application needs to do. One is to get the HTML. The other is to parse the HTML. Some people use regular expressions to parse the HTML and most experienced developers think regular expressions are a very bad solution for most requirements for HTML. See HtmlAgilityPack for the parsing part.

    I wrote the article Introduction to Web Site Scraping but it is client-side and does not use HtmlAgilityPack. HtmlAgilityPack is more modern and can be used server-side.

    1 person found this answer helpful.
    0 comments
  3. Yogi 356 Reputation points
    2022-09-01T14:26:20.043+00:00

    Check this tutorial on scrapping - How to Create a Web Scraper in ASP.NET MVC and jQuery

    1 person found this answer helpful.
    0 comments
  4. Lan Huang-MSFT 30,211 Reputation points Microsoft External Staff
    2022-09-02T08:35:53.54+00:00

    Hi @Jason06 ,
    There are two ways you can scrape the web to get data:

    • Downloading web content with HTTP requests and web browsers
    • Parsing with XPath and CSS

    For more information, refer to:
    https://github.com/devbridge/web-scraping-arcticle
    Best regards,
    Lan Huang

    If the answer is the right solution, please click "Accept Answer" and kindly upvote it. If you have extra questions about this answer, please click "Comment".
    Note: Please follow the steps in our documentation to enable e-mail notifications if you want to receive the related email notification for this thread.

    0 comments

Your answer

Answers can be marked as 'Accepted' by the question author and 'Recommended' by moderators, which helps users know the answer solved the author's problem.