Scrapping and Automation In ASP Dot Net

Jason06 21 Reputation points
2022-09-01T10:03:41.077+00:00

Hey There!

Basically, I want to make a website that can scrape data from other websites without using an API in the same way that cURL in PHP scrapes data from other websites. I would like to know if it is possible to do this type of programming in ASP.net . My final year project will be a ASP dot net project that focuses on automation and that I wish to do in ASP dot net.

Thanks.

ASP.NET
ASP.NET
A set of technologies in the .NET Framework for building web applications and XML web services.
3,253 questions
Accepted answer
  1. Bruce (SqlWork.com) 55,686 Reputation points
    2022-09-01T21:32:54.74+00:00
    0 comments

3 additional answers

Sort by: Most helpful
Most helpful Newest Oldest
  1. Yogi 346 Reputation points
    2022-09-01T14:26:20.043+00:00

    Check this tutorial on scrapping - How to Create a Web Scraper in ASP.NET MVC and jQuery

    0 comments
  2. Sam of Simple Samples 5,516 Reputation points
    2022-09-01T20:09:27.423+00:00

    There are two things that your application needs to do. One is to get the HTML. The other is to parse the HTML. Some people use regular expressions to parse the HTML and most experienced developers think regular expressions are a very bad solution for most requirements for HTML. See HtmlAgilityPack for the parsing part.

    I wrote the article Introduction to Web Site Scraping but it is client-side and does not use HtmlAgilityPack. HtmlAgilityPack is more modern and can be used server-side.

    0 comments
  3. Lan Huang-MSFT 25,471 Reputation points Microsoft Vendor
    2022-09-02T08:35:53.54+00:00

    Hi @Jason06 ,
    There are two ways you can scrape the web to get data:

    • Downloading web content with HTTP requests and web browsers
    • Parsing with XPath and CSS

    For more information, refer to:
    https://github.com/devbridge/web-scraping-arcticle
    Best regards,
    Lan Huang

    If the answer is the right solution, please click "Accept Answer" and kindly upvote it. If you have extra questions about this answer, please click "Comment".
    Note: Please follow the steps in our documentation to enable e-mail notifications if you want to receive the related email notification for this thread.

    0 comments