Accepted answer

getting html code from a website. you can use code like this:

string urladdress = "";

httpwebrequest request = (httpwebrequest)webrequest.create(urladdress);
httpwebresponse response = (httpwebresponse)request.getresponse();

if (response.statuscode == httpstatuscode.ok)
    stream receivestream = response.getresponsestream();
    streamreader readstream = null;
    if (string.isnullorwhitespace(response.characterset))
        readstream = new streamreader(receivestream);
        readstream = new streamreader(receivestream,
    string data = readstream.readtoend();

this will give you the returned html from the website. but find text via linq is not that easy. perhaps it is better to use regular expression but that does not play well with html.


try this solution. it works fine.

        string url = textbox1.text;
        httpwebrequest request = (httpwebrequest)webrequest.create(url);
        httpwebresponse response = (httpwebresponse)request.getresponse();
        streamreader sr = new streamreader(response.getresponsestream());
        htmlagilitypack.htmldocument doc = new htmlagilitypack.htmldocument();
        var atags = doc.documentnode.selectnodes("//a");
        int counter = 1;
        if (atags != null)
            foreach (var atag in atags)
                richtextbox1.text +=  atag.innerhtml +  "\n" ;
        catch (exception ex)
  "failed to retrieve related keywords." + ex);


you can use webclient to download the html for any url. once you have the html, you can use a third-party library like htmlagilitypack to lookup values in the html as in below code -

public static string getinnerhtmlfromdiv(string url)
        string html;
        using (var wc = new webclient())
            html = wc.downloadstring(url);
        var doc = new htmlagilitypack.htmldocument();
        htmlnode element = doc.documentnode.selectsinglenode("//div[@id='<div id here>']");
        if (element != null)
            return element.innerhtml.tostring();
        return null;            


here's an example of using the httpwebrequest class to fetch a url

private void buttonl_click(object sender, eventargs e) 
    string url = textbox_url.text;
    httpwebrequest request = (httpwebrequest) webrequest.create(url); 
    httpwebresponse response = (httpwebresponse) request.getresponse(); 
    streamreader sr = new streamreader(response.getresponsestream()); 
    richtextbox1.text = sr.readtoend(); 


i am using anglesharp and have been very satisfied with it.

here is a simple example how to fetch a page:

var config = configuration.default.withdefaultloader();
var document = await"");

and now you have a web page in document variable. then you can easily access it by linq or other methods. for example if you want to get a string value from a html table:

var somestringvalue = document.all.where(m =>
        m.localname == "td" &&
        m.hasattribute("class") &&

to use css selectors please see anglesharp examples.


best thing to use is htmlagilitypack. you can also look into using fizzler or csquery depending on your needs for selecting the elements from the retrieved page. using linq or regukar expressions is just to error prone, especially when the html can be malformed, missing closing tags, have nested child elements etc.

you need to stream the page into an htmldocument object and then select your required element.

// call the page and get the generated html
var doc = new htmlagilitypack.htmldocument();
htmlagilitypack.htmlnode.elementsflags["br"] = htmlagilitypack.htmlelementflag.empty;
doc.optionwriteemptynodes = true;

    var webrequest = httpwebrequest.create(pageurl);
    stream stream = webrequest.getresponse().getresponsestream();
catch (system.uriformatexception uex)
    log.fatal("there was an error in the format of the url: " + itemurl, uex);
catch ( wex)
    log.fatal("there was an error connecting to the url: " + itemurl, wex);

//get the div by id and then get the inner text 
string testdivselector = "//div[@id='test']";
var divstring = doc.documentnode.selectsinglenode(testdivselector).innerhtml.tostring();

[edit] actually, scrap that. the simplest method is to use fizzlerex, an updated jquery/css3-selectors implementation of the original fizzler project.

code sample directly from their site:

using htmlagilitypack;

//get the page
var web = new htmlweb();
var document = web.load("");
var page = document.documentnode;

//loop through all div tags with item css class
foreach(var item in page.queryselectorall("div.item"))
    var title = item.queryselector("h3:not(.share)").innertext;
    var date = datetime.parse(item.queryselector("span:eq(2)").innertext);
    var description = item.queryselector("span:has(b)").innerhtml;

i don't think it can get any simpler than that.


better you can use the webclient class to simplify your task:


using (webclient client = new webclient())
    string htmlcode = client.downloadstring("");

Related Query