Getting HTML DOM from URL...

I came across many of the sites and forums with topic reading similar to “How can I get HtmlDocument from a URL”. There were many forums which had suggested some good ideas on it. One of them was using the AxWebBrowser (MS WebBrowser Active X) component. I also got curious about the problem. I googled a lot and found out that there was a API createDocumentFromUrl() in the IHTMLDocument2 interface of mshtml 4.0.

After googling a lot on this API, I learned that there was some issues with the use of this API. Here are some standerd issues which developers might have came accross.

1. VB .NET implementation runs ok, but not the C# one. 

2. AccessViolation : Attempted to read or write protected memory.

3. ReadyState never changes from “loading”.

Here’s a solution to all the above things.

// code starts here
// Project needs reference to mshtml 4 
using System.Runtime.InteropServices;
using System.Runtime.InteropServices.ComTypes;
 [ComImport(), Guid("0000010c-0000-0000-C000-000000000046"), InterfaceType(ComInterfaceType.InterfaceIsIUnknown)]
interface IPersist
{
void GetClassID(Guid pClassId);
}
[ComImport(), Guid("7FD52380-4E07-101B-AE2D-08002B2EC713"), InterfaceType(ComInterfaceType.InterfaceIsIUnknown)]
interface IPersistStreamInit : IPersist
{
new void GetClassID([In,Out] ref Guid pClassId);
[return : MarshalAs(UnmanagedType.I4)]
[PreserveSig()]
int IsDirty();
[return : MarshalAs(UnmanagedType.I4)]
[PreserveSig()]
void Load(UCOMIStream pStm); //System.Runtime.InteropServices.ComTypes.IStream
[return: MarshalAs(UnmanagedType.I4)]
[PreserveSig()]
void Save(UCOMIStream pStm, [In,MarshalAs(UnmanagedType.Bool)] bool fClearDirty);//System.Runtime.InteropServices.ComTypes.IStream
void GetMaxSize([Out]long pCbSize);
[return: MarshalAs(UnmanagedType.I4)]
[PreserveSig()]
void InitNew();
}
//This is the COM interops which will be helpful in the AccessViolation issue  //Heres a function which takes URL as the input parameter and prepares HTMLDocument object from it.  private string GetHTML(string url)
{
mshtml.HTMLDocumentClass htmldoc;
htmldoc = new mshtml.HTMLDocumentClass();
mshtml.IHTMLDocument2 htmldoc2;
mshtml.IHTMLDocument3 htmldoc3;
HTMLDocument doc2 = new HTMLDocument();
// This is ver important part of the code
//If not done it raises exception.
//[AccessViolationOccured: Attempted to read or write protected memory. This is often an indication that other memory is corrupt.]
IPersistStreamInit ips = (IPersistStreamInit)htmldoc;
ips.InitNew();
htmldoc2 = (mshtml.IHTMLDocument2)htmldoc.createDocumentFromUrl(url, null);
while (htmldoc2.readyState != “complete”)
{
//This is also a important part, without this DoEvents() appz hangs on to the “loading”
Application.DoEvents();
}
htmldoc3 = (mshtml.IHTMLDocument3)htmldoc2;
return htmldoc3.documentElement.innerHTML;
}
//Code ends here

Depending on your need you can modify the function. The topic helped me a lot in understanding the MSHTML library and a bit of COM. As a .NET developer I’m not much awared of the COM technology.If anyone finds bugs in the above code then just post it in the comments section.

-Bugs! 

About these ads