Thursday, May 28, 2009

Automated Web UI Testing Adventures with WatiN and TFS

So...I want to get some automated UI testing in place for a web application. I'm using Visual Studio Team System and Team Foundation Server.
After a bit of research, I decide to give WatiN (http://watin.sourceforge.net/) a try. WatiN is a .Net based version of WatiR, which provides an object model for interacting with an instance of Internet Explorer.

Getting a test working on my local development environment was pretty straight forward. Here's what my first test looks like:

public class UITests
{
IE ie; // The IE class is defined by WatiN for interacting with Internet Explorer

public UITests()
{
ie = new IE("http://localhost/webapp");
}
public void TestAdminMenuIsPresent()
{
var result = ie.Element(Find.ByText("Admin"));
Assert.IsNotNull(result, "Admin menu was not found");
Assert.AreEqual("link ", result.ClassName, "Admin menu was not found");
}
}

So the next challenge (and its not a small one) is to get this test running on the TFS build server.
The first step towards achieving that is to get the web site deployed to the build server's local IIS. First I tried using the Visual Studio Web Deployment Project (support for this project type is available as a download). That turned out to be troublesome and not really what I wanted. All I really needed was a simple step to copy the files that make up the web app to a directory under the IIS web root (I manually configured the virtual directory in IIS - this doesn't need to be done for each build). With help from the Google I discovered the _CopyWebApplication target which will copy the application files to a destination directory. I encountered a bug with this target though in that it doesn’t copy linked files in the project. I found a blog posting about how to correct this and added a CopyLinkedContentFiles target to copy those also. The TFSBuild project for my web application now build these targets:
<Targets>Build;_CopyWebApplication;CopyLinkedContentFiles</Targets>
and it sets the following properties to cause the files to be deployed to the IIS webroot on the build server:
<Properties>OutputPath=bin\release;WebProjectOutputDir=c:\inetpub\wwwroot\webapp\;OutDir=c:\inetpub\wwwroot\webapp\bin\</Properties>


OK so then to get the test working on the build server...
Initially I was getting an error saying that the WatiN TimeoutException type couldn’t be loaded. Copying the Watin.core.dll file to the C:\Program Files\Microsoft Visual Studio 9.0\Common7\IDE directory on the build server fixed this (although I’m not sure why this was necessary/why the DLL wasn’t being deployed or found).
Then the timeout exception was being properly thrown. So, I set the WatiN timeout to 3 minutes with the following:
Settings.WaitForCompleteTimeOut = 180; // 3 minutes

Then I was hitting the VS Test default timeout, so I set the timeout on the test to 5 minutes:
[Timeout(300000), TestMethod]

Which didn’t seem to be working, so I set the per-test timeout to 5 minutes by creating a .testrunconfig file that specified this and pointing the TFS build project at this file.
Which got me back to getting the WatiN timeout:
WatiN.Core.Exceptions.TimeoutException: Timeout while Internet Explorer busy.
Based on similar reports I’ve seen it seems it is having trouble invoking IE. The reported solution was to disable “Protected Mode” in IE. I couldn’t see any way of doing this though when IE is being run under the TFSService account. But I did discover that Windows Server 2008 Server Manager has an option to turn off “IE Enhanced Security Configuration” for Administrators. So I tried turning that off.
That seemed to do the trick, leading me to the next error:
System.Threading.ThreadStateException: The CurrentThread needs to have it's ApartmentState set to ApartmentState.STA to be able to automate Internet Explorer.
OK, so I tried adding this call in the constructor of the test class:
Thread.CurrentThread.SetApartmentState(ApartmentState.STA);

But that resulted in the following error:
Error: System.InvalidOperationException: Failed to set the specified COM apartment state..
Hmmm…it turns out that you can't change the apartment state of a thread that is already running. (So what use is the SetApartmentState method?) You have to set it before the thread starts – e.g. by using the [STAThread] attribute on the entry point method of the thread. I don’t think there’s any way of doing that with unit tests though as they are invoked by the VSTestHost process. One solution to this I read of was to do the following:

Thread thread = new Thread(STAInit);
thread.SetApartmentState(ApartmentState.STA);
thread.Start();
thread.Join();

So, I put that into the test class constructor and marked the method that the thread starts with (STAInit) with the [STAThread] attribute. Then I tried running the test again, locally. The result this time:
System.Runtime.InteropServices.InvalidComObjectException: COM object that has been separated from its underlying RCW cannot be used.
It seems that the handle to the IE instance, which was created in my thread initializer (STAInit) was lost before the test method was called. OK, I’ll try creating the WatiN IE object in the test method itself… Yep, that worked locally at least. Now to try it on the TFS build server…
Ugh…the test is still not running in STA apartment state. OK, I’ll try having the test itself create an STA thread.
Argh, although it works fine locally, on the server it is back to timing out (exceeded the 5 minute limit I allowed for each test).
Hmmm...switching to the build server I see a debugger dialog has popped up letting me know that VSTestHost.exe has encountered an unhandled System.Threading.ThreadStateException.
Apparently WatiN will throw this if the thread's apartment state is not set to STA. Even though I am setting that on the thread I'm calling WatiN from. I wonder why this exception being thrown from a test case is causing VSTestHost to crash now?