Windows Azure: Automated UI testing using the power of cloud

Problem Statement

Automated UI testing needs to be performed from different parts of the world.

The tests can validate various UI behaviors from different geo locations e.g. behavior of JavaScript which makes service calls.

Background

As we have seen in my previous post on Performance testing using the power of cloud, it is possible to install test agents around the globe and run Load Tests from the agents. The same setup can be used to run unit tests or web tests by distributing them across different agents.

Challenge

The challenging part is to run UI tests (e.g. Coded UI) from the agents in cloud. This is because we will always need to have an active desktop session for the tests to interact with the UI. One option would be to have dedicated hardware on premise from where active remote desktop session is always maintained for all the agents in the cloud. The other option is to tweak the configuration a little as described below without requiring any additional hardware.

Solution

Perform the same steps which were done for the setting up the environment to run load tests i.e. Installing Test Controller on Premise and Setting up Windows Azure Connect.

Now below are the changes that the needed to specifically run Automated UI Tests

• Instead of having just one user on the Azure VM’s, we will need to have two users. One of them is used for Remote Desktop and the other one is used to Run Tests.
• The Remote Desktop user needs to be specified in the configuration file (.cscfg)

<Setting name="Microsoft.WindowsAzure.Plugins.RemoteAccess.AccountUsername" value="remoteuser" />

•  The Test Agent user needs to be created in setup.cmd (This user name and password should be the same as the one configured on Test Controller)

REM Create user vstestagent as an administrator

net user vstestagent <<password>> /expires:never /add

net localgroup administrators vstestagent /Add

•  Deploy the worker role and add it to the Local Endpoint Group for Windows Azure Connect as before.
• Now Remote Desktop to the Azure VM from the Azure Portal using the Remote Desktop user.
• Once connected, get the machine name of the VM from Computer->Properties.
• Go to Run and type mstsc and connect to the same VM using the machine name. While connecting this time, type the credentials of the Test Agent user.
• Configure the test agent with the test controller in Interactive mode.
• 
• Finally, a script needs to be run to turn off IE ESC, set default IE settings, turn off Phishing filter and disable Auto Lock. A command file can be added which does the following:
REM Configure IE ESC to Off
REG ADD "HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Active Setup\Installed Components\{A509B1A7-37EF-4b3f-8CFC-4F3A74704073}" /v "IsInstalled" /t REG_DWORD /d 0 /f
REG ADD "HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Active Setup\Installed Components\{A509B1A8-37EF-4b3f-8CFC-4F3A74704073}" /v "IsInstalled" /t REG_DWORD /d 0 /f
Rundll32 iesetup.dll, IEHardenLMSettings
Rundll32 iesetup.dll, IEHardenUser
Rundll32 iesetup.dll, IEHardenAdmin
REG DELETE "HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Active Setup\Installed Components\{A509B1A7-37EF-4b3f-8CFC-4F3A74704073}" /f /va
REG DELETE "HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Active Setup\Installed Components\{A509B1A8-37EF-4b3f-8CFC-4F3A74704073}" /f /va
REM Set Default IE Settings
REG DELETE "HKEY_CURRENT_USER\Software\Microsoft\Internet Explorer\Main" /v "First Home Page" /f
REG ADD "HKEY_CURRENT_USER\Software\Microsoft\Internet Explorer\Main" /v "Default_Page_URL" /t REG_SZ /d "about:blank" /f
REG ADD "HKEY_CURRENT_USER\Software\Microsoft\Internet Explorer\Main" /v "Start Page" /t REG_SZ /d "about:blank" /f
REM Disable Phishing Filter
REG ADD "HKEY_CURRENT_USER\Software\Microsoft\Internet Explorer\PhishingFilter" /v "ShownVerifyBalloon" /t REG_DWORD /d 3 /f
REG ADD "HKEY_CURRENT_USER\Software\Microsoft\Internet Explorer\PhishingFilter" /v "Enabled" /t REG_DWORD /d 0 /f
REM Disable Auto Lock
REG ADD "HKEY_CURRENT_USER\Software\Microsoft\Windows\CurrentVersion\Policies\System" /v "DisableLockWorkstation" /t REG_DWORD /d 1 /f
• Now don’t minimize any remote desktop windows and simply close the remote desktop window of the Remote Desktop user. This way the Remote Desktop window of the Test Agent user will be always active and the UI Tests can interact with the desktop.
Result
Now you can run your tests using the Test Controller. The Agent running as an interactive Process should be shown with an asterisk (*)

The Automated UI Test runs on the Test Agent

To confirm that the desktop doesn’t get locked, wait for 10-15 minutes, rerun the automated UI test and verify that it passes.

Azure Table Storage - One of the request inputs is not valid

I was getting this error while getting the count of the total number of entities for a given Partition Key in a Azure Storage Table:

System.Data.Services.Client.DataServiceQueryException: An error occurred while processing this request. ---> System.Data.Services.Client.DataServiceClientException: <?xml version="1.0" encoding="utf-8" standalone="yes"?>
<error xmlns="http://schemas.microsoft.com/ado/2007/08/dataservices/metadata">
  <code>InvalidInput</code>
  <message xml:lang="en-US">One of the request inputs is not valid.RequestId:xxxTime:xxx</message>
</error>

This was occuring when I was trying to get the count like this:
int count = (from e in tableContext.CreateQuery<MyEntity>(_myTable)
                     where e.PartitionKey == interalId
                     select e).Count();

I was hoping for it to work since I was able to get the entire List using the same approach:
(from e in tableContext.CreateQuery<MyEntity>(_myTable)
                     where e.PartitionKey == interalId
                     select e).ToList();

However, I didn't want to first get the entire List and then take the count which would be lot of size overhead over the network. Finally I was able to get the count by first creating a query and then calling Execute:
CloudTableQuery<MyEntity> query = (from e in tableContext.CreateQuery<MyEntity>(_myTable)
                     where e.PartitionKey == interalId
                     select e).AsTableServiceQuery();
int count = query.Execute().Count();

Azure Table Storage - One of the request inputs is out of range

I was getting this error while saving an entity in Azure Table Storage using the method SaveChangesWithRetries() in Microsoft.WindowsAzure.StorageClient.TableServiceContext

System.Data.Services.Client.DataServiceClientException: <?xml version="1.0" encoding="utf-8" standalone="yes"?>
<error xmlns="http://schemas.microsoft.com/ado/2007/08/dataservices/metadata">
  <code>OutOfRangeInput</code>
  <message xml:lang="en-US">One of the request inputs is out of range.
RequestId:xxxTime:xxx</message>
</error>

I was setting the Primary Key and the Row Key correctly so wasn't sure about the root cause of the error.
Then I found that one of the properties of the entity was of Type DateTime which was not being set.
After I set the property to a value, say DateTime.Now, the entity got saved successfully.

VS 2010 – Generating a TRX file for a list of Tests

I came across a situation where I needed to generate a TRX file dynamically for a set of tests. This TRX would then to be used to rerun the set of tests.

To create a utility for this, I needed 3 pre-requisites:

·         List of test names

·         A TRX file template with one test

·         VSMDI file which has the test case details

The TRX template looks like this:

<?xml version="1.0" encoding="UTF-8"?>

<TestRun id="75ef9a27-b8ce-4c81-a2bb-fb3e26088e32" name="xxx" runUser="xxx" xmlns="http://microsoft.com/schemas/VisualStudio/TeamTest/2010">

          <TestSettings name="Local" id="3402c4f6-3b10-4472-9f2c-14a883c33075">

                   <Description>These are default test settings for a local test run.</Description>

                   <Deployment enabled="false" runDeploymentRoot="anujc_ANUJDEV1 2012-10-09 17_43_06">

                   </Deployment>

                   <Execution hostProcessPlatform="MSIL">

                             <TestTypeSpecific>

                                      <UnitTestRunConfig testTypeId="13cdc9d9-ddb5-4fa4-a97d-d965ccfc6d4b">

                                                <AssemblyResolution applicationBaseDirectory="xxx ">

                                                          <TestDirectory useLoadContext="true" />

                                                          <RuntimeResolution>

                                                                   <Directory path=" xxx " includeSubDirectories="true" />

                                                          </RuntimeResolution>

                                                </AssemblyResolution>

                                      </UnitTestRunConfig>

                             </TestTypeSpecific>

                             <AgentRule name="LocalMachineDefaultRole">

                             </AgentRule>

                   </Execution>

          </TestSettings>

          <Times creation="2012-10-09T17:43:06.8818053-07:00" queuing="2012-10-09T17:43:08.3979569-07:00" start="2012-10-09T17:43:08.5789750-07:00" finish="2012-10-09T18:01:43.5194579-07:00" />

          <ResultSummary outcome="Failed">

                   <Counters total="1" executed="1" passed="0" error="0" failed="0" timeout="0" aborted="0" inconclusive="0" passedButRunAborted="0" notRunnable="0" notExecuted="0" disconnected="0" warning="0" completed="1" inProgress="0" pending="0" />

          </ResultSummary>

          <TestDefinitions>

                   <UnitTest name="DefaultTestName" storage=" xxx">

                             <Description>DefaultDescription</Description>

                             <Execution id="e58026f9-1eb9-4f96-8fc9-e672dce1e6c7" />

                             <Properties>

                                      <Property>

                                                <Key>JobID</Key>

                                                <Value>133069</Value>

                                      </Property>

                             </Properties>

                             <TestMethod codeBase=" xxx" adapterTypeName="Microsoft.VisualStudio.TestTools.TestTypes.Unit.UnitTestAdapter, Microsoft.VisualStudio.QualityTools.Tips.UnitTest.Adapter, Version=10.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a" className=" xxx, Federation4.5, Version=0.0.0.0, Culture=neutral, PublicKeyToken=null" name="DefaultTestName" />

                   </UnitTest>

          </TestDefinitions>

          <TestLists>

                   <TestList name="All Loaded Results" id="19431567-8539-422a-85d7-44ee4e166bda" />

          </TestLists>

          <TestEntries>

                   <TestEntry testId="5099c33a-7e6b-4dff-8872-c0b6f614bbed" executionId="e58026f9-1eb9-4f96-8fc9-e672dce1e6c7" testListId="19431567-8539-422a-85d7-44ee4e166bda" />

          </TestEntries>

          <Results>

                   <UnitTestResult executionId="e58026f9-1eb9-4f96-8fc9-e672dce1e6c7" testId="5099c33a-7e6b-4dff-8872-c0b6f614bbed" testName="DefaultTestName" computerName=" xxx" duration="00:00:15.4887422" startTime="2012-10-09T17:43:08.7109882-07:00" endTime="2012-10-09T17:43:25.0076177-07:00" testType="13cdc9d9-ddb5-4fa4-a97d-d965ccfc6d4b" outcome="Completed" testListId="19431567-8539-422a-85d7-44ee4e166bda" relativeResultsDirectory="e58026f9-1eb9-4f96-8fc9-e672dce1e6c7">

                             <Output>

                                      <StdOut>Fake Log</StdOut>

                             </Output>

                   </UnitTestResult>

          </Results>

</TestRun>

Now I created one method which reads the test properties from the VSMDI

static string GetTestcaseIdFromVsmdi(string testcase)

        {

            XmlDocument doc = new XmlDocument();

            doc.Load(oldVsmdi);

            XmlNamespaceManager namespaceManager = new XmlNamespaceManager(new NameTable());

            namespaceManager.AddNamespace("ns", "http://microsoft.com/schemas/VisualStudio/TeamTest/2010");

 

            XmlNode testLink = doc.SelectSingleNode("//ns:TestLink[@name='" + testcase + "']", namespaceManager);

            return testLink.Attributes["id"].Value;

        }

The below method creates a TRX for the list of test names

static void CreateNewTrx(List<string> selectedTests)

        {

            XmlDocument doc = new XmlDocument();

            doc.Load(oldTrx);

            XmlNamespaceManager namespaceManager = new XmlNamespaceManager(new NameTable());

            namespaceManager.AddNamespace("ns", "http://microsoft.com/schemas/VisualStudio/TeamTest/2010");

 

            XmlNode unitTest = doc.SelectSingleNode("//ns:UnitTest", namespaceManager);

            XmlNode testEntry = doc.SelectSingleNode("//ns:TestEntry", namespaceManager);

            XmlNode unitTestResult = doc.SelectSingleNode("//ns:UnitTestResult", namespaceManager);

 

            XmlNode counters = doc.SelectSingleNode("//ns:ResultSummary/ns:Counters", namespaceManager);

            counters.Attributes["total"].Value = selectedTests.Count.ToString();

            counters.Attributes["executed"].Value = selectedTests.Count.ToString();

            counters.Attributes["completed"].Value = selectedTests.Count.ToString();

 

            foreach (string selectedTest in selectedTests)

            {

                string testId = GetTestcaseIdFromVsmdi(selectedTest);

                string executionId = Guid.NewGuid().ToString();

                if (doc.SelectSingleNode("//ns:UnitTest[@name='DefaultTestName']", namespaceManager) != null)

                {

                    unitTest.Attributes["name"].Value = selectedTest;

                    unitTest.Attributes["id"].Value = testId;

                    unitTest.SelectSingleNode("ns:Execution", namespaceManager).Attributes["id"].Value = executionId;

                    unitTest.SelectSingleNode("ns:TestMethod", namespaceManager).Attributes["name"].Value = selectedTest;

                    testEntry.Attributes["testId"].Value = testId;

                    testEntry.Attributes["executionId"].Value = executionId;

                    unitTestResult.Attributes["executionId"].Value = executionId;

                    unitTestResult.Attributes["testName"].Value = selectedTest;

                    unitTestResult.Attributes["testId"].Value = testId;

                }

                else

                {

                    XmlNode newUnitTest = unitTest.Clone();

                    newUnitTest.Attributes["name"].Value = selectedTest;

                    newUnitTest.Attributes["id"].Value = testId;

                    newUnitTest.SelectSingleNode("ns:Execution", namespaceManager).Attributes["id"].Value = executionId;

                    newUnitTest.SelectSingleNode("ns:TestMethod", namespaceManager).Attributes["name"].Value = selectedTest;

                    unitTest.ParentNode.AppendChild(newUnitTest);

 

                    XmlNode newtestEntry = testEntry.Clone();

                    newtestEntry.Attributes["testId"].Value = testId;

                    newtestEntry.Attributes["executionId"].Value = executionId;

                    testEntry.ParentNode.AppendChild(newtestEntry);

 

                    XmlNode newUnitTestResult = unitTestResult.Clone();

                    newUnitTestResult.Attributes["executionId"].Value = executionId;

                    newUnitTestResult.Attributes["testName"].Value = selectedTest;

                    newUnitTestResult.Attributes["testId"].Value = testId;

                    unitTestResult.ParentNode.AppendChild(newUnitTestResult);

                }

            }

 

            doc.Save(newTrx);

        }

Now you are ready to run the tests from the TRX.

The only thing to note is that while running the tests from the Test Results window, don’t check any test. Just click Run and Enjoy!