I was doing some load tests recently and while Visual Studio Team Services (VSTS) has a reasonable UI and provides with some free load testing minutes, I eventually needed to spin up my custom load testing rig in Azure. The classical approach with Test Controller and Test Agents would work fine, but it adds a layer of complexity and management which has already been solved by VSTS.

Introduction

There are several ways to run load test with the assistance of VSTS:

• from web portal with automatically provisioned agents,
• from web portal with self-provisioned agents,
• from Visual Studio, connected to VSTS, with automatically provisioned agents,
• from Visual Studio, connected to VSTS, with self-provisioned agents.

Since first three options are quite straight-forward and well described, I'm going to focus on the fourth one.

The Goal

1. Prepare a Web perfomance testing scenario in Visual Studio 2017,
2. run it as a Load Test from Visual Studio
3. on custom test rig in Azure,
4. orchestrated by VSTS.

How to

Get VSTS Personal Access Token

For the test rig to communicate with VSTS properly, it needs to have a Personal Access Token (PAT). This piece of information is generated in VSTS, under your account and Security.

Click Add in the Personal access tokens section.

Select a fitting description for it, expiration period and account. Selected scopes should be Load Test (read and write).

After clicking Create token, copy the generated string somewhere as it will not be displayed again.

Provision a Test Rig in Azure

With the PAT in hand we are ready to provision our own test rig in Azure. This step is fairly easy, because Microsoft provides an ARM template which spins up a specified number of virtual machines and associates them with your VSTS account.

Click this link to open Azure portal and start deploying the template.

Make sure you are signed in to the correct Microsoft Account before provisioning.

Fill the form - there should be no catch as all fields are well described.

And finally click Purchase. The infrastructure will be provisioned in a few minutes.

(Optional) Use Test Rig from VSTS

To quickly test that the rig works properly we can simply create a new URL based test and select Use self-provisioned agents in the Settings section.

Use Test Rig from Visual Studio

The interesting part comes when you wish to run more complex load tests using Visual Studio. Actually, to be precise, the combination is Visual Studio + VSTS + custom test rig.

Keep in mind that load-testing is limited to the Enterprise edition of Visual Studio.

Start by preparing a regular Web Performance Test as described in the documentation.

When asked for the location, keep it Default (it will not be taken into account when running the test anyway).

Add two context parameters to your test's Run Settings:

• UseStaticLoadAgents = true
• StaticAgentsGroupName = [your test rig group name]

The number of agents your test will use can be set in the Cloud Settings property group of the Run Settings:

1 core = 1 machine of the test rig.

Run your test. You will immediately see in the log how many machine it will be using:

Shutdown

And finally, don't forget to Stop your test rig virtual machines in Azure when you're done to prevent unnecessary charges. They will connect to VSTS automatically on next start.

Summary

Not difficult, but not straight-forward, since the support for self-hosted rigs together with VSTS is missing in the Visual Studio UI as of yet. This was solved by adding two contextual parameters: UseStaticLoadAgents and StaticAgentsGroupName.