Are there any plans to create an Azure version of BizUnit

Jun 11, 2011 at 2:08 AM

Hi there, I have been porting some of my applications to Azure and I have used BizUnit previously to perform integration & load tests.

Now I have my application on Azure and I am able to spin up far more resources than I had access to previously I want to look at creating something that can run a bizunit but across multiple threads and compute instances.

Before I go off trying to make this, is there anything like this on the BizUnit roadmap? Would you be interested in a contribution that supplied this?

Regards

Nikolai Blackie

Coordinator
Jun 17, 2011 at 2:17 PM

Hi Nikolai,

 

I'd like to understand your scenario a little better in terms of what you are trying to acheive. Are you trying to test multi-threaded code, or run BizUnit test cases on multiple threads to improve the execution time of the tests or for load/pref testing purposes?

Certainly in the past we have driven BizUnit from VS Load Tester, this enabled us to effectively use BizUnit to help drive stress/perf runs by executing scenarios on multiple threads, instances of BizUnitwere not shared across threads since by their nature functional test cases are typically sequential. You could also use LoadGen to do something similar.

 

Thanks,

Kevin

Aug 5, 2011 at 7:57 AM

Apologies for the late response.

I am trying to test a multi tenant service simulating 1000's of organisations submitting requests to test the robustness of the scale path designed.

I have found that executing multi threaded requests from a single host does not generate the load required to simulate our target throughput.

I am doing some performance labs next month testing our services across 500 compute instances and I will be using a scalable framework to host and execute BizUnits across parallel instances and threads.

Contact me directly if you want to discuss my plans further.

Thanks

Nikolai Blackie

Coordinator
Aug 5, 2011 at 8:52 AM

Hi Nikolai,

So what you are descibing is a load test scenario as opposed to functional test scenario. BizUnit is targetted at functional testing, whilst tools like Visual Studio Load Tester are targetted at load testing. BizUnit will not be moving into the Load Testing space, since its a functional test framework, there are plenty of great tools out there for Load Testing already.

That said, in the past I have used bizUnit in conjunction with the VS Load Tester, each call from the Load Tester executes a BizUnit script, so the load is driven by VS, but BizUnit provides the functional scenario execution on top of it. this sounds like the scenario that you are working with potentially, I'd recommend you go down that route.

Thanks,

Kevin

 

Aug 5, 2011 at 10:38 PM

Yes I do see your point about the difference between functional and load testing scenarios.

I guess what I am attempting to do is build the load testing harness using BizUnit to drive the tests. Unfortunately all the VS load test tools are only available under Ultimate, and the way it uses agents to perform the scale out is too cumbersome as I do not have lots of infrastructure to scale out on, which is why I am looking to use Azure to I guess host the load agents. Also many of the commercial offering like LoadStorm are focused on the UI, where I am wanting to test the scalability of my backend services

The way I am approaching it is like this:

  1. Create a series of BizUnits functionally testing various aspects & configurations using messages of varying sizes
  2. Take my scale framework and implement a simple service that accepts a request to run a load test containing
    • The number of tests to run
    • Which test types to run
  3. This service will then create several load tasks that will be scattered via queues to a pool of processing nodes
  4. Each node pulls tasks in parallel off the queue and executes the BizUnit requested
  5. The results are stored in a blob store and referenced from a storage table where the details of all the runs are stored
  6. Performance counters will be recorded using standard Azure collection methods during the test and extracted at the end of the test run
  7. All instrumentation from the service being tested will also be extracted at the end of the test run

 

Coordinator
Aug 21, 2011 at 6:15 PM

Have you looked at LoadGen: http://www.microsoft.com/download/en/details.aspx?id=14925 ?

This is a load/perf tool that was developed by the BizTalk Load/Perf team to test BizTalk, its very flexible and extensible and from what you've described would meet your requirements, and the best bit ... its free! I recomend you take a look at using it in conjunction with BizUnit.

Cheers,

Kevin

Aug 21, 2011 at 8:54 PM

I have used LoadGen for our onsite testing previously and it works great.

I attempted to build the test service described above but I soon discovered that NUnit is very unthread safe with no support for concurrent collections etc, as I was hosting it across several threads in the same process I quickly discovered I could not use it.

What I have settled for is simply calling my tests directly from my agents, this is working quite nicely and with two azure compute instances, across 6 threads, I can quite easily generate the load of 1000 customers submitting 2000 line messages in under 1 minute.

Scaling this out now is as simple as adding more compute instances and reconfiguring the test requests.

It has immediately assisted me in resolving several issues I was not able to replicate testing from a single machine, even with multiple threads.

Thanks for all your input Kevin, your feedback has been excellent.

Nikolai