Archive for the ‘Scalability’ Category

Danny Tamez

Tsung Quick Start Part 2

In part 1 of this article (http://voices.canonical.com/isd/?p=118) we did sort of a hello world for a Tsung load test.  This time we’re going to continue by getting into more of the details of how Tsung works and how to use it for real world tests.

Using the Recorder

More likely than not, the application you want to test consists of lots of urls with lots of query string values or form post values.  Typing this in by hand is not practical and would take way too long.  What Tsung provides is a a proxy recorder that you can set up to record all of the http calls you are making to and receiving from you application.  The end result is a Tsung “session” in an xml file.  You can then tweak the file however you want to better suit your tests.

Here’s how to set it up.
1) Configure your browser to use a proxy server at http://localhost:8090.  For firefox you can follow the instructions at http://support.mozilla.com/en-US/kb/Options window – Advanced panel?as=u#advanced_network .
2) Start the proxy recorder

$ tsung-recorder start

3) Point your browser to your application and go through one or more scenarios that you would like to group together as a teset scenario.
4) When you’re ready to stop recording simply stop the recorder.

$tsung-recorder stop

You now have a Tsung session xml file.  You can now change things like thinktimes or change values you typed in to be dynamically generated.  In order to use the file in your main tsung.xml file all you need to is include it as an xml entity and then reference it inside the sessions tag.

<?xml version="1.0"?>
<!DOCTYPE tsung SYSTEM "/usr/share/tsung/tsung-1.0.dtd" [
<!ENTITY new_accounts SYSTEM "./new_accounts.xml">
<!ENTITY log_out SYSTEM "./log_out.xml">
<!ENTITY log_in SYSTEM "./log_in.xml">
<!ENTITY auth_logged_in SYSTEM "./auth_already_logged_in.xml">
<!ENTITY auth_logged_out SYSTEM "./auth_not_logged_in.xml">
<!ENTITY api_register SYSTEM "./api_register.xml">
<!ENTITY api_server SYSTEM "./api_server.xml">
]>

<tsung loglevel=”debug” dumptraffic=”true” version=”1.0″>

.
.
.
<sessions>
<session name="web" probability="70" type='ts_http'>
&new_accounts;
&log_in;
</session>
.
.
.
<sessions>

Creating Load

Load is configured in Tsung inside the ‘load’ tag. This tag has an optional ‘duration’ attribute that can specify a maximum time for the test to run regardless of whether or not all requests have been processed. Inside of the load tag you specify one or more ‘arrivalphase’ tags each with a specified duration.  Finally, within each arrival phase you specify via the ‘users’ tag the rate at which users are generated for that phase.

<load>
<arrivalphase phase="1" duration="6" unit="minute">
<users interarrival="1.0" unit="second"/>
</arrivalphase>
<arrivalphase phase="2" duration="5" unit="minute">
<users interarrival="0.8" unit="second"/>
</arrivalphase>
.
.
.
</load>

Some caveats:
Keep in mind that the number of concurrent users will depend on how long it takes for each user’s session to be processed.  If sessions are long then there will be overlap between arrival phases. Likewise, request made per second is not a direct correlation of the user arrival rate as each sessioin  may have thinktimes between each request that it makes. Additionally, as sessions usually have more than one request, each user created will overlap subsequent requests with new users’ requests.

One of the reasons that you would choose Tsung over other load testing tools is that it will generate much higher load than other tools.  Being built on Erlang, one server can typically generate tens of thousands of users.  However, you can also generate even more load if necessary by running a cluster of servers.

Using Clusters

It is easy to use Tsung with one client or several clients and just as easy to use it with one or more servers.

<clients>
<client host="&client_host_1;" use_controller_vm="true" maxusers="2000" cpu="2" weight="2"/>
<client host="&client_host_2;" use_controller_vm="true" maxusers="2000" weight="1"/>
<client host="&client_host_3;" use_controller_vm="true" maxusers="2000" weight="1"/>
</clients>
<servers>
<server host="&provider_host_1;" port="80" type="tcp"></server>
<server host="&provider_host_2;" port="80" type="tcp"></server>
</servers>

In the above example Tsung will send twice as many requests to client 1 than it sends to client 2 or 3.  Another way of looking at is that 1/2 of the requests will come from client 1, 1/4 from client 2 and 1/4 from client 3.  The weight will be evenly distributed between the two servers via simple round robin.

Using Dynamic Variables

Often you will have situations where you can’t hardocode parameters into your session files.  For instance, sometimes you need to use a value in one request that was generated by the previous request.  An example of this is if your application uses Cross Site Request Forgery tokens.

<thinktime min="2" max="10" random="true"/>
<transaction name="display_log_in">
<request>
<dyn_variable name="csrfmiddlewaretoken"/>
<http url='&provider_url;' version='1.1' method='GET'/>
</request>
</transaction>
<thinktime min="7" max="30" random="true"/>
<transaction name="log_in">
<request subst="true">
<http url='&provider_url;/+login' version='1.1'
contents='csrfmiddlewaretoken=%%_csrfmiddlewaretoken%%&amp;email=%%_email%%@%%_domain%%.com&amp;password=%%_password%%&amp;continue='
content_type='application/x-www-form-urlencoded' method='POST'/>
</request>
<request>
<http url='&provider_url;' version='1.1' method='GET'/>
</request>
</transaction>

In the example above a form is displayed in the “display_log_in” transaction.  A hidden csrf token is in that pages login form.  We are capturing that value via the ‘dyn_variable’ tag in the first requeset.  We need that value in the second requeset and use  it in the ‘content’ attribute as ‘%%_csrfmiddlewaretoken’. When using dynamic variables, be sure to use ‘subst=”true”‘ in the request tag or the variables will be interepreted as literals.

Sometimes the value you need is on the page but is not a form field.  You can use a regex to match it and store it in a variable.

<dyn_variable name="janrain_nonce" regexp='janrain_nonce=\([^\"]*\)'/>
<http url='&consumer_url;/verify?openid_identifier=&provider_url;/'
version='1.1' method='GET'/>

It’s also easy to share variables across requests as they can be defined for a session.

<session name="web" probability="70" type='ts_http'>
<setdynvars sourcetype="random_string" length="8">
<var name="password"/>
</setdynvars>
<setdynvars sourcetype="random_string" length="5">
<var name="first"/>
</setdynvars>
<setdynvars sourcetype="random_string" length="5">
<var name="last"/>
</setdynvars>
<setdynvars sourcetype="random_string" length="5">
<var name="domain"/>
</setdynvars>
<setdynvars sourcetype="random_string" length="5">
<var name="email"/>
</setdynvars>
<setdynvars sourcetype="random_string" length="16">
<var name="secret"/>
</setdynvars>
&new_accounts;
&log_in;
&auth_logged_in;
&log_out;
&auth_logged_out;
</session>

If you need something more powerful you can call custom Erlang code defined in an external module,

<setdynvars sourcetype="eval"
code="fun({Pid, DynVars}) -> sso:auth_header(DynVars) end.">
<var name="authorization_header"/>
</setdynvars>

or evaluate Erlang code inline,

<setdynvars
sourcetype="eval"
code="fun({Pid,DynVars})->
{ok,Val}=ts_dynvars:lookup('openid.assoc_handle',DynVars),
edoc_lib:escape_uri(Val)
end.">
<var name="assoc_handle"/>
</setdynvars>

or by looking up a value from an external file.

<options>
<option name="file_server" id="accounts" value="../accounts.csv"/>
</options>
<sessions>
<session name="all" probability="100" type='ts_http'>
<setdynvars sourcetype="file" fileid="accounts" delimiter="," order="iter">
<var name="email" />
<var name="domain" />
<var name="password" />
</setdynvars>

Looping and Branching

Tsung allows you to use simple branching and looping in your Tsung files. For example you can bracket a request inside an if tag so that it only is called under certain conditions,

<if var="is_super_user" eq="1">
<request> <http url="/delete_account"></http> </request>
</if>

or loop a certain request several times for the same user.

<for from="1" to="5" incr="1" var="product_id">
<request> <http url="/product?id=%%_product_id%%"></http> </request>
</for>

There is still much more that Tsung can do and if you’re interested in using it there is a great community of people around it that can be of assistatnce.  I hope these two articles will be of some help to those of you starting with Tsung and that it becomes a great tool in your toolbox.  Good luck!

Danny Tamez

Load Testing with Tsung Quick Start

What is Tsung and what is load testing?

One of the tools we use on the ISD team is Tsung.  We use it to load test our web based applications in our staging environment.  In this article we’re going to go through all the steps necessary to setup and run a simple as possible load test.  Later we’ll look at how to configure Tsung and your environment for more complex scenarios.

Load testing is the process of examining a system (such as an application or a device) under higher than normal load in order to see how it performs under those conditions.  It is a very important part of testing any application where load amounts are variable and unpredictable. Load testing is a huge topic by itself but in brief here a few things that load testing can do for you:

  • Benchmark the performance of the application under load so that it can be compared to previous or future releases.
  • Identify trouble spots in the application that break only when under high load.
  • Help managers more accurately anticipate and plan for resources such as hardware, employees, bandwidth etc.
  • Test whether the application is going to perform within its specifications under projected usage.

Tsung is a distributed, multi protocol load testing framework that can be used to stress web and database servers.  Tsung has many things in common with other load testing suites but what distinguishes it from the others is its ability to produce very high load with limited resources.  Tsung is written in Erlang, (a language built for high concurrency and fault tolerance) but uses simple XML files for its configuration of load testing scenarios.

Let’s Try it Out

The best way to get familiar with Tsung is just to try it out.  It doesn’t take long at all to get a working Tsung load test up and running.

Installation

My examples were all run on Kubuntu 10.10 so you may have to adapt tweak these steps a little for other distributions.

The first thing you’ll need to do is install erlang-nox on all of the servers that will be tested or used as clients.

 $sudo apt-get install erlang-nox

You should now be able to run

 $erl

from your terminal and be in an Erlang shell. Type ‘Control-C’ and then ‘a’ to get out of the Erlang shell.

Next you’ll need need to install Tsung on your client machine.  Tsung has not yet been packaged for Ubuntu but a Debian package is available.

 $wget http://tsung.erlang-projects.org/dist/debian/tsung_1.3.3-1_all.deb
 $sudo dpkg -i tsung_1.3.3-1_all.deb

The following are needed for generating reports from the collected data:

 $sudo apt-get install gnuplot perl libtemplate-perl python-matplotlib

You should now be able to run

$tsung -v

You should then see ‘Tsung version 1.3.3′ displayed.

In order for Tsung to communicate with the other servers in the cluster and obtain statistics from them, password-less ssh must be enabled between the client machine(s) and the server machine(s).

 $ssh-keygen
 $ssh-copy-id -i ~/.ssh/id_rsa.pub username@host

If for the purposes of experimenting you’re just running both client and server on one machine the above is not necessary.

Running:

You will recall that Tsung uses XML files for its configuration.  Here is an example of pretty much the smallest Tsung configuration file:

<?xml version=”1.0″?>
<!DOCTYPE tsung SYSTEM “/usr/share/tsung/tsung-1.0.dtd”[]>
<tsung loglevel=”info”>
<!– Minimal tsung configuration file –>
<clients>
<client host=”localhost”/>
</clients>
<servers>
<server host=”localhost” port=”8000″ type=”tcp”/>
</servers>
<load>
<arrivalphase phase=”1″ duration=”1″ unit=”minute”>
<users interarrival=”1″ unit=”second”/>
</arrivalphase>
</load>
<sessions>
<session name=”foo” probability=”100″ type=”ts_http”>
<request>
<http url=’/’ version=’1.1′ method=’GET’/>
</request>
</session>
</sessions>
</tsung>

Save the above file and give it a name, for example minimal.XML.

Basically before it can run Tsung has to know 4 things: the IP address of the machine on which the client is running, the IP address of the machine on which the server is running, how many users to create, and what each user is to do.  In the above example both the client and the server are running on localhost, a new user will created each second for one minute, and each user will run one get request of the home page of the tested site.

Before we can run our simple load test we’ll need to run our server. I am running a Django application that’s listening on port 8000 so you’ll need to change your server line to match your environment.  Once your server is running you’re ready to start.

 $sudo tsung -r erl -f minimal.xml start

Note: Normally you do not need to run Tsung via sudo, but the very first time that it is run it needs to create files that require root access.

If Tsung ran without any problems you should have seen displayed the path to the directory where the log files for this load test can be found. CD to that directory and generate your report like this:

 $/usr/lib/tsung/bin/tsung_stats.pl

You can now view your report from a browser:

 $firefox report.html &

Congratulations!  You’ve just setup and run your first Tsung load test. Next time we’ll look at more complex scenarios and delve deeper into Tsung’s capabilities.