I do like the idea of replaying logs as performance tests, this got me thinking:
http://www.igvita.com/2008/09/30/load-t ... og-replay/

When Steve Hedley and I did the ATM Service Management system performance testing, we took a recording of the service requests set by 7000 ATM's, parsed them and scaled them up to be 20,000 ATM's.

Record live system log, then tweak it to be input to a test system performance test.

I keep reading articles about Performance Testing in Virtualised Environments, like this one:
http://virtualization.sys-con.com/node/697934
which I spotted here on http://twitter.com

The common theme I'm seeing is that the authors are documenting Performance Testing good practices or experiences which is not specific to Virtualised Environments.

Certainly there are authors considering the performance testing of virtualised systems: http://blogs.intel.com/idf/2008/08/virt ... ce_tes.php

But I am wondering if the performance tester needs to consider doing things differently when all or part of their system under test is virtualised?

Certainly I spend more time on the monitoring front:
1. Monitor the physical server(s) machine stats, minimum of Network, CPU and Memory.
2. Monitor the virtual server(s) machine stats, minimum of Network, CPU and Memory.
3. Think about the counters/stats provided by virtualisation server (application)

I always speak to the application experts (which seem to be few and far between) to ask what performance stats they are interested in.

The system should have a set of performance requirements at a business level, use these to drive the load that is applied and don't treat virtualised performance testing like something that is wildly different to physical system performance testing.

Hammerhead looks interesting:
http://www.stevesouders.com/blog/2008/0 ... -upstream/

A quick way for developers to baseline a webpage and gauge performance tweak success.

May be a cheap way for performance testers to baseline a test - if it doesn't meet the requirement for one user - I'd like to know.

This question on linked in:
http://www.linkedin.com/answers/technol ... 81-3692291
... reminded me that I must spend *a lot* more time digging into the monitoring environments.

I was speaking last week with an Oracle DBA doing battle with a problematic 4.5 hour batch process. He had some great ideas on monitoring the performance of his process. The work is on going, but we were talking about getting his custom monitoring output shared through a standard format such as linux RSTATD.

This way the process can be monitored in the live environment if required. Of course I was thinking about getting the data across into LoadRunner Analysis.

People don't spend nearly enough time considering the monitoring requirements for performance testing.

I was recently helping a colleague to highlight how good some transaction response times were. However his graphs were plagued with the odd "spike" where a response time was really bad, i.e from 1.4 seconds average up to a 90+ second spike.

This made the transaction response time graph look bad and would have cause the reader of the report to focus on the spikes instead of the averages.

Instead a transaction response time distribution graph was used, this highlights a count just how many transaction had a particular response time:



The graph is much easier to talk about since the readers ear is immediately (and rightly so) drawn to the the "big number".

What I'm wondering about now is - why are there two clear groups of slow transactions? Between 46 to 58 seconds and 76 to 98 seconds. I've heard about defect clustering - but slow transaction clustering sounds strange.... but they are clustered.