Test Run

Stress Testing.

James McCaffrey

Code download available at:TestRun05.exe(116 KB)

Contents

Overall Program Structure
The Tool in Detail
Cool Console Output
Extending the Tool
Wrapping It Up

Stress testing is a fundamental quality assurance activity that should be part of every significant software testing effort. The key idea behind stress testing is simple: instead of running manual or automated tests under normal conditions, you run your tests under conditions of reduced machine or system resources. The resources to be stressed generally include internal memory, CPU availability, disk space, and network bandwith. To reduce these resources for testing you can run a tool called a stressor.

The image in Figure 1 shows one such stress tool, EatMem, in action. It is common to name stressors with an "Eat" prefix followed by the type of resource being consumed, so stressors that reduce memory, CPU availability, and disk space might be named EatMem, EatCpu, and EatDisk, respectively. The EatMem stressor is a command-line program that accepts a single argument specifying how long, in minutes, to run. Approximately every three seconds EatMem attempts to allocate a random amount of available memory. As you see in Figure 1, it is possible for the memory request to fail. In that case EatMem keeps trying until successful. Of course, the stressor tool by itself does not perform any actual testing; it simply prepares a machine state for testing. By running this program at the same time as the actual program under test, you can simulate conditions for your application that may otherwise be difficult to test for under normal testing circumstances.

Figure 1 EatMem Stressor in Action

Figure 1** EatMem Stressor in Action **

Overall Program Structure

The overall structure of the EatMem stressor tool, written in C#, is presented in Figure 2. I begin by adding a using statement to the InteropServices namespace so I can call native Win32® functions. In particular I call the GlobalAlloc and GlobalFree functions to allocate and free memory, and the GlobalMemoryStatus function to determine how much memory to allocate. I also use some Win32 API functions to manipulate the console text colors.

Figure 2 EatMem Program Structure

using System; using System.Runtime.InteropServices; namespace EatMem { class Class1 { [DllImport("kernel32.dll")] extern static IntPtr GlobalAlloc(uint uFlags, uint dwBytes); [DllImport("kernel32.dll")] extern static IntPtr GlobalFree(IntPtr hMem); [DllImport("kernel32.dll")] extern static void GlobalMemoryStatus(ref MEMORYSTATUS lpBuffer); public struct MEMORYSTATUS { public uint dwLength; public uint dwMemoryLoad; public uint dwTotalPhys; public uint dwAvailPhys; public uint dwTotalPageFile; public uint dwAvailPageFile; public uint dwTotalVirtual; public uint dwAvailVirtual; } static MEMORYSTATUS memStatusBuffer = new MEMORYSTATUS(); [STAThread] static void Main(string[] args) { try { // get number of minutes to run from command line // determine stopping time // print output header while (currentTime < stopTime) // main loop { for (int cycle = 0; cycle < 8; ++cycle) { // get the elapsed time so far // compute a random percentage // get current memory status // compute number bytes to eat // allocate memory // get current memory status // print current time, memory status // pause to give app under test stress time // free memory } // for loop } // main while loop } catch(Exception ex) { Console.WriteLine(«Fatal error: « + ex.Message); Console.ReadLine(); } } // Main() } // class } // ns

The GlobalAlloc and GlobalFree functions are defined in the kernel32.dll library. By using the DllImport attribute I can call them from my C# stressor program. The original C++ signature for GlobalFree is:

HGLOBAL GlobalAlloc(UINT uFlags, SIZE_T dwBytes);

The return type, HGLOBAL, is one of the thousands of Win32 symbolic constants. It is essentially a pointer to the first address of memory allocated by GlobalAlloc, so I translate that type into a System.IntPtr type. IntPtr is a platform-specific type that is used to represent either a pointer or a handle. I will explain GlobalAlloc and its partner GlobalFree more fully in the next section. I use the GlobalMemoryStatus function to retrieve the amount of currently available memory so I can determine how much memory to attempt to allocate with the GlobalAlloc function. The Win32 signature for GlobalMemoryStatus is:

void GlobalMemoryStatus(LPMEMORYSTATUS lpBuffer);

You pass a pointer/reference to a struct into GlobalMemoryStatus and it populates the struct with memory information. The struct is defined in Figure 2. The most important field in the struct is the dwAvailVirtual field, which will hold the amount, in bytes, of available virtual memory, after the function GlobalMemoryStatus has been called.

The EatMem tool is organized primarily as a while loop that will iterate until the current system date/time exceeds the date/time determined by adding the duration, in minutes, specified on the command line to the time when the tool started. Inside the main while loop, there is a secondary for loop that executes eight times. This secondary loop is not functionally necessary; it is there to provide a nice way to display progress output every so often.

The Tool in Detail

Now let's go through each section of the EatMem tool in detail. I will discuss fancy console output in the next section of this column, and I will discuss alternatives and modifications to the basic code later. Figure 3 shows the code that will get EatMem going.

Figure 3 Initialize EatMem

const uint basicUnit = 1024; // 1 KB Random r = new Random(); if (args.Length != 1 || !args[0].StartsWith("-d")) { Console.WriteLine("\nExample usage: eatmem -d100"); Console.Write("Randomly eats between 10% and 70% Console.WriteLine(" of virtual memory for 100 minutes"); } // determine stopping time int duration = int.Parse(args[0].Substring(2, args[0].Length-2)); TimeSpan ts = new TimeSpan(0, duration, 0); // hr, min, sec DateTime startTime = DateTime.UtcNow; DateTime stopTime = startTime + ts;

I begin by declaring a constant of 1024 because I will do all EatMem display in kilobytes, while the calculations will be performed in bytes. Next I instantiate a Random object so I can generate pseudorandom numbers later.

Then I use a very crude technique to grab the duration (in minutes) that EatMem should run. My code assumes there is exactly one command-line argument, in the exact form of -dN where N is a number. I can get that number as shown because args[0] will represent a string like "-d123", and Substring(2) will be the "123" part. I use a standard technique to determine the stopping time: I create a TimeSpan object of the desired number of minutes, then use that TimeSpan with the overloaded "+" operator applied to the time when EatMem started.

The last thing that I need to do before entering the main while control loop is to display a brief header for the output along the lines of the following code:

// print output headers WriteLine("Begin EatMem for " + duration + " minutes"); WriteLine("Consuming random 10%-70% of virtual memory"); WriteLine("Launch test harness while EatMem stressor runs"); WriteLine(" Available Available Current"); WriteLine(" Physical Virtual Memory"); WriteLine(" Time Memory Memory Allocation Pct."); WriteLine("---------------------------------------------");

Note that I've changed the actual statements slightly for clarity. You can get the exact statements in the code download that accompanies this column.

As you'll see later in this column there is a lot of additional information available to you. Because stressor tools are intended to modify the state of the machine on which they run, all output from the stressors is to some extent unnecessary. The most important data to monitor is the amount of virtual memory available at any given time.

Once inside the main while loop and the auxiliary for loop, I get the elapsed time into a TimeSpan object and compute a random percent, as you see here:

while (DateTime.UtcNow < stopTime) // main loop { for (int cycle = 0; cycle < 8; ++cycle) { // get the elapsed time so far TimeSpan elapsedTime = DateTime.UtcNow - startTime; // compute a random percentage double randomPct = (r.NextDouble() * (0.70 - 0.10)) + 0.10; if (randomPct < 0.10 || randomPct > 0.70) throw new Exception("Bad random pct");

Here I've hardcoded EatMem so that it generates a random number between 0.10 and 0.70 using a standard mapping technique. The method call Random.NextDouble will return a number in the range [0.0, 1.0], which means greater than or equal to 0.0 and strictly less than 1.0. If I multiply that interval by (0.70 - 0.10) = 0.60, the interval will map to a random number in the range [0.0, 0.6]. Then if I add 0.10, the final interval will map to [0.10, 0.70].

The next step is to compute the number of bytes to eat:

// compute number of bytes to eat GlobalMemoryStatus(ref memStatusBuffer); uint numBytesToEatInBytes = (uint)(randomPct * memStatusBuffer.dwAvailVirtual); uint numBytesToEatInKB = numBytesToEatInBytes / basicUnit;

I call the Win32 GlobalMemoryStatus method using the C# interoperability mechanism to get the number of bytes of virtual memory currently available. Notice I pass in a ref to the MEMORYSTATUS struct because the original C++ signature requires a pointer to the struct argument. Or, to put in managed code terms, I pass in a reference to a struct because the struct will change. After the memory status information is stored into the memStatusBuffer object, I can pull it out. I determine the number of bytes to allocate by multiplying the number of available bytes by the random percentage computed in the previous step. I also convert that result to kilobytes for display purposes.

I am now ready to attempt to allocate memory using the GlobalAlloc function (see Figure 4).The GlobalAlloc function accepts two arguments. The first argument is a flag that specifies one of several ways to allocate:

GMEM_FIXED = 0x0000 Allocates fixed memory. GMEM_MOVEABLE = 0x0002 Allocates movable memory. GMEM_ZEROINIT = 0x0040 Initializes memory contents to 0. GPTR = 0x0040 Combines GMEM_FIXED and GMEM_ZEROINIT. GHND = 0x0042 Combines GMEM_MOVEABLE and GMEM_ZEROINIT.

Here I use GPTR = 0x0040 in order to allocate fixed memory that is initialized to zero.

Figure 4 Allocate Memory

// allocate memory uint GPTR = 0x0040; // Combines GMEM_FIXED and GMEM_ZEROINIT IntPtr p = GlobalAlloc(GPTR, numBytesToEatInBytes); if (p == IntPtr.Zero) { string time = elapsedTime.Hours.ToString("00") + ":" + elapsedTime.Minutes.ToString("00") + ":" + elapsedTime.Seconds.ToString("00"); Console.Write("{0,8}", time); Console.WriteLine(" GlobalAlloc() failed. Trying again"); --cycle; continue; }

The second argument to GlobalAlloc is the number of bytes to allocate. If the allocation is successful, GlobalAlloc returns an IntPtr that points to the first address of the allocated memory. If the allocation fails, the return value is IntPtr.Zero (roughly equivalent to unmanaged code null). The allocation can fail for several reasons; in fact, I would expect the allocation to fail sometimes under conditions of stress. So I do not throw an exception, I merely print a warning message to try again. The GlobalAlloc function allocates the specified number of bytes from the heap.

When dealing with memory allocation there are three components to be aware of: physical memory, the page file, and virtual memory. A program can execute an instruction only when the instruction is in RAM. RAM has some fixed size; 512MB is a typical amount on a fairly recent desktop machine. If a program is smaller than the amount of RAM, then the entire program may be loaded into memory. The idea of virtual memory is to allow programs that are larger than RAM to run. A page file is used by the operating system to swap code in and out of memory when necessary. The default original page file size is 1.5 times the size of RAM, but the page file can increase if necessary. The maximum size of virtual memory varies, but 2GB is typical on a 32-bit version of Windows®.

After allocating memory, I need to get an updated memory status so I can display various diagnostic data:

// get current memory status GlobalMemoryStatus(ref memStatusBuffer); uint availablePhysicalInKB = memStatusBuffer.dwAvailPhys / (uint)basicUnit; uint availableVirtualInKB = memStatusBuffer.dwAvailVirtual / (uint)basicUnit; numBytesToEatInBytes = (uint)(randomPct * memStatusBuffer.dwAvailVirtual); numBytesToEatInKB = numBytesToEatInBytes / basicUnit;

I call GlobalMemoryStatus again. This time I compute the available amount of physical memory in kilobytes, as I do for virtual memory. Then I compute the memory to eat in both bytes and kilobytes, just for display purposes.

Most of the lines of code in EatMem are dedicated to output display. I begin by printing the current elapsed time:

// print time string time = elapsedTime.Hours.ToString("00") + ":" + elapsedTime.Minutes.ToString("00") + ":" + elapsedTime.Seconds.ToString("00"); Console.Write("{0,8}", time);

Then I print the available amounts of physical and virtual memory, and the number of bytes to allocate:

// print the memory numbers Console.Write("{0,12} {1,12} {2,12}", availablePhysicalInKB + " KB", availableVirtualInKB + " KB", numBytesToEatInKB + " KB");

Notice that for simplicity, the amount of allocated memory I display is based on the current memory status rather than on the memory status when GlobalAlloc was actually called. Next I display the percentage of available memory eaten and a pipe character to act as the left margin for the status bar display:

// print the current alooc percent number Console.Write(" " + (int)(randomPct*100.0)); Console.Write("|");

Now to display the status bars I use an old trick:

// do the bars uint totalNumBars = 20; uint eachBarIs = availableVirtualInKB / totalNumBars; uint barsToPrint = numBytesToEatInKB / eachBarIs; string bars = new string('#', (int)barsToPrint); string spaces = new string('_', 20-(int)barsToPrint); Console.Write(bars); Console.Write(spaces); Console.WriteLine("|");

Here I have decided to display the status bars as 20 characters. If I have 20 characters, I can figure the width that each character represents by dividing the amount of available virtual memory by 20. The status bar will consist of some characters plus some blanks. The number of characters will be the number of kilobytes allocated divided by the width of each bar. The number of spaces will be 20 minus the number of characters. In the previous code, I use the '#' character. I will explain how to print a fancy colored bar in the next section.

After all the display work is done, I need to halt the thread of execution for a while so that the system under test can be exercised by the tests that I assume are running at this time. (Remember, the whole idea of a stressor is to set up a machine state where tests can run under conditions of reduced resources.) Here I pause for an arbitrary three seconds and then free up the allocated memory:

// pause to give app under test stress time System.Threading.Thread.Sleep(3000); // free memory IntPtr freeResult = GlobalFree(p); if (freeResult != IntPtr.Zero) throw new Exception("GlobalFree() failed");

The GlobalFree function accepts a pointer to memory that was returned by a call to the GlobalAlloc function. It attempts to free up that memory. If the free operation succeeds, GlobalFree returns null/IntPtr.Zero. If the operation fails, GlobalFree returns the input argument. In terms of the stressor, if GlobalFree fails I have a serious problem and need to throw a fatal exception.

The EatMem tool finishes up by displaying an information bar at the end of each of the eight auxiliary for loop cycles, and a "done" message after the main while loop terminates when the current time exceeds the specified run time.

Cool Console Output

When writing command shell testing tools, appropriate use of text color can make your output easier to read and interpret. For example, EatMem uses green for memory numbers, red for error messages, and white blank space characters to create the memory allocation status bars. My colleagues tell me I went a bit overboard with the additional colors here. The trick to printing console text in color in a Microsoft® .NET Framework 1.1 environment is to use the Win32 functions GetStdHndle and SetConsoleTextAttribute. You declare them like this:

[DllImport("kernel32.dll")] extern static IntPtr GetStdHandle (int nStdHandle); [DllImport("kernel32.dll")] extern static bool SetConsoleTextAttribute( IntPtr hConsoleOutput, int wAttributes);

To set a text color, you first call GetStdHandle, which will return a handle to Standard Output. Then you call SetConsoleTextAttribute to specify the text color. This code will print "Hello" in pale yellow followed by "Bye" in pale red:

const int STD_OUTPUT_HANDLE = -11; IntPtr hStdOut = GetStdHandle(STD_OUTPUT_HANDLE); SetConsoleTextAttribute(hStdOut, 0x0006); Console.WriteLine("Hello"); SetConsoleTextAttribute(hStdOut, 0x0004); Console.WriteLine("Bye");

The value -11 is a magic constant that specifies Standard Output. The constants 0x0006 and 0x0004 are pale yellow and red, respectively. The specified text color stays in effect until a new call to SetConsoleTextAtribute is made. With console output, the "pale" versions of color constants are usually too dull for text. You can intensify any text color constant by logical OR'ing it with the value 0x0008. So this code will print "Howdy Dowdy" in bright yellow:

SetConsoleTextAttribute(hStdOut, 0x0006 | 0x0008); Console.Write("Howdy Dowdy");

Because dealing with hex constants is a bit annoying, it is common to place symbolic constants in your tool. Here are the ones I like to use with command shell programs:

const int BLACK = 0x0000; const int BLUE = 0x0001 | 0x0008; // bright blue const int GREEN = 0x0002 | 0x0008; // bright green const int CYAN = 0x0003 | 0x0008; // etc. const int RED = 0x0004 | 0x0008; const int MAGENTA = 0x0005 | 0x0008; const int YELLOW = 0x0006 | 0x0008; const int WHITE = 0x0007 | 0x0008;

In addition to specifying text colors, you can specify background colors. To do this you logical OR the text color with another constant representing the background color. For example, this code will set the text context to bright yellow (0x0006 | 0x0008) on a pale red background (0x0040):

SetConsoleTextAttribute(hStdOut, 0x0006 | 0x0008 | 0x0040);

In general, I like to use only bright white, pale white (gray), and black as my backgrounds. The following shows the constants I used in EatMem:

const int BGWHITE = 0x0070 | 0x0080; const int BGBLACK = 0x0000 | 0x0000; const int BGGRAY = 0x0070;

With this code you can do a lot of clever color formatting. Here is how I print the memory status bars in EatMem:

string bars = new string('_', (int)barsToPrint); string spaces = new string('_', 20-(int)barsToPrint); SetConsoleTextAttribute(hStdOut, BLACK | BGWHITE); Console.Write(bars); SetConsoleTextAttribute(hStdOut, WHITE | BGBLACK); Console.Write(spaces); SetConsoleTextAttribute(hStdOut, GREEN | BGBLACK); Console.WriteLine("|");

By printing blank spaces as black text against a white background, you can produce what appears to be a white bar. Then if you switch to printing white text against a black background, you will produce a black background. I use the underscore character to produce a line under each bar.

The Console class in the .NET Framework 2.0 has greatly enhanced support for fancy output. If you are working in a .NET Framework 2.0 environment, using the new Console class is a much better approach than using the P/Invoke mechanism. The Console class contains methods and properties to get or set the size of the screen buffer, console window, and cursor; to change the position of the console window and cursor; to move or clear data in the screen buffer; to change foreground and background colors; to change the text displayed in the console title bar; and to play the sound of a beep.

Extending the Tool

The EatMem tool I've presented here can be used as is, but was specifically designed so you can easily modify it. Let's look at a few of the ways you might want to extend or enhance the basic version of EatMem.

The most significant possibility for modification involves the technique used to allocate memory. EatMem uses the Win32 GlobalAlloc and GlobalFree functions, but there are several other memory allocation functions you can use. One possibility is to use the Marshal.AllocHGlobal and Marshal.FreeHGlobal managed methods; they are managed wrappers around LocalAlloc and LocalFree. The AllocHGlobal method is somewhat easier to use than GlobalFree though not quite as flexible. But if you want to avoid using the P/Invoke mechanism directly, using AllocHGlobal is your solution.

Another alternative to using GlobalAlloc and GlobalFree is to use the VirtualAlloc and VirtualFree Win32 functions. The VirtualAlloc function reserves or commits a region of pages in the virtual address space of the calling process. For an example of VirtualAlloc, see Figure 5.

Figure 5 Using VirtualAlloc

P/Invoke Declarations

[DllImport("kernel32.dll")] extern static IntPtr VirtualAlloc(IntPtr lpAddress, uint dwSize, uint flAllocationType, uint flProtect); // LPVOID VirtualAlloc(LPVOID lpAddress, SIZE_T dwSize, // DWORD flAllocationType, DWORD flProtect); [DllImport("kernel32.dll")] extern static int VirtualFree(IntPtr lpAddress, uint dwSize, uint dwFreeType); // BOOL VirtualFree(LPVOID lpAddress, SIZE_T dwSize, // DWORD dwFreeType);

Allocating Memory

// VirtualAlloc uint MEM_RESERVE = 0x1000; uint MEM_COMMIT = 0x2000; uint PAGE_READWRITE = 0x04; IntPtr p = VirtualAlloc(IntPtr.Zero, numBytesToEatInBytes, MEM_RESERVE | MEM_COMMIT, PAGE_READWRITE); if (p == IntPtr.Zero) { string time = elapsedTime.Hours.ToString("00") + ":" + elapsedTime.Minutes.ToString("00") + ":" + elapsedTime.Seconds.ToString("00"); SetConsoleTextAttribute(hStdOut, CYAN | BGBLACK); Console.Write("{0,8}", time); SetConsoleTextAttribute(hStdOut, RED | BGBLACK); Console.WriteLine(" VirtualAlloc() failed. Trying again"); --cycle; continue; }

Freeing Memory

uint MEM_RELEASE = 0x8000; int freeResult = VirtualFree(p, 0, MEM_RELEASE); if (freeResult == 0) throw new Exception("VirtualFree() failed !");

An enhancement you might want to consider adding to the basic version of EatMem is to protect yourself against failed attempts to allocate memory. As written, if a memory allocation attempt fails, EatMem simply prints an error message and tries again. This could lead to a series of failed attempts that might go unnoticed. You could easily add a counter that tracks the number of consecutive failed allocation attempts, and if the counter exceeds some threshold value, you could throw an exception or dynamically reduce the random maximum percentage (currently hardcoded at 70 percent) of memory attempted to allocate. A possible variation on this enhancement is to track the percentage of failed memory allocation attempts and adjust stressor parameters accordingly.

If you are in a .NET Framework 2.0 environment and do not want to use the managed AllocHGlobal because you need the flexibility of the Win32 GlobalAlloc, you can take advantage of the new SafeHandles namespace, which contains classes that provide functionality supporting file and operating system handles. You could create a SafeHandle-derived class for GlobalAlloc, which would look something like Figure 6.

Figure 6 Using the SafeHandles Namespace

[SecurityPermission(SecurityAction.LinkDemand, UnmanagedCode=true)] public sealed class SafeGlobalAllocHandle : SafeHandleZeroOrMinusOneIsInvalid { [DllImport("kernel32.dll")] public static extern SafeGlobalAllocHandle GlobalAlloc(uint uFlags, IntPtr dwBytes); private SafeGlobalAllocHandle() : base(true) { } protected override bool ReleaseHandle() { return GlobalFree(handle) == IntPtr.Zero; } [SuppressUnmanagedCodeSecurity] [ReliabilityContract(Consistency.WillNotCorruptState, Cer.Success)] [DllImport("kernel32.dll", SetLastError=true)] private static extern IntPtr GlobalFree(IntPtr handle); }

This approach provides several benefits. You can use a using statement to make it easy to clean up the memory in the event of an exception being thrown. For example:

using(SafeGlobalAllocHandle mem = SafeGlobalAllocHandle.GlobalAlloc(0, numBytes)) { if (mem.IsInvalid) { ... } else { ... } }

This approach makes the system much more reliable, which can be particularly important in situations where EatMem stressor code is copied directly into an application or test harness that is prone to asynchronous failures.

Some of my colleagues have randomized the sleep time and memory allocation pattern. As written, EatMem sleeps a constant 3000 ms each time through the auxiliary for loop. Specifying a random sleep time gives you a more representative background memory consumption in some testing situations. Instead of EatMem simply asking for a block of memory, you can ask for the memory in increments. The key to this approach is to maintain an ArrayList collection that holds the incremental allocation IntPtr objects so you can incrementally free memory as you need it by iterating through the collection.

Aside from these relatively major changes, I'm sure you can think of many modifications, such as parameterizing the random range of memory allocation, fetching additional memory status information using the GlobalMemoryStatus function, and adding additional error-checking. Note that I decided to strip away most of my error handling for brevity in this column.

Wrapping It Up

In addition to using the memory stressor presented here, you should also stress your system by reducing CPU availability and reducing disk space. Those stressors are similar to memory reduction, but there are a few additional tricks involved that I'll investigate in a future Test Run column.

In addition, other types of stress testing you might want to explore include running test automation over an extended period of time under normal conditions. Stress analysis, or subjecting your application under test to increasingly large reductions in memory, CPU availability, or disk space until the system fails, is another variation. The idea in this case is to determine functional boundaries of the system under test. Still another type of stress test, especially suited to Web apps, is called "load testing." This usually refers to subjecting the Web application and/or Web server to heavy user loads. So as you see, there's a lot to discover in the area of stress testing. Stay tuned to future columns for more.

Send your questions and comments for James to  testrun@microsoft.com.

James McCaffrey works for Volt Information Sciences Inc., where he manages technical training for software engineers working at Microsoft. He has worked on several Microsoft products including Internet Explorer and MSN Search. James can be reached at jmccaffrey@volt.com or v-jammc@microsoft.com. Thanks to Lutz Roeder for his help with this column.