Category Archives: Programming

Using a UK postcode to find Lat and Long and other useful information

So I have a list of UK postcodes and I want to find out the latitude and longitude of that postcode to plot it on to a map. There are numerous ways to do this, including:

1. Visiting Google Maps

2. Entering the postcode in to the search box

3. Right-clicking on the little point on the map and selecting “What’s here?”

4. Copying and pasting the lat and long that magically appears in the search box.

That’s nice, quick and easy BUT what if I’ve got hundreds or thousands of postcodes to search? I can’t spend the whole sunny weekend doing the same procedure can I?

No.

I’m self-learning Tableau at the moment and it seems to have great location support…if you provide it with US addresses (gnrrr), but I wanted to find a way of plotting a series of points on a map of the UK derived using their addresses. A bit of Google searching led me to UK Postcodes (set up by Stuart Harrison and based on Ordnance Survey OpenData) , a site that lets you enter a UK postcode and returns lots of information about that location (e.g. long, lat, county and right down to district and ward) and what made me excited was that the site had an API allowing you to pass it a postcode via URL (e.g. http://www.uk-postcodes.com/postcode/NW18TQ.json) and it would output the meta data it in either XML, CSV, JSON or RDF for you. PERFECT!

After a further read around the site, I found that Edd Robinson had created a library acting as a wrapper for the API which I could import in to my own Python project. And so, without further ado here is my Python code:

[code language=”Python”]
from postcodes import PostCoder
import csv

f = open(‘Extracted_Data_from_Postcodes.csv’, ‘w’)
i = 0
pc = PostCoder()

loc_lat = ""
loc_long = ""

with open(‘Postcodes_to_Extract.csv’, ‘rU’) as g:
reader = csv.reader(g)

for row in reader:
#Col 0 = ID :: Col 1 = Postcode
result = pc.get(str(row[1]))

if str(result) != "None":
location = result[‘geo’]

for key, value in location.items():
if (key == "lat"):
loc_lat = value
if (key == "lng"):
loc_long = value

#ID, Postcode, Lat, Long
write_to_file = str(row[0]) + ", " + str(row[1]) + ", " + str(loc_lat) + ", " + str(loc_long) + "n"

#Add the iteration count to output to screen to see how far we are up to.
print str(i + 1) + ", " + write_to_file
f.write(write_to_file)
else:
#If there is a problem translating the postcode, output "BROKEN" to file to manually check.
write_to_file = "BROKEN, " + str(result) + ", " + str(row[0]) + ", " + str(row[1]) + "n"

print str(i + 1) + ", BROKEN, " + str(row[0]) + ", " + str(row[1])
f.write(write_to_file)
i = i + 1

f.close()

[/code]

My input file looked like this:

[code]
109484, SG5 4PF
109486, MK44 2DB
109487, LU4 9UJ
109488, LU6 1RE
109489, MK43 8DY
109490, MK45 5JH
109491, MK44 3QD
109492, MK45 3BX
109493, MK17 9QL
109494, MK43 9JT
[/code]

And my screen output looked like this:

[code]
1, BROKEN, 109484, SG5 4PF
2, 109486, MK44 2DB, 52.214741, -0.461977
3, 109487, LU4 9UJ, 51.927696, -0.500824
4, 109488, LU6 1RE, 51.879322, -0.563452
5, 109489, MK43 8DY, 52.164539, -0.623209
6, 109490, MK45 5JH, 51.982376, -0.495111
7, 109491, MK44 3QD, 52.137440, -0.377085
8, 109492, MK45 3BX, 52.080341, -0.446214
9, 109493, MK17 9QL, 51.989906, -0.619803
10, 109494, MK43 9JT, 52.095955, -0.528752
[/code]

Afterwards, I was then able to manually check out why SG5 4PF failed, actually, I’m not sure why it failed but I was able to correct the lat & long via a Google Map search. Exactly what I needed with minimal effort and a chance to flex my muscles with Python again. I then imported the CSV in to Tableau, but that’s the subject of a future blog post…

Setting up Python and Easy_Install on Windows 7

This weekend, I needed to set up Python on a Windows 7 virtual machine that I had set up on my Mac. I don’t know why, but I always seem to have a problem installing Python and setting up the environment and so I’m going to document the process here so that in X months time when I need to do it again, I just need to follow my own instructions!

As part of the process, I also install easy_install to install libraries and so I’ll also document that here.

1. Download the Python Windows installer from here -> http://www.python.org/download/ (I downloaded Python 2.7.5) and run the executable on your machine.

2. Run the installer – I normally leave all the defaults as they are, so end up installing it in C:Python27

3. Once installed, SET UP YOUR ENVIRONMENT VARIABLES, I can not express how important this is as you’ll hit so many problems without do it (try opening the command prompt and typing python to see Windows shrug its shoulders and say “huh…dunno…”)

  • If you haven’t played with environment variables before, just stick to following these instructions as you can set them up through the Windows GUI.
  • Right click on “My Computer”, select “Properties” > “Advanced system settings” and click on the “Environment Variables” button
  • In the System Variables box, find the variable called “path” and click on the “Edit…” button
  • In the “Variable value” box, at the end of the entry, add the following text: ;C:Python27;C:Python27Scripts (note the leading semi colon, also note that I’m using the default installation folder – if you have installed Python elsewhere, just change the path structure).
  • Click “OK” a couple of times and hey presto, your environment variables are set up.
  • To test it has worked, open a new command window and type in python and if you see this, you’re on to a winner…

Microsoft Windows [Version 6.1.7600]
Copyright (c) 2009 Microsoft Corporation. All rights reserved.

C:Usersadelegilpin>python
Python 2.7.5 (default, May 15 2013, 22:43:36) [MSC v.1500 32 bit (Intel)] on win32
Type “help”, “copyright”, “credits” or “license” for more information.
>>>

4. The next step in the process is to set up easy_install and so we need to go to the setuptools page (links to version 0.8) and download the ez_setup.py script (I normally put it straight in to my Python27 folder).

5. Open a command prompt and type python ez_setup.py install – you’ll see a load of code whizz by which will hopefully end as follows;

Processing dependencies for setuptools==0.8
Finished processing dependencies for setuptools==0.8

C:Python27>

6. easy_install has now been set up and you can test to see if it is there, by typing easy_install in to a command prompt, which will throw an error about no URLs, you know that the tool has been set up successfully

Done!

To use easy_install to get new libraries, just use the following syntax: easy_install <library name> (e.g. easy_install postcodes)

Using Python & BeautifulSoup to scrape a Wikipedia table

Well, it was only a couple of weeks ago that I set myself a challenge to complete the Python course on Codecademy and I did it – I completed the Python track and it was fantastic! I was given the opportunity to put my newly found Python skills in to action this week as I needed to scrape some data from a Wikipedia page – I have a table of addresses and need to compare the County in the list that has been provided to the one that it really should be. This page on Wikipedia contains the data I need, for each Postcode district there’s a Postal County and I could use this data as a comparison – formatted in an HTML table like this:

WikiCapture

Normally, I’d just copy & paste the table in to Excel for use later on BUT it’s not as easy as that (oh no!), as there can be are multiple Postcode Districts within a row which is slightly annoying! To be of any use to me, I need the data to be formatted so that there is a row for each Postcode District like so (I don’t necessarily need the Postcode Area & Town but I’ll keep them anyway – I don’t like throwing away data!):

Postcode Area Postcode District Post Town Former Postal County
AB AB10 ABERDEEN Aberdeenshire
AB AB11 ABERDEEN Aberdeenshire
AB AB13 ABERDEEN Aberdeenshire
AB AB15 ABERDEEN Aberdeenshire

And so I thought this would be the perfect project for me to undertake in Python and to familiarise myself with friend-of-the-screen-scrapers, BeautifulSoup. I won’t jabber on too much about BeautifulSoup as I’m not fully up to speed on it myself yet, but from reading around the subject I gather it’s a great way to grab elements from web pages for further processing.

Step One: Wikipedia doesn’t like you…

Wikipedia doesn’t like this code:

[code language=”Python” highlight=”7″]
from bs4 import BeautifulSoup
import urllib2
wiki = “http://en.wikipedia.org/wiki/List_of_postcode_districts_in_the_United_Kingdom”
page = urllib2.urlopen(wiki)
soup = BeautifulSoup(page)
print soup
#urllib2.HTTPError: HTTP Error 403: Forbidden

[/code]

Wikipedia only allows access to recognised user agents in order to stop bots retrieving bulk content. I am not a bot, I just want to practise my Python and so to get around this you just need some additional code to the header (thanks to Stack Overflow for coming to the rescue).

Step Two: Hunt the table

If you look at the code behind the Wikipedia article, you’ll see that there are multiple tables but only one (thankfully the one we want) uses the “wikitable sortable” class – this is great as we can use BeautifulSoup to find  the table with the “wikitable sortable” class and know that we will only get this table.

[code language=”Python”]
from bs4 import BeautifulSoup
import urllib2
wiki = “http://en.wikipedia.org/wiki/List_of_postcode_districts_in_the_United_Kingdom”
header = {‘User-Agent’: ‘Mozilla/5.0’} #Needed to prevent 403 error on Wikipedia
req = urllib2.Request(wiki,headers=header)
page = urllib2.urlopen(req)
soup = BeautifulSoup(page)

area = “”
district = “”
town = “”
county = “”
table = soup.find(“table”, { “class” : “wikitable sortable” })
print table
[/code]

Output looks like this:

TableOutput

Great! This means that we just have the HTML table stored in our variable. Now, it’s just a case of iterating through the rows and columns…easy…*ahem*

Step Three: For your iteration pleasure

We need to do the iteration in two stages – the first stage is to iterate through each row (tr element) and then assign each element in the tr to a variable. At this stage, we will grab everything in the Postcode Districts column and store it in a list for further iteration later. To do this, I used the following code:

[code language=”Python” firstline=”19″]
for row in table.findAll(“tr”):
cells = row.findAll(“td”)
#For each “tr”, assign each “td” to a variable.
if len(cells) == 4:
area = cells[0].find(text=True)
district = cells[1].findAll(text=True)
town = cells[2].find(text=True)
county = cells[3].find(text=True)
[/code]

The .findAll function in Python returns a list and so on line 20, we obtain a list containing four elements, one for each of the columns in the table. This means they can be accessed via the cells[n].find(text=True) syntax. You’ll notice that I’ve used .findAll for the Postal Districts column, this is because I want a list of the items within the cell for iteration purposes later!

After this code executes, I have a value for the area, a list of districts, a town and a county. Now for the second part of my iteration:

[code language=”Python” firstline=”28″]
#district can be a list of lists, so we want to iterate through the top level lists first…
for x in range(len(district)):
#For each list, split the string
postcode_list = district[x].split(“,”)
#For each item in the split list…
for i in range(len(postcode_list)):
#Check it’s a postcode and not other text
if (len(postcode_list[i]) > 2) and (len(postcode_list[i]) <= 5):
#Strip out the “n” that seems to be at the start of some postcodes
write_to_file = area + “,” + postcode_list[i].lstrip(‘n’).strip() + “,” + town + “,” + county + “n”
print write_to_file
[/code]

I found that, instead of district being a standard list of postcodes, in some cases it was a list of lists (oh joy!). I was expecting it to looks like this:

[u’AB10, AB11, AB12, AB15, AB16, nAB21, AB22, AB23, AB24, AB25, nAB99, non-geo’] *

*Ignore the n signs and non-geo text – we’ll deal with them later!

I got this…

[u’AB10, AB11, AB12, AB15, AB16,‘, u’nAB21, AB22, AB23, AB24, AB25,‘, u’nAB99‘, u’non-geo‘]

And so I needed an additional layer of iteration: one for the whole list and then another for the items in the individual lists. Simple.

For each item in the list, the .split(",") function in Python allowed me to split out the comma separated list of postcodes in to a list that could be iterated over. For each item in that list, we just check to see if it’s a postcode (a check on string length sufficed nicely this time!) and then build up our output string. To deal with the n that was appended to some of the postcodes, I just left-stripped the string to remove the n characters and hey presto it worked!

I flushed the output to a CSV file as well as to the screen and it worked beautifully!

Here is the full code:

[code language=”Python”]
from bs4 import BeautifulSoup
import urllib2

wiki = “http://en.wikipedia.org/wiki/List_of_postcode_districts_in_the_United_Kingdom”
header = {‘User-Agent’: ‘Mozilla/5.0’} #Needed to prevent 403 error on Wikipedia
req = urllib2.Request(wiki,headers=header)
page = urllib2.urlopen(req)
soup = BeautifulSoup(page)

area = “”
district = “”
town = “”
county = “”

table = soup.find(“table”, { “class” : “wikitable sortable” })

f = open(‘output.csv’, ‘w’)

for row in table.findAll(“tr”):
cells = row.findAll(“td”)
#For each “tr”, assign each “td” to a variable.
if len(cells) == 4:
area = cells[0].find(text=True)
district = cells[1].findAll(text=True)
town = cells[2].find(text=True)
county = cells[3].find(text=True)

#district can be a list of lists, so we want to iterate through the top level lists first…
for x in range(len(district)):
#For each list, split the string
postcode_list = district[x].split(“,”)
#For each item in the split list…
for i in range(len(postcode_list)):
#Check it’s a postcode and not other text
if (len(postcode_list[i]) > 2) and (len(postcode_list[i]) <= 5):
#Strip out the “n” that seems to be at the start of some postcodes
write_to_file = area + “,” + postcode_list[i].lstrip(‘n’).strip() + “,” + town + “,” + county + “n”
print write_to_file
f.write(write_to_file)

f.close()
[/code]

Disclaimer(ish)

This code has no additional error checking or handling and was merely written to solve a small problem I had, and to put in to practise everything I’d learned so far. It does also just work for this particular table on the Wikipedia page – although it could be adapted for use on other tables. But, it was great fun to put the learning in to action and work on a real-life problem to solve. Here’s to more exercises like this!

Self development fortnight

Well, it’s been a while since my last update and a lot has been happening with this blog behind the scenes (bye, bye 1&1 hosting and hello to the wonderful Squirrel Hosting) and with myself.

In short, I am going in to hospital for an operation on 6th June which is a little bit nerve-wracking, exciting (strange…but this op will vastly improve my quality of life) and unknown – I haven’t had an operation since my tonsils were removed when I was six and so I don’t know what to expect, I only have Holby City to go on! As a consequence of this, I have two weeks off work – time off beforehand for preparation & relaxation and then time off afterwards to rest & recuperation. I am putting my foot down now and saying that these two weeks will be for self development, learning new things and essentially NOT SITTING AROUND AND WATCHING DAYTIME TV (like I always find myself doing when I have the odd day off!)…except for Pointless, I love Pointless and so that’s my only exception.

So, I have created an account on Codecademy and am slowly making my way through the Python course. At the time of writing, I am up to lesson 8 on Loops and I am thoroughly enjoying it, it’s such a great way to learn a new language. I have written C++ and C# code in the past and so am not a complete beginner, but it’s great to start right at the beginning and learn a new language from scratch. It’s a bit of a revelation not to have to add a semi-colon at the end of a line of code…it feels a bit naughty!

When I have completed the Python course, I hope to undertake a small project of my own. My main motivation for learning Python is for screen-scraping and data extraction purposes and so I’d like to start a project to help me to gain experience in these areas. I will of course keep blogging about my progress and the new discoveries that I make.

The fortnight of self development starts now…

C++ Unit Testing

Unit testing in managed languages such as Java and C# is easy. The general theory is that you create a Test annotation and apply that to your test functions. Next, you create a series of assert functions that throw an AssertFailed exception if an assertion doesn’t match what you expect. Finally, create a test runner that scans your assemblies or JARs using reflection for functions marked with your Test annotation and invoke them. The test runner just needs to catch any exceptions thrown by the test functions and report them somewhere. The test runner doesn’t have to care too much if the exception thrown is a failed assert or something more serious such as a null pointer exception, it can handle both of them in pretty much the same way. Tools such as NUnit or TestNG will provide all this for you so you will very rarely ever need to write any of this yourself.

With C++, things aren’t quite so easy. There isn’t really any form of annotations or reflection so discovering tests is harder. You do have exceptions, but you might not be able to use them due to the environment you are using and you don’t get call stacks with them either. And anyway, you could just get a fatal crash deep in the code you’re testing before you ever get the chance to throw an exception from one of your assertions.

This doesn’t mean that you can’t get C++ unit testing frameworks with a similar level of functionality as the ones for managed languages, Google Test is a pretty good one for example and CppUnitLite2 is another example of a very portable framework. I want to take a look at how a C++ unit testing framework could be implemented as I find it an interesting problem.

Goals

  • Easy to implement test functions that can be discovered by the test runner.
  • Assert functions that will tell me what test has failed along with a call stack.
  • Fatal application crashes won’t kill the test runner but are reported as failed tests along with a call stack.
  • Possible to plug into an continuous integration build server so that it can evaluate if a build is stable or not.

For my example framework, I’ll only be targeting Unix type platforms (Linux, Mac OS) as the methods I’ll be using are cleaner to implement making it easier to explain the theory. This also allows me to to provide a sample that will work on Ideone so you can have a play with the framework and see it running without needing to download any code.

The framework I present here takes its inspiration from Google Test so I highly recommend taking a look at that.

The Sample Framework

You can try out my sample framework on Ideone. Due to Ideone being primarily a code fiddle site to try out ideas, all your code must live in a single source file so don’t judge the structure too harshly! Normally you would separate everything out a bit and have clear interfaces between the test runner and your tests.

Test Function Registration

This is achieved by defining a macro to generate the test function declaration. The macro also creates a static object that contains the test function details and registers itself with the test runner in its constructor. The test function details contain the name of the function, the source file, line number and a pointer to the function to execute. They can then be stored in a simple linked list for the test runner to iterate over when it comes to run the tests. By using static objects, we can ensure that all our tests are registered automatically before main() is executed saving us the need to explicitly call a set up function that contains a list of all our test functions that needs to be maintained as new tests are added.

Test Reference Class

[code language=”cpp” firstline=”36″]
//—————————————————————————————–//
// Class for storing reference details for a test function
// Test references are stored in a simple linked list
//—————————————————————————————–//
// Type def for test function pointer
typedef void (*pfnTestFunc)(void);

// Class to store test reference data
class TestRef
{
public:
TestRef(const char * testName, const char * filename, int lineNumber, pfnTestFunc func)
{
function = func;
name = testName;
module = filename;
line = lineNumber;
next = NULL;

// Register this test function to be run by the main process
registerTest(this);
}

pfnTestFunc function; // Pointer to test function
const char * name; // Test name
const char * module; // Module name
int line; // Module line number
TestRef * next; // Pointer to next test reference in the linked list
};

// Linked list to store test references
static TestRef * s_FirstTest = NULL;
static TestRef * s_LastTest = NULL;
[/code]

This is a pretty simple class as it doesn’t need to do much more than register itself. In my sample, registerTest() is a global function that just add the object to the linked list.

[code language=”cpp” firstline=”206″]
// Add a new test to the linked list
void registerTest(TestRef * test)
{
if (s_FirstTest == NULL)
{
s_FirstTest = test;
s_LastTest = test;
}
else
{
s_LastTest-&gt;next = test;
s_LastTest = test;
}
}
[/code]

Test Registration Macro

[code language=”cpp” firstline=”22″]
// Macro to register a test function
#define TEST(name)
static void name();
static TestRef s_Test_ ## name(#name, __FILE__, __LINE__, name);
static void name()
[/code]

It simply declares the test function prototype, constructs a static test reference object passing the function pointer into the constructor and then declares the first line of the function implementation. Here’s an example of using it:

[code language=”cpp”]
TEST(MyTest)
{
// Your test implementation…
}
[/code]

When the macro is expanded by the preprocessor, it effectively becomes:

[code language=”cpp”]
static void MyTest();
static TestRef s_Test_MyTest(&quot;MyTest&quot;, &quot;example.cpp&quot;, 1, MyTest);
static void MyTest()
{
// Your test implementation…
}
[/code]

I’ve inserted line breaks to make it easier to read.

Test Execution

This isn’t the exact code I’ve used in my sample, but it’s doing pretty much the same thing.

[code language=”cpp” light=”true”]
TestRef * test = s_FirstTest;
while (test != NULL)
{
test-&gt;function();
// Report success or failure…
test = test-&gt;next;
}
[/code]

Assert Function

In my sample, I’ve just used an assert macro similar to one you’re probably already using in your own code.

[code language=”cpp” firstline=”14″]
// Assert macro for tests
#define TESTASSERT(cond)
do {
if (!(cond)) {
assertHandler(__FILE__, __LINE__, #cond);
}
} while (0);
[/code]

If the assert condition fails, it turns it into a string and passes it along with the current file and line number into an assert handler function to actually report the failure.

This actually isn’t the best example for a unit testing framework as it’s really only testing for a true condition. If you were developing a fully featured framework, you would probably want more assert functions along the lines of ASSERT_EQUALS(actual,expected) and ASSERT_NOTEQUALS(actual,notexpected) so that you can report how the actual result from a test differs from what was expected. Implementing these types of functions isn’t too hard, so I won’t dwell on that now.

Assert Handler

[code language=”cpp” firstline=”242″]
// Handler for failed asserts
void assertHandler(const char * file, int line, const char * message)
{
fprintf(stdout, &quot;nAssert failed (%s:%d):n&quot;, file, line);
fprintf(stdout, &quot; %sn&quot;, message);
fflush(stdout);

dumpStack(1);

_exit(1);
}
[/code]

The functions reports the location of the failed assert along with the failed condition before dumping a stack trace and exiting. The reason for calling exit is because my framework actually runs tests in a child process separate to the test runner (more on that later). This is also why I’ve used fprintf with the stdout file handle rather than just using printf(). The child and parent processes actually share the same file handles and so I need to be explicit about where my output is going and when buffers are flushed so that I don’t get overlapping test output.

Dumping the Call Stack

For this, I’ve used a feature of glibc which is one of the reasons my sample is written for *nix.

[code language=”cpp” firstline=”221″]
// Dump a stack trace to stdout
void dumpStack(int topFunctionsToSkip)
{
topFunctionsToSkip += 1; // We always want to skip this dumpStack() function
void * array[64];
size_t size = backtrace(array, 64);
backtrace_symbols_fd(array + topFunctionsToSkip, size – topFunctionsToSkip, 1); // Adjust the array pointer to skip n elements at top of stack
}
[/code]

I provide the ability to skip a number of calls at the top of the stack so that the assert and stack dumping functions aren’t reported in the call stack. The call stack is then written to stdout directly.

The function backtrace_symbols_fd() will attempt to resolve function symbols when it outputs the stack trace but it can be a bit hit or miss with getting the names and will be affected by optimisation level. For the most likely chance to get symbols out, you need to compile with the -g option and link with -rdynamic if using gcc. When I compile and run the sample on my Raspberry Pi, I get the following call stack for a failed assert:

Assert failed (main.cpp:89):
    1 == false
./a.out[0x8c44]
./a.out(_Z7runTestP7TestRef+0x70)[0x8ed8]
./a.out(main+0xb8)[0x8d40]
/lib/libc.so.6(__libc_start_main+0x11c)[0x403cc538]

As you can see, it’s managed to find the symbols for some functions but not the one at the very top of the call stack which is where our assert failed. Fortunately, we can use the addr2line tool to look up this function:

pi@raspberrypi:~/Devs/cunit_sample$ addr2line -e a.out -f -s 0x8c44
TestAssertFailed
main.cpp:90

Calling addr2line can become quite tedious, so you might find it worth writing a script (e.g. in Python) to feed stack traces into addr2line if you find yourself needing to do this regularly which is something I’ve done in the past.

Sample Test Function

[code language=”cpp” firstline=”74″]
TEST(TestPass1)
{
int x = 1;
int y = 2;
int z = x + y;
TESTASSERT(z == 3);
}
[/code]

Nothing too shocking there and hopefully very easy to implement.

Handling Fatal Crashes

The first C++ testing framework I wrote had all the tests running in the same process. If everything was working, this wouldn’t be a problem as all test would pass without incident. However, if there was a fatal crash (e.g. attempting to use a null pointer), the entire test application would crash halting all the tests making it very difficult to assess the overall code health. This can be resolved by signal handlers that wait for crash conditions and attempt to gracefully clear up so that the test runner can keep on running. However, I still ran into bugs that could screw up the stack or heap in fatal ways leaving me no better off in these situations.

In this sample framework, I’ve borrowed an idea from Google Chrome in that I run each test in its own process. This way a test can mess up its own process as much as it wants and it’s completely isolated from any of the other tests. It also enforces good practice with your tests as you can’t have one test depending on the side effects of another test. Each test is completely independent and can be guaranteed to run in any order which makes them much easier to debug. In addition, it makes my crash handling code much simpler as I don’t need to do any more than report the error and exit the process. Simpler code is good in my opinion.

Signal Handler

[code language=”cpp” firstline=”230″]
// Handler for exception signals
void crashHandler(int sig)
{

fprintf(stdout, &quot;nGot signal %d in crash handlern&quot;, sig);
fflush(stdout);

dumpStack(1);

_exit(1);
}
[/code]

The handler uses the same stack dumping code as the assert handler and exits with a non-zero exit code to notify the parent test runner application that the test has failed.

The handler is registered with the following code in main():

[code language=”cpp” firstline=”104″]
// Register crash handlers
signal(SIGFPE, crashHandler);
signal(SIGILL, crashHandler);
signal(SIGSEGV, crashHandler);
[/code]

Here, I’ve used the antiquated signal() interface when really, I should be using sigaction(). I’ve probably not registered all the signals that could indicate a fatal code bug either. This is something I may address in the future, but for now it provides a simple example of what I’m trying to achieve.

Spawning the Child Test Process

For simplicity, my test runner just forks itself as that’s one of the easiest way to launch a child process on *nix. It also has the advantage of not needing to do much configuration in the child process in order to run the test.

I’ve wrapped the forking and running of a single test in a function to keep all the logic in one place:

[code language=”cpp” firstline=”156″]
// Wrapper function to run the test in a child process
bool runTest(TestRef * test)
{
// Fork the process, the test will actually be run by the child process
pid_t pid = fork();

switch (pid)
{
case -1:
fprintf(stderr, &quot;Failed to spawn child process, %dn&quot;, errno);
exit(1); // No point running any further tests

case 0:
// We’re in the child process so run the test
test-&gt;function();
exit(0); // Test passed, so exit the child with a success code

default:{
// Parent process, wait for the child to exit
int stat_val;
pid_t child_pid = wait(&amp;stat_val);

if (WIFEXITED(stat_val))
{
// Child exited normally so check the return code
if (WEXITSTATUS(stat_val) == 0)
{
// Test passed
return true;
}
else
{
// Test failed
return false;
}
}
else
{
// Child process crashed in a way we couldn’t handle!
fprintf(stdout, &quot;Child exited abnormally!n&quot;);
return false;
}

break;}
}
}
[/code]

After the process is forked, the child process calls the test function referenced in the passed in TestRef object. If the function completes without incident, the child exits with a zero exit code to indicate success. The parent process waits for the child process to exit and then logs success or failure of the test based on the exit code of the child process.

The main test runner loop is:

[code language=”cpp” firstline=”111″]
int testCount = 0;
int testPasses = 0;

// Loop round all the tests in the linked list
TestRef * test = s_FirstTest;
while (test != NULL)
{
// Print out the name of the test we’re about to run
fprintf(stdout, &quot;%s:%s… &quot;, test-&gt;module, test-&gt;name);
fflush(stdout);

testCount++;

bool passed = runTest(test);
if (passed == true)
{
testPasses++;
fprintf(stdout, &quot;Okn&quot;);
}
else
{
fprintf(stdout, &quot;FAILEDn&quot;);
}

// Get the next test and loop again
test = test-&gt;next;
}
[/code]

Plugging Into Build Server

This is just a case of following the Unix principal of your process returning 0 if you’re happy or non-zero if not. In my main() function, I keep a count of the number of tests run and the number of tests passed. I then have the following at the end of main():

[code language=”cpp” firstline=”139″]
// Print out final report
int exitCode;
if (testPasses == testCount)
{
fprintf(stdout, &quot;n*** TEST SUCCESS ***n&quot;);
exitCode = 0;
}
else
{
fprintf(stdout, &quot;n*** TEST FAILED ***n&quot;);
exitCode = 1;
}
fprintf(stdout, &quot;%d/%d Tests Passedn&quot;, testPasses, testCount);

return exitCode;
[/code]

Pretty much every build server has the ability to launch external processes as part of a build and report a build failure if that process doesn’t exit with a zero code. It’s just a case of building your test framework as part of your normal build process and then executing it as a post build step. Everyone should be doing it!

Other Platforms and Future Improvements

As I mentioned earlier, this sample will only work on *nix platforms. However with a bit of work, most of these ideas can be ported to other platforms.

Call Stack Dumping

Although there is no standard way to get a stack trace, it’s been possible on every platform I’ve used so far. Some platforms being easier than others though.

For Windows, here’s one example, There also the CaptureStackBackTrace() API in the Windows API.

Fatal Exception Handling

I’ve already mentioned that I should switch to using sigaction() rather than signal() for registering my crash handlers.

On Windows, you could use Structured Exception Handling (SEH) to detect access violations and other fatal errors. Here’s a Stack Overflow question that covers some of the pros and cons of SEH. This is something that’s always going to be very platform specific so you may have to research this for yourself if you’re using something a bit more esoteric.

Child Process Spawning

This is one area I could put a lot more effort in. Currently, I’m only using fork() which isn’t available on all platforms and only gives me limited control over the child process. If instead I launched the child processes as completely separate processes that I attached to using stdout/stderr and specified which test to run using command line arguments, I’d have a much more portable solution. It would make debugging individual tests much easier as I could launch the test process directly from my debugger without needing to run through a complete test cycle. This would also give me more options over how I implemented my test runner as I could develop a GUI application in a completely different language if I wanted or implement distributed tests across multiple machines if my test cycle could take a long time. Finally, reading test output such as failed assertions and call stacks from stdout of the child process rather than letting the child write directly to stdout of the parent process would allow the test runner to present the output in a much nicer way or redirect it to a separate log file that only contained info about failed tests.

If I were to develop this sample further, this is an area I would certainly put more effort into.

More Restrictive Platforms

A few platforms I’ve worked on have only supported running a single process at a time. Launching another process results in the current running process being unloaded and completely replaced by the child process. This makes running tests in a background child process completely impossible. In these situations, I’d have the runTest() function run the test directly in the current process. The assert and crash handlers would also need to be updated to return control to the test runner in the case of a test failure. Your best bet would be to use normal C++ exceptions for this, but if you really don’t want to use them, you could use setjmp()/longjmp(). Which ever way you go, fatal application crashes are likely to halt your test cycles early.

If possible, I’d try to get the code I was testing to also compile on another platform such as Windows or Linux and perform most of my testing there. If you get to the point where all your tests are passing, running the tests on your target platform should just be a formality to make sure the platform specific parts of you code are also working.

Before/After Method Events

Something that I haven’t implemented in this sample but would be very easy to add would be before and after method events so that common set up and tear down code could be automatically called by the test runner. This is a standard feature of just about every other framework, so I wouldn’t consider a framework I wrote complete without it.

Classical Objects in JavaScript

I’ve recently found myself needing to do more with JavaScript than I have in the past and I thought a good way to understand some of the more interesting features of the language would be to attempt implement Java/C# style classes and inheritance. This has obviously been done before with various JavaScript libraries and frameworks such as Mootools, but I wanted to actually understand what was going on. With that in mind, this post is not about the best way to implement classes, it’s simply about what is happening when you do.

Java vs JavaScript

The big challenge to trying to copy what Java does is that Java and JavaScript are two very different languages. Java is a classic OOP language and all the features I’m trying to emulate are a core part of it. JavaScript has objects but they’re little more than hash tables and so everything is dynamically typed.

My ultimate goal is to implement an object hierarchy in JavaScript such as this Java sample. The features I’m aiming to emulate are:

  • Member functions
  • Public and private member variables
  • Public and private statics
  • Class extending

JavaScript Objects

And to get stuck in, here’s the JavaScript implementation I came up with:

[code language=”javascript”]
ObjectA = (function(){

var privateStatic = &quot;private static&quot;;

function make(x, y) {
this.publicMember = x;
var privateMember = y;

this.setPrivate = function(v) {
privateMember = v;
}
this.getPrivate = function() {
return privateMember;
}

this.calc = function() {
return this.publicMember + privateMember;
}
};

make.prototype.setPublicStatic = function(v) {
make.prototype.publicStatic = v;
};

make.prototype.setPrivateStatic = function(v) {
privateStatic = v;
};
make.prototype.getPrivateStatic = function() {
return privateStatic;
};

make.prototype.publicStatic = &quot;public static&quot;;

return make;

})();

ObjectB = (function(){

function make(x, y, z) {
ObjectA.call(this, x, y);
var privateSubMember = z;

var __calc = this.calc;
this.calc = function() {
return (__calc.call(this) * z) + this.publicMember;
}
}
make.prototype = new ObjectA();
make.prototype.constructor = make;

return make;

})();
[/code]

I’ve created a jsFiddle for the above along with some tests here. The first thing I should point out is that although I achieved my goals, I haven’t necessarily done it in the best way. The public static variable is almost certainly not done well, but I’ll go into that later.

Constructors

When implementing objects like this the the classes are actually functions and those functions also double up as the constructor. When you call new MyObject(); the interpreter allocates a new empty object and then passes it to the function you’ve specified for initialisation. In the code above, I’ve actually declared my constructors as functioned called make() within an anonymous function which returns the constructor.

Member Functions

This is probably the easiest thing to achieve as functions are first-class objects in JavaScript. This means that they can be passed around as data and assigned to other objects in the same way as any other data type. In my above code, I attach functions to my objects in two different ways. The first is by assigning it within the constructor:

[code language=”javascript” highlight=”4,5,6″]
MyObject = (function(){

function make() {
this.doStuff = function() {
// Stuff happens here…
}
}

return make;

})();
[/code]

This approach will allocate a new instance of the function and store it directly within the constructed object each time the constructor is called. This may seem wasteful, but it’s the only way to access private members within the object.

The second way for defining member functions is by assigning the function to the constructors prototype:

[code language=”javascript” highlight=”6,7,8″]
MyObject = (function(){

function make() {
}

make.prototype.doStuff = function() {
// Do stuff here…
}

return make;

})();
[/code]

The prototype member is a dictionary/hash table that is shared by all instances of the same object type. If a member isn’t found directly within the object instance, the prototype will be searched. Assigning functions to the prototype will cause only one instance of the function to be allocated per class, but these functions are only able to access public members of your object.

If you look back at my sample JavaScript above, you can see where I’ve used the two different types of function definition where I’m accessing variables with different access levels.

Public Member Variables

These are defined by simply assigning them to the this object within the constructior:

[code language=”javascript” highlight=”4″]
MyObject = (function(){

function make() {
this.member = 123;
}

return make;

})();
[/code]

The JavaScript language has no syntax for public or private access levels so all members of an object are publicly accessible. As such, there’s not much more to say about public members.

Private Member Variables

This is where things start to get interesting. As I mentioned previously, JavaScript has no concept of public or private variables so we have to fake it using closures. Closures are functions that are able to access variables that while not global, are accessible within the scope that the function way originally defined. That probably didn’t make much sense, so here’s an example:

[code language=”javascript” highlight=”4,5,6,7,8″]
MyObject = (function(){

function make() {
var private = 123;

this.doStuff = function() {
return private * 2;
}
}

return make;

})();
[/code]

The variable private has been defined locally within the constructor which would normally mean that I couldn’t be accessed from outside the constructor, it’s private if you will. However, the function doStuff() was also defined within the constructor and so has access to the same scope including the local variable private.

Public Static Variables

In the code above, I’ve implemented public statics by assigning them to the constructors prototype. I did this so that I could access the statics in a syntactically similar way to Java. However, it has the very big problem in that if I did the following:

[code language=”javascript”]
var o = new ObjectA();
o.publicStatic = &quot;New Value&quot;;
[/code]

The static variable wouldn’t be updated, instead a new member called “publicStatic” would be assigned to the object instance o.

A much better way to implement statics would have been for me to assign them as members of the constructor itself:

[code language=”javascript” highlight=”6,12″]
MyObject = (function(){

function make() {
}

make.static = 123;

return make;

})();

document.write(MyObject.static);
[/code]

Each time I wanted to access the static, I would have to explicitly reference the class that the static belonged to which is not a requirement of Java and C#, but it would be clear exactly what I was doing which is no bad thing.

Private Static Variables

Back to closures again to simulate private variables. This time the static variables are local variables in the anonymous function scope.

[code language=”javascript” highlight=”3,8,9,10″]
MyObject = (function(){

var private = 123;

function make() {
}

make.prototype.getPrivate = function() {
return private;
}

return make;

})();
[/code]

This is the whole reason for using anonymous functions for defining classes. If you didn’t need private static functions, then you could implement everything in this post without using anonymous functions. Depending on how you feel, that might be a good thing or a bad thing.

Class Extending

By setting the constructors prototype to a new instance of the class you want to extend, you are able to simulate inheritance as you know it in other languages. Because the prototype is searched if a requested member isn’t found on the object instanced being referenced, it has the effect of exposing all all the super class members to the new class.

[code language=”javascript” highlight=”16,17″]
ObjectA = (function(){
function make() {
}

make.prototype.hello = function() {
document.write(&quot;Hello!&quot;);
}

return make;
})();

ObjectB = (function(){
function make() {
}

make.prototype = new ObjectA();
make.prototype.constructor = make;

return make;
})();

var o = new ObjectB();
o.hello();
[/code]

By creating a new instance of ObjectA and assigning it to the prototype, it means that we can add new items to the prototype without affecting the original ObjectA class or any other classes that extend it. However, as we’ve replaced our original prototype with a copy from another class, the constructor needs to be set back to the correct one again. constructor is a standard property that contains a reference to the function used to construct the object and so we should always make sure it points to the correct function.

Obviously, any additions you want to make to the prototype need to happen after you’ve set the super class!

Calling the Super Class Constructor

Generally, you should also call the super class’s constructor so that it is set up correctly in your new objects. In Java and C#, the default constructor will be called for you automatically so you often don’t need to worry about this but you must always do this explicitly in JavaScript.

[code language=”javascript” highlight=”15″]
ObjectA = (function(){
function make(message) {
this.message = message;
}

make.prototype.hello = function() {
document.write(this.message);
}

return make;
})();

ObjectB = (function(){
function make() {
ObjectA.call(this, &quot;Hello World!&quot;);
}

make.prototype = new ObjectA();
make.prototype.constructor = make;

return make;
})();

var o = new ObjectB();
o.hello(o);
[/code]

You’ll notice that I’ve used the call() method to invoke the super class constructor. This is because the super class constructor isn’t being called as a member of any object and so the this pointer won’t be what you expect.

Here is a jsFiddle in which the super constructor is called directly. As you can see, when the ObjectA constructor is called directly from the ObjectB constructor, the this reference is actually the browser window. Using the call() method allows us to specify the object instance to use for this when calling the constructor.

This is another down side to attempting to emulate OOP in JavaScript. The call() method is similar to MethodInfo.Invoke() in other languages and so is actually much slower than calling methods directly. If you’re creating a lot of objects frequently and you need your code to run quickly, you don’t want to be calling super class constructors. Mind you, if you want your code to run quickly, you probably won’t be dynamically creating a large number of objects at run time.

Overriding Super Class Methods

The last thing to tackle is overriding super class methods. This is done by storing a copy of the super function and then overriding it with a closure that calls your stored copy.

[code language=”javascript” highligh=”16,17,18,19″]
ObjectA = (function(){

function make() {
}

make.prototype.calc = function(x, y) {
return x + y;
}

return make;
})();

ObjectB = (function(){
function make() {

var _calc = this.calc;
this.calc = function(x, y) {
return _calc.call(this, x, y) * 2;
}

}

make.prototype = new ObjectA();
make.prototype.constructor = make;

return make;
})();

var o = new ObjectB();
document.write(o.calc(1, 2));
[/code]

The reason for using a closure like this rather than simply accessing your class’s prototype is because you may not know if the super function is a closure or attached to the prototype. This way reduces the requirement for the sub class to know the exact implementation details of the super class and therefore reducing maintenance overheads if the super class implementation changes without an interface change.

You also need to use the call() method of the super function again in order to ensure the this pointer is set correctly. You will need to keep the overheads of overriding super class methods in mind if you need to implement fast code.

Conclusions

Although it is certainly possible to emulate classical OOP in JavaScript, in my opinions the overheads of doing so won’t always be worth it. Your code will be more complex and will have less optimal code paths unless you’re willing to trade maintainability and implementation isolation. I can understand the desire for someone to use a familiar programming style but I would question the need to use classical OOP in JavaScript. You can’t achieve an exact copy of what’s available in Java or C# so it’s important to know how it differs when you simulate it.

JavaScript is a great language and it feels like you’re doing it a bit of a disservice if you try to blindly make it act like another language it wasn’t meant to be.

Debugging JMeter Plug-ins

When developing your own plugins, you’ll no doubt need to debug some sort of problem sooner or later. Writing log messages or even just printing directly to stdout will often give you the info you need, but eventually, you’ll want to hook up a debugger and step through your code. Fortunately, this is very easy and just needs the following lines added to the top of your start up scripts:

Linux/Mac – bin/jmeter.sh

JVM_ARGS="-Xdebug -Xnoagent -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=8000"

Windows – bin/jmeter.bat

set JVM_ARGS=-Xdebug -Xnoagent -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=8000

Now when you launch JMeter using either of these scripts, the JVM will start up with the debugger enabled and listening on port 8000. You can then use your IDE of choice to debug your plug-in. In my case, I’ve used Eclipse. To set up the debugger connection, select Debug Configurations… from the Run menu and you should see a dialog similar to the following:

The important things to note here is that you want to add a new Remote Java Application debug configuration and the project should be set to your plug-in project. Optionally, you might also want to add this configuration as a favourite to your debug menu which can be done on Common tab.

When you launch this configuration, the Eclipse debugger will establish a connection to your running JMeter instance and you will then be able to set break points and step through your code as normal. As an added bonus, if you’ve also downloaded the JMeter source code, you’ll be able to attach it to the JMeter Jars in your project allowing to step through the JMeter code as well without having to build it yourself. This is a great way to learn how JMeter works internally which may help you better understand any problems you see with you plug-ins.

Faster JMeter If Controllers

When load testing to a large scale, it’s just as important for your test scripts to be optimized as it is for your servers. Your tests will often need to simulate thousands of users on a single physical box running JMeter and so you need to be running as efficiently as possible. If you are using the standard If Controller in your test with their default configuration, then it’s likely that you’re burning CPU cycles and resources just to check if two variables are the same.

By default, the If Controller will use JavaScript to evaluate the condition you’ve specified. This means that JMeter will create a brand new JavaScript execution context each time an If Controller is processed. This happens for every controller in every thread. That will give your garbage collector a good work out as well as your servers!

Fortunately, the JMeter team knew this could be a problem and so added the option evaluate the condition as a variable expression. This allows you to create a JMeter function to check your condition that returns either true or false. This is tens to hundreds of times faster.

Creating custom functions is even easier than creating a test component. You just need to add the class file to your plugin project and JMeter will automatically find it and do the rest. Here’s another article that goes into the specifics of JMeter functions a bit more and you can also read my original guide on creating custom components.

Common Checks

The most common checks I perform in my scripts are to test if a variable is equal to some value. Using a JavaScript condition, you would specify something like ${value} == 1. However, for my tests, I’ve created my own __eq() function for quickly comparing two values:

[code language=”java”]
package org.adesquared.jmeter.functions;

import java.util.ArrayList;
import java.util.Collection;
import java.util.Iterator;
import java.util.List;

import org.apache.jmeter.engine.util.CompoundVariable;
import org.apache.jmeter.functions.AbstractFunction;
import org.apache.jmeter.functions.InvalidVariableException;
import org.apache.jmeter.samplers.SampleResult;
import org.apache.jmeter.samplers.Sampler;

public class EqFunction extends AbstractFunction {

private final static String NAME = &quot;__eq&quot;;
private final static ArrayList&lt;String&gt; DESC = new ArrayList&lt;String&gt;(2);

private CompoundVariable a;
private CompoundVariable b;

static {
DESC.add(&quot;a&quot;);
DESC.add(&quot;b&quot;);
}

@Override
public String getReferenceKey() {
return NAME;
}

public List&lt;String&gt; getArgumentDesc() {
return DESC;
}

@Override
public String execute(SampleResult previousResult, Sampler currentSampler) throws InvalidVariableException {

String a = this.a.execute();
String b = this.b.execute();

return a.equals(b) ? &quot;true&quot; : &quot;false&quot;;

}

@Override
public void setParameters(Collection&lt;CompoundVariable&gt; parameters) throws InvalidVariableException {

if (parameters.size() &lt; 2) throw new InvalidVariableException(&quot;Not enough parameters for &quot; + NAME);

Iterator&lt;CompoundVariable&gt; it = parameters.iterator();
this.a = it.next();
this.b = it.next();

}

}
[/code]

Using that, I can enable the Interpret Condition as Variable Expression option in the If Controller and change the condition to ${__eq(${value},1)}. That will give me exactly the same functionality as before but without JMeter creating a new JavaScript context each time.

How much faster?

To test how much faster, I put together a small script that had a 100,000 iteration Loop Controller that contained an If Controller that contained a Test Action Sampler (a sampler that will do nothing in this case). I defined the variable value to have the value 1 in the test. You can download the test script from here.

When the test is executed using JavaScript to interpret the condition, it takes 78 seconds to complete the test. Changing the condition to a variable expression using my new __eq() function, the test took less than 2 seconds. Both tests ran using the server VM.

I know which one I’m going to continue to use in the future!

Bonus Not Equals Function

Of course I also often need to check if two values aren’t equal, for those, I use this function:

[code language=”java”]
package org.adesquared.jmeter.functions;

import java.util.ArrayList;
import java.util.Collection;
import java.util.Iterator;
import java.util.List;

import org.apache.jmeter.engine.util.CompoundVariable;
import org.apache.jmeter.functions.AbstractFunction;
import org.apache.jmeter.functions.InvalidVariableException;
import org.apache.jmeter.samplers.SampleResult;
import org.apache.jmeter.samplers.Sampler;

public class NeqFunction extends AbstractFunction {

private final static String NAME = &quot;__neq&quot;;
private final static ArrayList&lt;String&gt; DESC = new ArrayList&lt;String&gt;(2);

private CompoundVariable a;
private CompoundVariable b;

static {
DESC.add(&quot;a&quot;);
DESC.add(&quot;b&quot;);
}

@Override
public String getReferenceKey() {
return NAME;
}

public List&lt;String&gt; getArgumentDesc() {
return DESC;
}

@Override
public String execute(SampleResult previousResult, Sampler currentSampler) throws InvalidVariableException {

String a = this.a.execute();
String b = this.b.execute();

return a.equals(b) ? &quot;false&quot; : &quot;true&quot;;

}

@Override
public void setParameters(Collection&lt;CompoundVariable&gt; parameters) throws InvalidVariableException {

if (parameters.size() &lt; 2) throw new InvalidVariableException(&quot;Not enough parameters for &quot; + NAME);

Iterator&lt;CompoundVariable&gt; it = parameters.iterator();
this.a = it.next();
this.b = it.next();

}

}
[/code]

Using JMeter’s Table Editor

Following on from my previous post about creating custom JMeter components, I thought it worth taking a look at using the test bean table editor as it’s a good way of editing lists for your own components. Unfortunately, it doesn’t have much documentation and only actually works due to type erasure!

I’ve put together another sample that uses the table editor to show how it works. This time, I’ve created a config element that resets variables on each loop iteration. This is different to the User Defined Variable components as that will only set the variables once.

Table Editor Class Overview

The table editor can be found in the org.apache.jmeter.testbeans.gui package along with a number of other additional property editors. It will allow you to edit properties that are set using a list of classes (e.g. a list of key/value pairs) without having to try to create a complex interface using the standard text boxes. An example of using a table is the User Defined Variables config element (although this doesn’t actually use the table editor that’s available to test beans).

To use the table editor, you must define a class the represents a row of data in the table. This class must have a public, zero argument constructor and get/set property functions for the columns you want to edit. In addition to this, the class must also extend the AbstractTestElement class. This is what caused me a lot of problems initially as it’s not mentioned in the docs and although it may appear to work if you don’t do this, you will run into problems such as breaking the JMeter GUI or not being able to save your tests.

Once you’ve extended that class, you must also make sure that you save any of your properties in the AbstractTestElement properties map otherwise they won’t be saved to the JMX file or passed to your components correctly when you run your test. In my table editor sample, I used the following class for my row data:

[code language=”java” firstline=”23″]
// A class to contain a variable name and value.
// This class *MUST* extend AbstractTestElement otherwise all sorts of random things will break.
public static class VariableSetting extends AbstractTestElement {

private static final long serialVersionUID = 5456773306165856817L;
private static final String VALUE = &quot;VariableSetting.Value&quot;;

/*
* We use the getName()/setName() property from the super class.
*/

public void setValue(String value) {
// Our property values must be stored in the super class’s or they won’t be saved to the JMX file correctly.
setProperty(VALUE, value);
}

public String getValue() {
return getPropertyAsString(VALUE);
}

}

[/code]

My property functions in my component are:

[code language=”java” firstline=”47″]
// Our variable list property
private List&lt;VariableSetting&gt; settings;

public void setVariableSettings(List&lt;VariableSetting&gt; settings) {
this.settings = settings;
}

public List&lt;VariableSetting&gt; getVariableSettings() {
return this.settings;
}
[/code]

Configuring the table editor

The table editor is configured through property descriptor variables in your bean info class and so is really easy to set up. The only additional work you need to do is manually request the localised strings for your column headers from your resource file. Here’s my bean info file as an example:

[code language=”java”]
package org.adesquared.jmeter.config;

import java.beans.PropertyDescriptor;
import java.util.ArrayList;
import java.util.ResourceBundle;

import org.adesquared.jmeter.config.ResetVariablesConfig.VariableSetting;
import org.apache.jmeter.testbeans.BeanInfoSupport;
import org.apache.jmeter.testbeans.gui.TableEditor;

public class ResetVariablesConfigBeanInfo extends BeanInfoSupport {

private final static String VARIABLE_SETTINGS = &quot;variableSettings&quot;;
private final static String HEADER_NAME = &quot;header.name&quot;;
private final static String HEADER_VALUE = &quot;header.value&quot;;

private final static ArrayList&lt;VariableSetting&gt; EMPTY_LIST = new ArrayList&lt;VariableSetting&gt;();

public ResetVariablesConfigBeanInfo() {

super(ResetVariablesConfig.class);

// Get the resource bundle for this component. We need to do this so that we can look up the table header localisations
ResourceBundle rb = (ResourceBundle) getBeanDescriptor().getValue(RESOURCE_BUNDLE);

PropertyDescriptor p;

p = property(VARIABLE_SETTINGS);
p.setValue(NOT_UNDEFINED, Boolean.TRUE);
p.setValue(DEFAULT, EMPTY_LIST);

// Set this property to be edited by the TableEditor
p.setPropertyEditorClass(TableEditor.class);
// Set the class that represents a row in the table
p.setValue(TableEditor.CLASSNAME, VariableSetting.class.getName());
// Set the properties for each column
p.setValue(TableEditor.OBJECT_PROPERTIES, new String[] {
&quot;name&quot;,
&quot;value&quot;
});
// Set the table header display strings
// These must be read directly from the resource bundle if you want to localise them
p.setValue(TableEditor.HEADERS, new String[] {
rb.getString(HEADER_NAME),
rb.getString(HEADER_VALUE)
});

}
}
[/code]

And how the table editor shows up in the JMeter GUI:

And once again, you can see how easy it is to extend JMeter with your own components when you know how.

A final note on type erasure

As I mentioned earlier, the table editor only works due to type erasure and I wanted to look at it quickly as it’s something that catches quite a few people out. In Java, type erasure is the loss of generic type specification at runtime. A List<String> becomes a plain old List. This means that at runtime, it is possible to assign a List<String> reference to a List<HashMap<Integer, Boolean>> reference without any problems until you try to get a value from the list. Often, the compiler can spot these problems and will report an error, but as with anything, it’s possible to trick the compiler.

Here’s the code from JMeter that takes advantage of this:

[code language=”java” firstline=”285″ highlight=”299″]
/**
* Convert a collection of objects into JMeterProperty objects.
*
* @param coll Collection of any type of object
* @return Collection of JMeterProperty objects
*/
protected Collection&lt;JMeterProperty&gt; normalizeList(Collection&lt;?&gt; coll) {
if (coll.isEmpty()) {
@SuppressWarnings(&quot;unchecked&quot;) // empty collection
Collection&lt;JMeterProperty&gt; okColl = (Collection&lt;JMeterProperty&gt;) coll;
return okColl;
}
try {
@SuppressWarnings(&quot;unchecked&quot;) // empty collection
Collection&lt;JMeterProperty&gt; newColl = coll.getClass().newInstance();
for (Object item : coll) {
newColl.add(convertObject(item));
}
return newColl;
} catch (Exception e) {// should not happen
log.error(&quot;Cannot create copy of &quot;+coll.getClass().getName(),e);
return null;
}
}
[/code]

What this function is doing is accepting a collection of any type and converting it to a collection of JMeterProperty objects. However, on line 299, a new collection is created using the class of the collection that is passed in. This new collection is then assigned to a Collection<JMeterProperty> reference. When using the table editor, the collection passed into this function will be of whatever type you have used for your table row data (Collection<VariableSetting> in the case of my sample). The compiler has obviously picked up on creating and assigning an object this way as being unsafe which is why the @SuppressWarnings annotation has been added.

However, this isn’t a mistake or bug in the code, this is quite intentional and a clever way of constructing a new collection with the same implementation as the input collection. Collection<E> is just an interface and so you can’t construct a collection object directly. Instead, you must construct something like a HashSet or an ArrayList, objects that implement the Collection interface. The function above has decided not to make any assumption about which implementation the normalised collection should use and instead constructs a new collection using the input collection’s class.

So, if the input collection was of type ArrayList<String>, the collection that is returned will be ArrayList<JMeterProperty>, the same implementation but storing a different object type.