To record or not to record, that is the question for record and playback test tools.

Introduction

A less technical blog post for once! The subject of recording tools is a touchy one because of its foundation. Experience level, the tool itself and your personal preferences all count towards the decision to use a recording tool or not. I will try to keep the points that I bring in this post objective and will lean heavily on past experiences, the training I give and the tales of others.

Perhaps a good starting point would be explaining what a recording tool is. For this blog post, we’ll focus on recording web pages in the Selenium standard style. Any other tools that can record traces, such as Jmeter are out of the scope. Maybe they can be handled in a future post by a colleague more intimate with performance testing.

A recording what?

The term ‘recording tool‘ is used to point towards a set of applications such as the Selenium IDE which is available as a plugin for Firefox and Chrome. I will also refer to a less known recorder buried deep within the confines of the Quamotion WebDriver developed by our friends at Quamotion. In the basics, both these recording tools follow the same workflow. You press the record button, go through your test case, press the stop recording button and, voilà, a play button becomes available for you to click. Upon clicking this button, the steps you manually executed while the recorder was running will be repeated. Perfect right? No, not really. But it’s not all bad news, there are some upsides to using a recorder too.

Cons of recording

As you might already have guessed, I’m not a fan of recorders as discussed above. They seem easy to work with, in such a scenario, but there are pitfalls that will quickly start causing headaches and cost a lot of time.

I think the downsides of recorders are best summed up by a little (true) story. Once upon a time, I was sent to a customer to have some exploratory conversations about the test setup they already had inhouse and how they wanted to evolve it. Their testing consisted of about 20 test scenarios which they would run after each deploy by someone who already had a full-time task. They had, of course, found the Selenium IDE, quickly tried it and concluded testing was easy. The quick start which the Selenium IDE had, ground them to a halt just as fast.

As above, they started by recording all the scenarios and then replaying them. During the first week, they recorded and replayed the scenarios and everything worked. The week after however, everything broke and none of their tests still ran. What happened? Simple: developers pushed a new version and exactly one element had its place changed. This causes the DOM tree to be different and thus the Selenium IDE couldn’t find the element anymore at the place where it was previously recorded.

Then, they decided to implement a bandaid fix: re-record the scenarios every week, then replay them. Upon hearing this, I was baffled. They had done it, created “automation” which was exactly equal to manual testing (the record step) and then made it worse (longer) by replaying it again. Recording and replaying these 20 scenarios cost this person one day per week to achieve. To finish it off, there was no CI, so the replaying was done manually, by hitting the play button. I’m sure some of these points could have been mitigated by putting someone with knowledge on the job but at that point, I’m sure a more knowledgeable person would scrap Selenium IDE anyways and put something more robust in its place.

To sum it up, recorders are a bad idea because they are:

• Initially fast to record but very intensive to maintain, thus costly and time-consuming.

• Breaks all the time due to a funky way of finding elements.

• Pain to implement into CI.

Pros of recording

After reading the negatives about recording, you would think that I would never use it. Wrong! I’ve used recording before but not the way of the Selenium IDE.

This is where the Quamotion WebDriver steps in. While the basics were the same (record, do your stuff, press stop record), the Quamotion Webdriver has different recording modes which makes it more of a tool for helping you find sweet selectors and less of an “IDE”, in which you have everything and the kitchen sink integrated except the generated code you need to see. In one of these modes that I would call half-automated-recording, the webdriver shows you a tree of elements and a preview of the page.

Clicking within the page shows you which element in the DOM tree you clicked and an xpath gets suggested to you. You don’t have to stick to that xpath however, you can modify it on the fly and instantly see the effect it has on the element you want to select. Next, there is a dropdown with all the actions one can do on the selected element. That’s all. After that, a line of code will be generated that can be straight copy-pasted into a script (it can export Java, Powershell, and many more languages). There are a few upsides to this approach:

• Clearly a tool: it doesn’t want to do everything but does one thing great.

• Rapid prototyping: need to throw something together quickly for a demo? Use this!

• Fast setup times: a few clicks and you have something to replay.

Conclusion

Recording has its very specific use cases. Use it as a tool to help you automate, not as your main test framework (this will be costly in the long run). Do you need something quick and dirty to convince someone of test automation? Then go ahead and use a recording tool.

Perhaps it’s better to explain it using an analogy:

When one has a hammer, everything looks like a nail.

When doing hardcore automation, a hammer, or in this case, record-playback tooling is not the solution.