Tips and tricks on writing better end to end tests

28 min readApr 22, 2019

At the beginning of this year I held an end to end testing workshop together with one of my colleagues. As a preparation for the workshop I wrote several pages of what I wanted to present that I later decided to put in an article. This is the result and although it’s a relatively long read, I hope you’ll find it useful.

The accompanying code with examples in NightwatchJs can be found there: https://github.com/cdohotaru/e2e-testing-public

Testing software code is an activity that is part of routine software development. Writing code without testing it is bound to failure on, at least, longer time frames. In this article I will talk about some common-sense principles to use when writing good end-to-end tests that I gathered along my programming years. Some were hard learned lessons while some might seem so obvious when read on a paper but not so obvious during the tunnel view that many of us suffer from while developing software. The focus is on the front-end side automation and some examples are given in a React based mock web-app but should be easily followed in other technologies.

Here’s a summary of what is being covered:

The testing pyramid
Page objects
Tips and tricks
Learnings from a recent migration

The testing pyramid

The testing pyramid is often used as an example of good balance of tests. While each team may have a different percent of unit, integration and end-to-end (e2e) tests that they are comfortable with, there is a general acceptance that the principle is solid. Does it mean that is followed? Not really. Quite often the pyramid will look like an inverted one — much more e2e tests than integration and unit tests — or like an hourglass meaning that there are a lot of unit tests and e2e tests but not so many integration tests. See the image below to get a quick idea.

I worked on projects where everything — yes, no exaggeration, everything — was meant to be covered by e2e tests. You could imagine that the build was talking hours to finish and would usually end up with checking the failed tests manually and deploying anyway. I believe one of the reasons why the pyramid is often inverted or the hourglass shape is because people get hang up on using one library/framework or 2 without exploring other ways and you know the saying: if all you have is a hammer, everything looks like a nail. This is one of the reasons why I decided to write this article: way too often times I spend way too much time debugging and fixing e2e tests. So if we keep on braking the pyramid pattern, let’s at least follow a set of common-sense principles.

One aspect that is often not so well treated in the literature is why the test suite should even look like a pyramid. So let’s try to understand that before moving on. Without going into too many details let’s give these short definitions for the 3 types of testing.

Unit tests

Focuses on the smallest unit of code under test, usually a function or a class depending on the language
Is fast to run, one can have all the unit tests run after every code change automatically without being bothered by that
Is simple by nature due to its reduced testing scope
On failure, finding the issue due to its simplicity and scope is a usually a fast task
Very fast to write and maintain

Integration tests

The generally accepted definition of an integration test is that it tests the interaction between two distinct entities; those entities can range from two classes or two services depending on your design/architecture granularity
They are slower than unit tests usually several orders of magnitude because more things needs to be setup or mocked before being run
Their testing scope is bigger due to their nature
They are more complex than unit tests again due to their nature and scope
On failure it is more difficult to find the issue because the failure range is bigger due to the bigger interactions
They are harder to write and maintain because of more advanced mocking or setups needed for them to be run

End to end tests

The scope of these tests range from a simple flow (login into a website) to more complex scenarios like put a product in a shopping cart and go to checkout; generally the scope is very big in comparison to unit tests and integration tests
They are quite slow to run because they involve building and deploying, firing up a browser and click around the website; in the checkout example above since this is on a user’s account it involves logging in (maybe even account creation before) and then navigating through the pages; it generally means that all services — pieces of the website under tests at least — must be up an running
The scope of the test is huge by comparison to unit tests or even integration tests; this is likely one of the reasons why many teams decide to test functionality that is much easier to test with unit tests directly in the browser: if I fire up this big setup then I should at least test as much as possible in one go. This is a wrong approach from many aspects but we’ll talk about this later.
On failure finding the issue is much more complicated than in the case of unit or integration tests; say the total amount to pay on the checkout page is not correct and thus the test fails, what are possible reasons for that? Many actually! The product didn’t load correctly, the product was not added to the shopping cart, the addition of the total amount is wrong; the display of the value is wrong and so on. There is myriad of issue that might have happened before the test error-ed
This is by far the hardest type of tests to write and maintain (from the 3 types considered) for all the reasons above and the ones we will be talking further

This gives us a clue as to why the testing pyramid should be the way it was designed. The cost for writing and maintaining tests rises with their complexity. And the complexity rises from unit tests to integration tests and then to e2e tests several orders of magnitude due to the several reasons like scope, setup, complexity, time of execution an so on. In the test type comparison image below we summarise these aspects.

There is the idea that the acceptance tests offer more coverage because they test the whole system. This is a flawed idea in the sense that although the system under test is the whole system, achieving the same coverage with e2e tests versus unit tests would mean a huge number of e2e tests that would rise the writing and maintenance cost extremely. This huge cost would offset the pseudo-advantage of whole system coverage via e2e tests. Besides this, how would you measure it? There are plenty of tools to measure test coverage for unit tests but I’m not aware of such tools for e2e testing and I fail to understand how would they work accurately (the unit tests coverage is measured by inspecting the source code and verify how many of the code conditions are covered by the tests, but this seems a huge task, maybe even impossible with e2e tests given that they test via the graphical user interface and not calling functions directly).

To finish with this section let’s try to conclude some things:

Writing, running and maintaining e2e tests is costly
Given the associated cost, the decision to add e2e tests should be weigh carefully: are there other types of tests that would suit better?
Consider the reason of being for an e2e test: your are testing a certain flow or scenario and expect that all pieces under test work correctly but the focus here is on high level interactions (e.g.: testing a login form input validation — that is, that an error message appears when you don’t type a password or a username after clicking the login button — is not the job of an e2e test; but you are interested that after successful login you are redirected to the user account page and that is the job of an e2e test). The assumption here, and I think is a fair one, is that the empty field and other types or form validation don’t require a server round-trip and thus can be unit tested.
More tests don’t always mean more coverage -> test coverage is not a subjective concept but rather a measurable metric

Page objects

So we are writing e2e tests. Not too many and only after careful consideration, but how do we even write them? The scenario is clear: I want to make sure that the user can add a product to its shopping cart and get to the checkout page correctly. So let’s break this into some parts:

We obviously need a user account
We need to login with our user account
We need to go on a product’s page (let’s say directly and not through search to keep the example simple)
Add the product to the shopping cart
Go to the checkout page

We will talk about the user account creation later but we will have a point about it here as well. So what would be the obvious coding steps to do:

Put the login page into the browser’s address
Locate the username, password and submit button elements and interact with them: input the username and password and click the submit button
Let’s say that after login we end up on a product list page so here we need to locate our product and click on it to go to its own page
We are now on the product page so we need to click on buy and validate that the shopping cart item count was updated
We now click on the shopping cart icon and go to the checkout page
Here we validate that the product info that we chose is visible and no other product info is present, we check that the total amount to pay is correct and if everything is fine we end the test
At any of the steps above of the conditions that we expect are not true, we fail and finish the test

There are libraries out there to help you start a browser and manipulate it (locate elements, inspect them, click them etc). The most widely used are based on Selenium or the WebDriver standard. It is beyond the scope of this article to show how to setup them but you can find in the mentioned repository and example of using NightwatchJs (an automation solution that can work with both Selenium and different WebDrivers).

After you picked up your tool you are to start writing the test code. What often happens at this point is that the engineer simply writes the code to do the job. By this I mean that soon there will be plenty of small functions that are doing a certain thing like locate and click whatever element, inspect the attributes or another and so on. These functions will likely use the low level APIs of the library thus using the webDriver object all over the places. If the engineer is thoughtful in his process there might be some sort of structure in the code like certain function types are in a certain folder and certain assertions are in a different folder and so on. Or the low-level library API calls would be structured in specific files or isolated from assertions. But more often than not, what happens is that a lot of people with different skill level work on that repository without giving too much attention to the testing code and consider it as such a second class citizen. “It is just testing code, here we can cut corners and write unstructured or less than ideal code”. As the project matures, there are now several functions with similar names that do the same or similar things. It is hard to understand and follow the steps in the test and hard to define boundaries between responsibilities like low level API calls, locators and assertions. From time to time there is some bigger refactoring that changes the HTML thus breaking a big amount of tests, creating a lot of tension (usually between quality assurance engineers and development engineers) and unintended work. Sounds familiar? If not, I am happy for you. But I’ve worked on too many projects that at different phases were displaying these behaviours. So, if this will not cut it, maybe there are some patterns that we can follow? Yes, there are indeed some. Let’s talk about what a page object is.

In a nutshell, a page object is a way of abstracting the inner bits of your user interface. That is, the HTML that forms your website is being abstracted away behind an interface and only that interface is used to write the test. It is an encapsulation technique that works well and even has support in some libraries or frameworks.

Let’s consider the login page in our test example. The login has a form with two inputs and a submit button. A page object for this form would include at least one method that accepts a username and password. Let’s consider this method signature:

loginWith(userName, password)

What happens when it’s called is this:

The username field is located on the page and the input is filled with the userName parameter value
The same happens for the password field
The submit button is located and is clicked
Potentially there is a wait for a landing page after login and then the method returns successful
Should any of the steps above fail, the method would just throw an exception that would finish the test with a failure; the actual exception would be thrown by the library that you are using, the function would just not catch it

Now let’s revisit the example we started the section with. These are the steps we had (I copied them here to be able to follow easier):

Put the login page into the browser’s address
Locate the username, password and submit elements and interact with them: input the username and password and click the submit button
Let’s say that after login we end up on a product list page so here we need to locate our product and click on it to go to its own page
We are now on the product page so we need to click on buy and validate that the shopping cart item count was updated
We now click on the shopping cart icon and go to the checkout page
Here we validate that the product info that we chose is visible and no other product info is present, we check that the total amount to pay is correct and if everything is fine we end the test
At any of the steps above of the conditions that we expect are not true, we fail and finish the test

Using page objects this is an example of what we could have:

Login using the login page object (it knows the URL, how to start the browser and how to check that the login succeeded)
Select the product we want and add it to the shopping cart (here we would pass a product id to the shopping cart page object that adds products to the shopping cart, the rest is taken care inside it: finding the product item, selecting it and so on)
Using the same shopping cart we navigate to the checkout page where we validate the data with the help of the checkout page object

There is a lot of explanatory text above but the test actually becomes much more readable and short. The complexity, or at least most of it, moves to the page objects but we are not bothered by that because that is their purpose.

The page object model is not to be regarded as an abstraction over a whole page, although the name implies it. It would be quite impractical to do so on a rich user interface. I tend to use the component object to make this obvious in my work but that’s because “component” in my project actually has a rather accurate meaning as the project is a React based one. But regardless of how you call it, keep in mind that is not meant to abstract the HTML interaction over an entire page.

Let’s find some other examples in our test. After logging in we will be redirected to the product list page. Therefore a page object abstracting this product list (here I mean the actual list) could be a good idea. The product list is likely part of a richer user interface: there is a header and a footer, somewhere there is the shopping cart that shows how many products you have in your basket, maybe some filter for the products and so on. Each of these logical parts of the product list page can be abstracted behind a page object. So in order to interact with the product list we would use the right page object or maybe more of them. We could click the Buy button on a product via the product list page and check that the shopping cart product count is updated via its own page object. One would interact with these objects at the test level and not necessarily inside another page object. But in practice there might be situations when one page object is used from inside another. Maybe the shopping cart is part of the page header so we would create a shopping cart page object that we would then “import” and use from the header page object.

I guess you can see where this is going. We are abstracting the page HTML behind sets of functions with clear contracts. The benefit becomes obvious soon: a change in the HTML would mean a change in that page object only. So, say the selector for the username or password changes: there is only one place where this needs to be changed and that is in the login form page object. You refactor the HTML for the product list to use a different way of showing the products. That’s right, you only need to change this in the product list page object. Every HTML or CSS change has usually a very limited set of changes in the page objects. And these page objects can be used like puzzle pieces to create e2e tests. You will quickly realise how this speeds up the test writing and maintenance as it brings structure in your tests.

Before we conclude this section let us mention one important thing: a page object is a pattern and not a library or framework functionality. This implies that when used, the page object should abstract away also the technology that you use behind the scenes. If you are having an in-house locator and DOM (Document Object Model) manipulation solution or you use Selenium or the WebDriver standard API it shouldn’t be relevant or visible outside of the page object. You should be able to change the underlying technology used for starting and controlling the browser by updating only your page objects. So a key benefit of using the page object pattern is that your tests can easily be migrated between technologies.

A word or warning: some consider that the page objects should not contain assertions. That is, they should return values that are asserted back into the main tests. I don’t believe there is something wrong having assertion functions inside the page objects (one could argue that even waiting for an element to appear is a shallow form of asserting that the element is there) and the code examples are written with this idea in mind. Feel free to structure your tests as you and your team find it fit.

So let’s sum up some key aspects of the page objects:

They abstract away HTML and implementation of a certain user interface
They are generally built upon logical separation of the page: footer, header, loginForm, checkoutForm etc and don’t abstract an entire page
They don’t combine functionally unrelated user interface parts
They don’t give access to the underlying technology
They allow for structured, easy to maintain tests and migration between technologies

Tips and tricks

If you are using the page object pattern you are up to a head start. But there are some other things that you can do to improve your tests. Let’s dive into them.

Data-test-ids

Locating and selecting HTML elements is at the base of any e2e test. There are various ways I’ve seen people try to accomplish this: xpaths, element ids, classNames, CSS selectors. By far the best approach is to use data-test-id attributes on elements. Let see why:

XPaths are heavily HTML order dependent; a refactoring of the HTML could easily break an XPath leading to more work to maintain the tests; just avoid them
Element ids were heavily used in the jQuery era; every tiny element would have an id but imagine a very rich user interface in a product developed by multiple teams: how do you make sure they are unique and your selector would not pick a different element than the intended one? You could for sure find different solutions that would take you only so far but the same refactoring would easily change the ids to something else to match the new code and thus breaking the tests
classNames are among the things that are rapidly changed when refactoring and they are easily used in several places because their job is to style with the least amount of code and not to be there for element location and selection
A dedicated data-test-id attribute (see data-* attributes) is therefore one of the best approaches for the issues above: refactoring something that contains such an attribute alerts already the engineer that there is a test that might need a change and the sooner you make that change the less costly it will be; it doesn’t matter too much if it is not unique because it can be concatenated (e.g.: div[data-test-id=”listA”] [data-test-id=”item1”]); you can argue that one can do the same with ids but doing so losses the visibility that that element is used in a test

Account creation

Many websites require an account to be used. Thus, the tests would also need to make use of accounts. There are couple of good principles that can be applied here as well:

Keep account creation independent from tests; this means that you could have a script which creates X number of test accounts that are then used during testing; creating an account before each test is a time consuming operation and remember that you should also clean up after your test which should include account deletion
Regardless of how you create your test accounts you should isolate the creation in such a way that you can return a unique account for each test; this can easily be achieved by using a random email address generator (or string generator for just the first part before the @domain); this is extremely important if you intend to run your tests in parallel: the last thing you want to do is debug a test that never fails on your machine but fails from time to time on the build agent because the tests accounts are not unique
Test account creation is sensible to the registration process: if your registration is laborious, just create them up front, if is just a small step you can also do it before the test starts but make sure your account is unique

Test suites versus independent tests

The e2e tests are usually testing a certain flow or scenario as we described above in the login -> product list selection -> checkout example. There is usually enough complexity or length in even moderate scenarios that shout out to not add more assertions on top of that but the temptation is big to keep adding tests because “it’s expensive to open up the browser”. This ends up having big tests suites that just test too much and makes tests too coupled. An example would be to start to test item removal on the checkout page, then again addition of 2 products and other two operations that might make sense from a business perspective but not necessarily in the same test. The problem with these long suites is that they become hard to maintain. Just imagine how happy an engineer would be to fix the last test in a 10 test suite: run all the 9 tests just to debug the last one because each test depend on the actions done by the test that run just before it. Keep tests as short as possible and as independent as possible from each other.

Test setup via API and not UI

It is often the case that tests have certain setup requirements. In our example there are several: the test account must exist and there must be at least one product on the product list page. And just in the last paragraph I mentioned that the tests should be independent as much as possible. The best approach here would be to prepare your test environment (create the account and insert some products into your database or just mock them if possible) via the API and not by using the user interface. In some cases, like adding a product, this might not even be possible via the user interface unless you use maybe some administration page. The point here is to keep the things as simple as possible: there is less likely for an API to break a test then it is for the user interface and you will likely save a lot of time both in writing and in running the tests.

So, getting back to the test dependency above: if you need to test item removal from the shopping cart you can add the product in the cart during the setup and just go directly the the checkout page rather then do all these operations via the user interface. Regression tests are maybe an exception to the rule here: if there was a bug that was happening after a certain sequence of steps then just follow the bug reproduction steps as funny as they might be. But a good rule would still be to keep this test separately and not just add to existing suites.

Avoid conditional statements

This one comes from unit testing. While learning how to write good unit tests I read, sadly not in enough books, that conditional statements are to be avoided. I didn’t quite understand back then why is that so important. But with time and via my own mistakes, it became obvious: a conditional statement (like an if statement) is not one unit test anymore. It’s two tests. One for each branch of the if statement. Therefore it makes sense to write 2 unit tests and not one. Besides this, the actual business logic code is complex enough to understand and follow. Why add more complexity in the testing code? The tests should be as easy to read and understand as possible. An if, for, while statement is taking away from that readability and invites for even more complexity. Don’t think only at yourself, think also about who is going to maintain that code in the future. I believe this great principle for unit testing should be followed also when writing e2e tests. So please don’t use conditional statements.

Screenshots and logging

Tests fail. No news, I know. But we should know as much as possible why the test has failed. Otherwise we might debug in the wrong location. Most testing products, Nightwatch and Selenium included, offer the possibility to take screenshots during tests. A good practice is to take screenshots on test failure so that you see how the screen looked like when the test failed. (Something that would be interesting to try here is to take screenshots after each action you do in a test. This would create a lot of images but after a big CSS refactoring this would be very helpful to make sure the user interface is showing up as expected). Equally important is the logging. Here the libraries differ when it comes to the relevance of the logged information. Some would just print out the stack trace which is not always very telling while some offer a bit more context. What you aim for is to see assertion failures because that tells you what you instructed the test to tell you. So make sure you write good assertions and given enough context in your messages to find the problem fast.

Screen resolution

This one is a short one: some tests require a certain resolution to work fine. That is part of your business requirement and what one usually does is to tell the driver (of your testing library) to start the browser maximised or at a certain resolution. Most times this is enough but depending of your testing environment your tests might be running inside services on the build agent which don’t have screen information attached to them. So the better approach is to specify an explicit screen resolution. If happened just recently that some tests were failing on the build agent but not on my machine. After looking at the screenshots (see above) I realised that the screen resolution was too small because some Gitlab configuration was not properly updated.

Sleeps and waits

This is one of my favourite. When a test fails, just increase the waiting time, right? Wrong. Very wrong! But yet a lot of engineers do this in the hope the a non deterministic problem — when will an element be visible — will be solved by adding even more non deterministic context to it. By far the most often failure in tests is because some element didn’t show up (or disappeared) after a certain amount of time. We can’t wait forever for an element to appear so then what is the right amount of time to wait?

There is no golden rule because there are many variables at play but here’s what you can do. Some of the libraries, Nightwatch and Selenium included, offer the so called explicit wait. This means that the driver (of your library) will wait for the element/condition a maximum amount of time given by this explicit wait value but if the element is found earlier or the condition is met earlier it will continue. This is not to be confused with the implicit wait which waits for the amount regardless if the condition was met earlier.

The problem of waits is a quite complex one. Thing about the context for a second: you try to run as many tests as possible on the infrastructure that you have. You often run the tests in parallel to achieve short run times. In the same time there is an overall system pressure from databases, services and what not on your infrastructure that you have to cope with. The networking comes on top of that and your website uses AJAX calls to talk with the server. There are a lot of variables at play so don’t get too frustrated if it takes time to put all these things in order. It is a complex problem. But not always the solutions are also complex. Simply separating the tests to be run sequential or in parallel (depending on the overall system pressure each of the setups creates) will achieve more than the other solution. You could use the implicit wait if your infrastructure is too overloaded but this is just postponing the inevitable hardware upgrade or a complex refactoring.

There is another type of wait, the pause (something like Thread.Sleep(some milliseconds here)) which is generally a bad idea because the explicit wait covers this in a much flexible way. If you use animations then pausing the execution for the duration of the animation can actually help. But don’t abuse this as you won’t achieve but non-deterministic failures for your tests.

Plan the testing and write tests yourself

This will gone sound like a cliche but the tests should be part of story planning. That is, the same way you — hopefully — get together with your colleagues to break down a story and write tasks, the same way you should do for the tests. Here are some obvious advantages of this:

It is clear upfront what should be unit, integration and e2e tested; in the example of the login form above we said that the form input validation is not the job of the e2e or integration testing but purely the job of the unit tests (this assumes a certain implementation but this is what should be discussed during the planning); if no discussion or decision about the tests happens it is possible that some e2e tests will be written for something that unit tests would have sufficed
Ideally the e2e tests don’t depend on implementation details but in practice there is real benefit in knowing how a certain implementation works (I give two examples in the learnings section below); this can prove very time saving if the coder and tester are different persons
Code tends to becomes cleaner if it cares for testing because the way the relationship between logic and mocking the logic’s dependencies work; if you think upfront about how you’re gone test that code you are likely to write it in a more testable manner and thus more structured; this has maybe more validity in the unit tests world but I believe there is value also for the e2e tests
Not seldom it happens that you find bugs in the implementation while writing the tests thus increasing the code quality
You become a better developer; this is what I noticed time and again in my development career: I started to think about code from different perspectives like public versus private methods (relevant for unit tests), contracts and interfaces, environments (different browsers, resolutions, networks) and libraries and frameworks; it is not an understatement to say that you become a better coder by writing tests yourself but I wouldn’t blame anyone thinking otherwise given that I thought the same for a while :)

Testing code is no second class code

We touched on this point earlier but I feel there is value in emphasising it more. Quite often it happens that the testing code is not getting the same attention that the production code (application code) is getting. I’m not sure what are the reasons behind this but I see it happening also in mature teams and projects. An application is worthless without it being tested. You can’t expect the users to test your app. And if that’s the case then why make our lives more complicated by writing sub optimal testing code?

It you consider the project a living repository, so it is for its tests. Almost as often — if not even more often if your testing code it poorly written — as the production code is modified, the tests will need to be updated or even new tests will need to be written. But a lack of dedication to write good code for tests will lead to a quality gap that will likely be covered by long time investigating why the tests are failing and then throwing even more sub optimal code at it just to get over it. It is a slippery slope and I’ve seen big amount of tests — and implicitly time and resources — simply being abandoned because nobody would want to invest the time in fixing them.

I want to close this section with 2 remarks.

What I suggest here is not rocket science. These are just practical, common-sense things that you can follow when writing your tests. The difference, at the end, will be made by how consistent you will be in following all this and discovering more and better tips that work for you and your team.

The last remark is this: it is not always going to be easy to just take some principles and start applying them in your projects. Software engineers, and here I mean both quality assurance and development engineers, are dedicated people. This has sometimes the side effect of making them being afraid of change being it in code, process or day to day routine. It can be challenging to have your arguments heard even when working in mature teams where communication is fluent.

Equally challenging can be to let go at things that you invested in, like an in-house library or certain ways you write the tests, even when you see they are not necessarily the best in town. I myself wrote a library to run Selenium tests across several machines in parallel to suit the infrastructure the company was using only to have it let go some years later. I also changed libraries and frameworks, not to mention styles, quite a few times in my career already. It takes an open-mind and lots of patience. Don’t give up! :)

Learning from a recent migration

Not long ago I fixed and migrated (from a library and testing style to another) some tests that were blocking the release process on our continuous integration environment. These are some learnings from this migration that might help you in your test writing. They are pretty particular to that project but try to focus on the idea.

Testing performance should be done separately from the other tests

We had this situation where you could invite 100 users in a form. After invitation the form would remain open until the response would be processed and came back from the server. The structure or the test and what exactly needs to be tested was not discussed in the planning so the tester wanted to cover the acceptance criteria and used 100 random email addresses for the test. The problem is that the server processing part used in the continuous integration system was quite problematic and couldn’t easily handle 100 users at a time back then. This made the test fail almost always. You can probably see some design issues here already like why waiting (blocking the user) within a modal window for the operation to finish and you would be right to spot that. Sometimes you fix a test by changing the design like do the processing in the background. But what we did was to decrease the number of users to just 5. You could argue that this is a workaround but I would argue that performance testing — that 100 users can successfully be invited — via the browser didn’t make sense in that case. An API test in a batch of performance tests is the right way to test this. But for our continuous integration system we only want to know if the logic works as expected. We don’t need 100 users for that.

Overloaded page turns unresponsive

In one situation we would start the test on a page only to navigate away to a subpage. The initial page would load the necessary JavaScript for the app to work and it was the starting point of the test. Because the data was loaded with several AJAX requests fired on various conditions, for a short while — between a couple of milliseconds and up to one or two seconds depending on the environment like local machine or the build agent — the browser/page would become unresponsive. The navigation away from the page would be done via the browser driver, Chrome in this case, but because it was busy handling the responses from the AJAX requests and drawing the DOM and so on, the request would simply be ignored. We fixed this test by waiting for certain element to appear on the page. This element would only appear after the AJAX requests would be handled so there was nothing left to choke the browser during the test. A note here: all these things would happen very fast on the build agent. That is, the element would be clicked very early, basically immediately after the page was loaded. The browser drivers used in the test libraries can be very fast nowadays.

A variation of the problem above was in a test where we would restore a deleted item and then we would navigate to the restore location to see if the element was indeed restored. What was happening here is that the navigation from the initial page to the page where the restored item would appear was happening so fast that on the server side the element would not have been restored yet so it wouldn’t appear in its restored location. There is a notification that appears after the successful restore is done. This notification, as an implementation detail, would appear only when the AJAX request would return, it was triggered by the AJAX callback. So we fixed the test by waiting for this notification to appear before we would navigate to the restore location. There are 2 important things here: give time to the system under test to actually finish the operation (and check if that’s the problem when the test is failing) and second: should the tester didn’t know about the implementation detail above — that the notification would appear only after the restore would be done — if could have taken him some time to figure out how to fix the test.

If you want to read more about these topics I recommend you have a look at these links:

https://martinfowler.com/articles/practical-test-pyramid.html

https://testing.googleblog.com/2015/04/just-say-no-to-more-end-to-end-tests.html

https://martinfowler.com/bliki/PageObject.html

https://github.com/SeleniumHQ/selenium/wiki/PageObjects

https://docs.seleniumhq.org/docs/04_webdriver_advanced.jsp#implicit-waits