Opinions on Automated UI/Web Testing Frameworks

[Archival]

As talked about in WebDev WAYWO this is a PSA regarding the differences and capabilities of two of the major automated testing frameworks (Telerik Test Framework and Selenium)
I felt this deserved its own thread instead of existing just there due to the size of this and the ease of managing questions.
This is of course my personal experience which will differ from others, if you disagree with what I'm saying feel free to dispute that and reason why, don't just rate this dumb or disagree.

For anyone that wants some info without the essay there is a TL;DR at the bottom.

First off a bit of backstory. 5 months ago I started a new job which involved me writing and maintaining the automated tests that are run on a web application daily (Among other things).
During this time I have primarily used Telerik's 'Test Framework' or more specifically, Test Studio which utilises TTF, as well as a small amount of selenium through one of my colleagues.
While this is going to cover TTF and Selenium im going to make a mention of Telerik's Test Studio at the end as a PSA.

One thing to mention before I dive into this is that while these frameworks can cover a few different types of testing, they may not be the best tool to choose outside of that they excel at.
Both of these frameworks (Selenium and TTF) perform simulated user interactions as their primary function, whether you use this for integration or regression testing (or anything else that uses UI interaction), they both do this well but are each better in some ways than others.
[b]API testing is not what these are for[/b] (There are other ways to do this, use them instead)
These frameworks are designed to test from the users perspective, and to test what the front end receives and displays is correct.
They are also very good at finding workflow errors due in part to their ability to churn out potential user interactions. (data that becomes erroneous over a full workflow but is effectively correct per call)

Now on to the fun stuff.
The main differences I have found between TTF and Selenium is basically OSX versus Android. One overall feels like its designed to be quick and easy to use but can be frustrating to customise because of it, and one is designed to be very malleable but also time consuming to get everything working due to the expectation that the user will do it all.

So, whats the same between the two?
Well at their core, they both function pretty much the same. Both of them follow the workflow of finding elements then interacting with them and validating the results. How they do this is the same, you 'get' the element from the DOM and then either interact with it, or extract attributes from it and validate them. The actual code for this will vary, but they can both find elements based on attributes or the elements index within the DOM.

For example, this is how one finds an element in TTF

Manager.ActiveBrowser.FindByExpression<HtmlDiv>("name=hello","class=tester");

And this is how one finds an element in Selenium

driver.findElement(By.xpath("//*[@name='hello'][@class='tester']"));

Pretty simple right?

Both of them will also expect you to write up the architecture of them yourself, this will be time consuming in both and there are many ways of doing so.
An example of this is the Page Object Model (POM) (For reference [url]https://www.guru99.com/page-object-model-pom-page-factory-in-selenium-ultimate-guide.html[/url]) which is effectively what TTF will implement if you choose to use the paid IDE discussed later.
Selenium has existing implementations that you can download via extensions to help speed this up, TTF's paid IDE straight up eliminates this step.
Effectively in the case of just using the extensions, you are required to choose how you want to structure your tests, and from there how they will run, report and handle failures.

Where they differ is primarily in the interactions with the DOM, how they handle certain validation and how much control over these processes you have.
For example: I want to click a download button and then handle that download dialogue.
In TTF I create a download handler object, push it to the managers list of dialogue handlers and then after clicking the button, tell it to handle it with a file destination (and then in another method, validate that it did actually download).

String path = "D:\\Downloads_sel";
SaveAsDialog dlg = SaveAsDialog.CreateSaveAsDialog(Manager.Current.ActiveBrowser,DialogButton.SAVE, path , Manager.Current.Desktop);
Manager.Current.DialogMonitor.AddDialog(dlg);
dlg.WaitUntillHandled(30000); //where the int is the timout

In Selenium I have a lot more options when controlling this such as showing an alert on completion, closing the window when it is complete and a ton more. This does make it more time consuming to set up the handler, but gives me a lot more avenues when it comes to what I want to happen. You also have direct control over browser settings such as Always ask for a save location instead of chromes quick download.

FirefoxProfile profile = new FirefoxProfile();
    String path = "D:\\Downloads_sel";
    String fileTypes = "application/x-msexcel"; //In this case, what types of files we want to ignore the save to disk dialog
    profile.setPreference("browser.download.folderList", 2);
    profile.setPreference("browser.download.dir", path);
    profile.setPreference("browser.download.alertOnEXEOpen", false);
    profile.setPreference("browser.helperApps.neverAsksaveToDisk", fileTypes);
    profile.setPreference("browser.download.manager.showWhenStarting", false);
    profile.setPreference("browser.download.manager.focusWhenStarting", false);
    profile.setPreference("browser.helperApps.alwaysAsk.force", false);
    profile.setPreference("browser.download.manager.alertOnEXEOpen", false);
    profile.setPreference("browser.download.manager.closeWhenDone", false);
    profile.setPreference("browser.download.manager.showAlertOnComplete", false);
    profile.setPreference("browser.download.manager.useWindow", false);
    profile.setPreference("browser.download.manager.showWhenStarting", false);
    profile.setPreference("services.sync.prefs.sync.browser.download.manager.showWhenStarting", false);
    profile.setPreference("pdfjs.disabled", true);

    WebDriver driver = new FirefoxDriver(profile);

An explanation on what some of these do at [url]http://toolsqa.com/selenium-webdriver/how-to-download-files-using-selenium/[/url]
Note: You won't need to change these settings every single time you want to download something if you structure your tests properly, this is just to illustrate what you have available to you!

Now this example is rather universal between the two, TTF is simpler to perform a lot of things but lacks the immediately accessible customisation that Selenium offers and has alot of these minor kinks that will trip you up from time to time. On the flip-side maintaining tests is a lot simpler in TTF than in Selenium as debugging in TTF will have a lot less variables effecting a given issue. With proper architecture you can minimise this difference, but for someone just starting out TTF will afford you an easier time.

The main selling point that sets these apart for me, has been the availability of support and documentation regarding both functionality and best practise.
I found while using TTF I more often than not, wrote something in what I thought was the best way to do so, only to find weeks later that there was a better way of doing it simply because I was not aware of something that the framework could do.
Selenium on the other hand has such a large amount of user questions/answers that you would be hard pressed not to find an answer on a question you have that will lead you in the wrong direction. This coupled with selenium having user written extensions gives you a far more powerful framework in my opinion.

To quickly touch on performance, TTF has shown to us that in general, it will run a tad slower than Selenium but there are many factors that can effect these results, in the end the performance of your tests is up to you. Best practises such as using steps that wait for an element to be visible rather than just waiting 10 seconds flat will improve your tests times but that's an entire write up on its own.

In the end while TTF is far faster to get new tests written which is fantastic for people that don't have a lot of developer experience (though it has a plugin for VS so go figure), you will have a far better time eventually with Selenium IF you have the time to set it up in the first place and spend a while researching the best practises. My colleague who has been using Selenium is unfortunately not a developer and such, has been having an extremely tough time getting it to work where with TTF he would have made far more progress during that time due to its simple to use methods.

[b] I will now whine/shill about Test Studio [/b]

The biggest saving grace that Test Framework has (and quite possible biggest stress generating part), is its existence within another of Telerik's products called "Test Studio" (paid product alert).

This is effectively Telerik's own take on an IDE which comes in two parts, a development platform and an execution server (which exists in the prior and as a standalone).
TS gives you the ability to organise tests ,automatically generates code and has an elements repository that stores all elements you use without the need to declare this yourself.
All of this is done through a drag and drop style interface.
While more often than not the auto generated elements and code will perform so poorly you will need to rewrite them yourself for them to actually work (see download handler/text entry/grid support/anything complex) the real gem here is its reporting and remote execution features.
You are able to run multiple execution servers on different machines (or VM's) and then have your list of tests run, split between your available machines and be given a nice report of the results, complete with exception outputs and screenshots of when errors occurred. This is extremely helpful as it means you need not worry about writing any of this yourself which saves a ton of time for setting up for a new project/for the first time. This is extremely useful for non-developers trying to perform the role of auto test as it does almost everything for you (when it works) and practically a must for any corporate testing if you would rather spend some money than have your workers write the thing from scratch (which if they could, you might as well use selenium if you have the time available)

[b] TL;DR [/b]

Multi browser support: Selenium does it better because extensions
Multi window support: TTF is a bit finicky but easier to do in
Multi tab support: Please don't do this (TTF has a bad time, Selenium idk)
Download/Upload support: TTF easier, Selenium more control
Find and store DOM elements: basically the same
Speed of writing/maintaining tests: TTF faster in both cases
Extensible and control over what your tests do: Selenium does it better
Documentation and support: Selenium by far as TTF is basically the API and that is it.

The rest is generally up to the user.
If you want to get in and do your tests quickly and without investing a lot of time in making sure your architecture is perfect, use Test Framework, it is in general faster to write and maintain.
If you want complete control over your tests, a community that has already answered practically any question you could ask and have enough time to get the ball rolling, use Selenium.
Essentially TTF can do anything that Selenium can, there are just many instances of it being finicky or requiring some workarounds to get that added functionality out of it, similarly Selenium can have its development time sped up through the use of extensions but given an equal skill in both, the stats remain around the same.

For the sake of anyone reading this from WebDev I would suggest Selenium, if you are already doing your own development chances are you can handle the added effort involved and will make alot more use of the extensions and capacity than someone who isn't.

For reference some getting started information, note that if you do begin using one of these be sure any tutorial resource you use was made within the past 1-2 years. While TTF has not changed a great deal in that time, Selenium has, so older tutorials are almost guaranteed to be out of date. There is also an almost endless supply of Selenium tutorials, so be careful.
TTF:
[url]http://docs.telerik.com/teststudio/testing-framework/getting-started[/url]
[url]https://automatetheplanet.com/getting-started-telerik-testing-framework/[/url] (This is one resource I personally used, found it very helpful to start with)

Selenium:
[url]https://wiki.saucelabs.com/display/DOCS/Getting+Started+with+Selenium+for+Automated+Website+Testing[/url]

[Tamschi]

This is alright as a short overview, but I think it would really benefit from a few links to the relevant documentation (e.g. showing a few examples of the test implementation in TTF vs. Selenium or how to get started with these tools).

[brianosaur]

Yeah, I'd prefer to see code for the differences. These testing frameworks are more for integration testing and not to be confused with unit tests. I was taught to test every bit of my code. You should be too.

[Archival]

[QUOTE=Tamschi;52785077]This is alright as a short overview, but I think it would really benefit from a few links to the relevant documentation (e.g. showing a few examples of the test implementation in TTF vs. Selenium or how to get started with these tools).[/QUOTE]

I agree it is missing alot, I was rather busy when I wrote it yesterday and probably shouldn't have submitted it in the state it was in, adding a bit more to it now once FP stops blocking me from doing so. I may come back and re-write some of it to be more informative but after my workload basically doubled (on the day I said i'd write this up too!) thats doubtful, it may be best to just try and answer any questions you guys have instead of changing this any further.

[QUOTE=brianosaur;52785504]Yeah, I'd prefer to see code for the differences. These testing frameworks are more for integration testing and not to be confused with unit tests. I was taught to test every bit of my code. You should be too.[/QUOTE]

Actual unit testing is better to happen elsewhere, where I work currently we perform unit tests both when changes are tested for merging with the master and whenever we run a deployment script to put the current version on any testing server for validation. There are use cases and regression tests that get run through both TTF and Selenium and then a few others run by a manual team.
I agree that you should test absolutely -everything- that you can, I once had a friends partner scoff at me for being in this role (and telling me, to my face, that what I was doing was not useful :v:) claiming the software development company he worked for had no testers because their developers didn't make bugs. Hearing that brought me physical pain.

[sarahemi]

Give your codes a brief and detailed audit, As I already know how annoying is that. But I've found this strategy very helpful and promising, and not only by developers, this is also a key practice for most of the software development and [URL="https://www.logodesigner.ae/webdesign"]web designing companies[/URL]. Thanks and share the ressolved issue, if it solves quick.

[highlight](User was permabanned for this post ("Spambot" - Sgt Doom))[/highlight]

[Umerali]

Its obvious, yeah it is to say that the codes not always work as we want them to. A developer is already has much stress of the work. As of the coding, the main flaws are found while we perform the QA of the software and programs. The entire process is very critical and time taking. [URL="http://www.found.com.pk/services/web-development"]Web Development Companies[/URL] must have some QA Specialist for their optimum testing and regular and scheduled check up.

[highlight](User was permabanned for this post ("Spambot" - Sgt Doom))[/highlight]