Selenium is a great open source project that can be useful for monitoring simple multi-page apps, but using Selenium for testing or monitoring SaaS servicea can be a challenge. Occasionally, our sales and engineering teams talk to prospects who say, “We have other fellas who write Selenium scripts, I’m just going to ask them to write tests for SharePoint Online, Microsoft Teams or Exchange Online”. We get it: you’re trying to save your company money. Or not invented here (NIH) syndrome. Or “our environment is too special and different, we need something custom”. We’ve got you, been there.
Read the rest of this article to figure out why it’s hard to use Selenium for testing cloud services and SaaS properties like Office 365. In the end, your organization won’t get the benefit and it will be more costly to write your own Selenium scripts for monitoring cloud services.
Quick Primer On Selenium
Selenium was originally developed in 2004 by Jason Huggins as a quick testing tool for HTML/Web pages. Commercial tools at the time were insufficient, overly complicated, and mostly designed for thick-client apps. Selenium enabled easier automation of web browsers to test and execute web pages, submit forms, click buttons/links and generally navigate around.
Selenium is one of the most popular automation testing frameworks. Being a free, and open-source framework has made it a popular choice among the global testing community. Selenium over time has gained critical support of multiple programming languages and browsers making it a perfect choice to perform automated cross browser testing.
Here’s what sample Selenium script looks like:
driver.manage().timeouts().implicitlyWait(10, TimeUnit.SECONDS); driver.get("https://www.exoprise.com/freetrial/"); driver.findElement(By.linkText("Sign Up Now!")).click(); driver.findElement(By.className("mycheckbox")); driver.findElement(By.linkText("Login")).click(); //using Selenium click button method
No too bad but not the prettiest and this script doesn’t do anything except click a button and wait for a form. It doesn’t fill anything out, it doesn’t performance monitor anything, it doesn’t upload, it doesn’t send an email. You catch our drift.
Its Not You Selenium, Its The New Web
Its not that Selenium is bad, its that the web and web applications have gotten more complex. Web apps like Outlook Web App, SharePoint, OneDrive, and, whoa, Microsoft Teams look and behave like real desktop applications. These applications, sometime referred to as Single Page Apps (SPAs), are complex to build, maintain and there are subtle changes all the time.
The SPAs are driven by background iframes, AJAX requests, WebWorkers and all kinds of new stuff in browsers to make the web and Internet a richer and more flexible development environment. And these things are hard for Selenium to deal with, test, automate and keep up with. It becomes more difficult when the apps change all the time. You’ve got to be really good at Selenium scripts and understand the way browsers work, asynchronously (in the background), and how it all comes together.
That’s what we do here at Exoprise; we build automated tests at scale to monitor web services and apps on behalf of customers.
Selenium Can Be Brittle
We’ve established that Selenium is a remarkable project and open source solution for testing lots of web pages and simple web apps. But for complex SPAs, like those typically delivered from cloud vendors like Microsoft, Google, and Salesforce, Selenium starts to fall down. Here’s a few reasons why Selenium is brittle and can be difficult to master for complex Web apps:
Finding Elements Is Tricky for SaaS Apps
Locators in Selenium lingo are the way you find elements on a page, aka selectors. The buttons, the input fields, you find them so you can assert their presence and interact with them. Choosing and building the right locators (they are often hierarchical) for Selenium so they last and won’t suffer disappearing through a version or change is a skill that takes practice and experience. And locators are often specific to the framework and development environment of the web app.
Locators for SaaS applications can be even trickier. Large scale, popular SaaS services like Microsoft Office 365 support lots of different versions, editions, customization’s, and languages. That brings very dynamic usually generated on the fly and that means that locators will often be transient, ephemeral, or just change frequently. This translates into very difficult web pages to test and monitor for correctness, delivery and performance.
Once You Find The Element, Interacting Is Difficult Too
Half the battle is finding an element. That helps you test that the page was properly delivered and materialized. Now what are you going to do once you find the element? You want to click on a link, fill in an input form, click a button. That can be difficult too for various reasons.
- Asynchronous technologies
- Background updating makes for lots of waits and DOM settling
With all of the background updating and the dynamic nature of these new fangled web pages, there’s lots of subtle waiting required to make testing and navigating pages difficult. And if you’re trying to performance monitor the page for anomalies then you have to balance waiting for elements vs finding errors and problems on the page.
With these newer asynchronous technologies comes a very responsive Web page and app. Data can be retrieved in the background and the whole page can be morphed and changed. For example, clicking on a button within a page can involve the following steps and measures when using Selenium:
- The element can be obscured behind other elements as an optimization or due to varying page size and media render settings.
- Finally, we’ve even seen cases where an element is positioned off-screen or not within the viewport of the browser. This also varies by browser and framework.
That’s a lot of factors just to click a button.
How Exoprise Tests & Monitors Web Services
So how does the team at Exoprise test cloud services and avoid some of these Selenium issues? To start, we don’t currently use Selenium. Instead, we use proprietary headless environments and “drivers” that are more closely tied to each execution unit.
There are other differences as well:
- Deep understanding of headless browsers
- Eat our own dog food
We utilize most of the SaaS services we monitor and continuously run our own sensors in various development, stage and production environments. This enables us to detect problems or updates and fix monitoring scripts hopefully before the changes are encountered by customers.
- Deep understanding of the applications we are performance monitoring
For complex apps like SharePoint, Salesforce, or Microsoft Teams, we have a deep understanding of the technologies (WebRTC) and frameworks utilized by the app. This enables us to write less brittle monitoring scripts that interact with the application.
- Rapid updates
CloudReady has infrastructure that supports updating the scripts in real-time if they have to be. Usually, monitoring scripts and environments can be securely signed and updated within 1-2 hours across the environments. This enables us to quickly fix issues and deploy updates if required.
- Lots of experience choosing locators (selectors) using CSS, XPath, Regular Expressions, and more
Choosing locators that stand the test of the time is one of the keys to writing robust scripts that to monitor SaaS services. The development team at Exoprise has experience choosing the right locators that often survive application updates, language changes and versions.
- Low-level plumbing
Finally, because our browser environments are proprietary we can leverage lower-level browser state from the driver scripts that Selenium wouldn’t give us access to. This lets us understand more about the asynchronous state of the browser and web application content and what is making the application slower.
Give It a Try
If you’re trying to write your own Selenium scripts to monitor Office 365 or other web applications, we invite you to give Exoprise a try. We think you’ll see that it’s an easier and more compelling way to monitor the web and Internet. And there are plenty of other benefits too:
If you write and host you own Selenium testing scripts, you won’t get the benefit of our crowd-sourced monitoring. Read more about the benefits of crowd-sourced monitoring. or download a whitepaper about it.
- No servers required
With CloudReady you don’t have to deploy or maintain any servers. Our monitoring sensors can run from any spare desktop, MiniPC or virtual machine.
- Your place or ours
With Exoprise, you can run the same scripts inside your firewall as well as outside in public clouds. Monitoring SaaS services from both locations is super critical.