Experiments built entirely within SaaS platforms’ web interfaces often take longer and require unnecessary busy-work. This article explores the reasons we would rather build experiments in an IDE and how Mojito supports this approach, so you don’t have to touch a web interface.
WYSIWYG editors are a marketing tool#
Many SaaS experimentation platforms come with a WYSIWYG editor that translates your on-screen manipulations into an experiment. We get it - this feature helps de-mystify CRO and simplifies experimentation. It’s a great marketing tool for split-testing platforms.
CRO experts, choose wisely.
At Mint Metrics, we do not use WYSIWYGs to build experiments. They abstract away too much detail which disallows fine, granular control over elements, and are a black box in how they apply transformations. Our clients mandate pixel-perfect designs and implementations so building experiments in WYSIWYG editors won’t pass muster. We must delve into JS, HTML and CSS.
A case for WYSIWYG?#
What if I just want to test a simple colour or copy change? Wouldn’t it be easier to use the editor?
If you’re working in this space, it’s probable you have a basic level of understanding of JS, HTML and CSS. This is more than enough to set up an experiment without touching a WYSIWYG editor. In fact, changing an element’s colour requires a minimal amount of CSS to execute, so you might not even need to write JS. Browser development consoles make it all too easy these days - crack yours open to see!
Asos.com: change the button colour by playing around with the CSS.
I would also argue that the editor is still not suitable for “simple” transformations. You cannot trust that changes in the editor carry across to a live environment. WYSIWYGs won’t deal with front-end frameworks and 3rd party JS libraries that mess with your DOM elements. If you encounter such issues, you might be deferring to a developer anyway (who won’t be using a WYSIWYG).
Experiment setup is usually a poor experience in a SaaS platform#
SaaS platforms look nice but most make experiment setup a chore. Bouncing around between multiple setup screens, modals and input fields is inefficient and creates opportunities for human error because of the amount of context switching. It’s also inconvenient to see your setup at a glance and make quick adjustments.
Google Optimize: where can I adjust the experiment's sample rate?
With Mojito, we’ve chosen to use a YAML template for configuration. We can define typical experiments in about 10 lines of human-readable code. A super easy to scan format and schema validation helps you minimise setup errors.
state: livesampleRate: 0.5id: ex123name: My experimentrecipes: 0: name: Control 1: name: Treatment css: 1.csstrigger: trigger.jsCoding in web interfaces is not ideal#
As we’ve established, building front-end web experiments involves coding a bunch of JS, HTML and CSS. The code editor interface in SaaS platforms will become your second home after a while. If only they were easy to work with! Want to adhere to best practices such as code re-use and maintainability? You won't have much luck coding in a web app.
What might happen if you do all your coding in a web interface.
There’s nothing worse than writing out a nice chunk of code and losing it through no fault of your own. Raise your hand if you've ever tried saving work in a web app and the platform decides your credentials are no longer valid and kicks you back to the login screen - 🙋🏻♂️. Built-in linters have also caused our browsers to hang or disallow particular payloads (e.g. minified code).
We noticed you did a lot of coding and just want to remind you that you need to login again. Oh, you just lost all your work? We don't care. kthxbye
To avoid these rage-inducing moments, develop your code in a dedicated editor or IDE (we recommend Visual Studio Code) before transferring to your chosen platform. You might soon realise that the volume of copy-pasting becomes very tedious and makes you prone to some minor mistakes - but the extra effort offsets the potential frustration of not doing so.
Optimizely's code editor: your new favourite copy-paste destination!
That’s why with Mojito, we’ve kept everything in the IDE. You can build, review via git pull requests and launch/publish all from the same place.
Version control and continuous integration#
Speaking of git… as far as we know, none of the SaaS platforms offer any proper version control or continuous integration for experiment code. Again, super inconvenient if you want to integrate experiment builds into existing workflows to ensure code integrity. More copying and pasting, anyone?
Mojito’s libraries and all experiment code live and breathe git and we leverage Bitbucket pipelines for a full CI workflow.
We put together a short video demo-ing how fast it can be (under 2 minutes) to setup a simple experiment with Mojito.