User testing, rebuilt: websites, phones and native apps

What changed

User testing on Synthetic Users has had its biggest update yet. You can now test a native Android or iOS app on real devices, not just a website. Every session explains itself: how it ended, how hard it felt, and where the participant explored, got stuck or found their way. And the Summary turns all of those sessions into one screen map per task.

The screenshots follow a made-up coffee roaster, Fernhill, testing how people find a coffee subscription on its site.

Test any website, on any screen

Point a study at a live site or a prototype link, and each Synthetic User opens it in a real browser and works through your tasks in order.

Laptop, tablet or phone. Phone and tablet studies use touch, so a tap behaves the way it does on a real device.
A start URL per task. From the second task on, a task can begin on its own page instead of wherever the last one ended. Start URLs are checked before the study runs.
Sites behind a sign-in. Give the study a login and participants start signed in.
Better tasks before you run. Task review reads each task as you write it and suggests how to make it clearer.

The Testing panel: task 2 with its Start URL set to the shop's coffee page, checked. — Task 2 starts on the coffee page. Leave the start URL blank to continue from the previous task.

Native apps, on real devices

Create a study from the User testing template and open Testing.
Open the Mobile App tab and upload your build: an APK for Android, or an IPA for iOS.
Choose Phone or Tablet, and Portrait or Landscape.
Write your tasks. They run in order on the same device, as someone would use your app in one sitting.

The Mobile App tab: an APK that's ready to test, with Tablet and Portrait selected and a preview of the device. — The build is checked as soon as it's uploaded, and you can keep writing tasks meanwhile.

Every participant gets the same current device model with the latest OS, picked when the study runs, so their screens line up when you compare them. When a session ends, its recording starts the moment the participant first looks at your app, not while the phone was being set up, and each tap is placed on the video.

Which builds work. Android: an APK. iOS: an IPA signed for ad hoc, development or enterprise distribution. Simulator builds and App Store or Play Store links can't be tested. One build per study (uploading another replaces it), usable for 30 days.

Every session explains itself

Open any participant to watch their session. Each click or tap comes with what they were thinking at that moment, and above the recording you see how the task went.

A finished session: Gave an answer, 5 steps, did not go back and forth, confidence 6/7, ease 5/7, and a strip of phases above the recording. — How the task ended, the numbers behind it, and the phases of the run. Click a phase to jump the recording there.

How it ended: Gave an answer or Gave up.
The numbers: steps (with scrolls and clicks or taps), whether they went back and forth, and their own ratings of confidence and ease, from 1 to 7.
The phases: where they were Exploring, Stuck or lost, Found the way and Answering.

Follow-up questions work as before: ask every participant, or just one, what they thought after each task.

The Summary: one screen map per task

The Summary pulls every session together, task by task. Each task shows how many people gave an answer, flags the hardest one, and lists how many went back and forth, the most clicked (or tapped) element, the average steps, and the average confidence and ease.

The screen map: the pages people reached, the subscriptions page with markers where they clicked, and On this page: Where they clicked and Where they went from here. — Every page they went through, where they clicked on it, and where they went next.

The same page in Attention mode: a heatmap over the plan names, and Where they looked: Plan names 80%, Delivery small print 60%, Prices 40%. — Switch to Attention to see where they looked. It comes from their own reasoning as they acted, not from eye tracking.

Paths: four of five went Home to Coffee subscriptions and gave an answer; one went on to Brew guides and gave up. Below, every participant against every task. — The paths people took and how each ended, then every participant against every task.

Filter the whole map to one segment or one person, and jump from any quote to that moment in the recording. App studies get the same Summary: screens are named by what they are (Cart, Search results, Listing details), and identical screens are matched across participants even when they reached them by different routes.

Watch it in two and a half minutes: User Testing | Synthetic Users tutorial.

To try it, create a study from the User testing template, then add an entry URL, or open Mobile App and upload your build.

All data in the screenshots is fictional.