Understanding First-Time User Experience in IBM RPA: An Expert Usability Evaluation

Context

One of the long-term goals for the IBM RPA team was developing an RPA product that was simple enough that the average office worker could use it. Most professionals that currently use RPA are technical users, such as developers, engineers and architects. However, there was a vision toward democratizing automation — making it accessible for non-technical business users as well.

To make this transition successful, the team needed a clear, objective understanding of the first-time experience from the business user’s perspective. Without that foundation, we risked designing for an audience we didn’t yet understand. I conducted a dual-method expert evaluation to simulate the onboarding journey of a true, non-technical user.

  • Cognitive Walkthough: Task-based usability evaluation used to assess flows and learnability

  • Heuristic Evaluation: Use heuristics to measure the usability of IBM RPA’s user interface by reporting violations

The objective of the user research was to evaluate the experience from the perspective of a non-technical business user. I ultimately delivered a journey map of the bot-building experience to provide our stakeholders an understanding of this user’s first-time experience.

Methods Used: Cognitive Walkthrough, Heuristic Evaluation, Journey Map

Participant: 1 User Researcher

Timeframe: 2 weeks

Cognitive Walkthrough

The cognitive walkthrough focused on a common automation scenario: building an RPA bot to open a URL, download a file via the browser, and then close the URL. This scenario was selected because it reflects a frequently performed automation task.

  • Step 1

    Open the Google Chrome browser.

  • Step 2

    Open the URL.

  • Step 3

    Enter the username and password credentials on the website.

  • Step 4

    Download the Excel file on the website.

  • Step 5

    Log out of the credentials on the website.

  • Step 6

    Close the Google Chrome browser.

The 10 General Principles for Interaction Design by Nielsen Norman Group were followed to conduct the evaluation. These principles were developed based on experience in the field of usability engineering and they’ve became rules of thumb for human-computer interaction.

Heuristic Evaluation

Journey Map

As a result of the cognitive walkthrough and the heuristic evaluation, a journey map was created detailing the experience of “Elisa the Business User’s” first time building a bot. In the journey map, I highlighted bright spots with the experience, pain-points, and instances were Elisa was stuck and had to contact support.

Ultimately, 17 UX issues were discovered and recommendations were made for each.

Readout

In the readout to the team, the journey map was presented to the team along with a detailed user story following Elisa’s experience with building a bot. These insights led to early product refinements that simplified bot-building steps, improved guidance, and reduced cognitive burden for non-technical users.

By walking through of the journey map, I was able to point out specific aspects of the design that violated one of Nielsen’s 10 principles for interaction design.

Impact

Insights from the cognitive walkthrough and heuristic evaluation directly informed usability improvements to IBM RPA, contributing to the platform being ranked #1 in Implementation and Usability on G2 and earning top scores for ease of bot creation, administration, customization, and likelihood to recommend.