Benchmark & A/B Testing Usability Testing for Micromod Oven

Screen Shot 2021-02-13 at 1.59.56 PM.png

Context

​The "Micromod" is a unique product in that it is a combination built-in convection microwave and wall oven. I served as the UX & Human Factors Lead for the Micromod project where I met weekly with a group of stakeholders, develop and conduct usability testing, work closely with designers, and provide my expertise in design decisions.

I began by conducting benchmark testing on current Micromod, and then use those recommendations to create a new prototype designs. After creating multiple prototypes, the designers and I will narrow down the prototypes to 2 working prototypes, and I performed A/B testing on the two designs, in order to determine which design to move forward with in production.

The objective of the user testing was twofold:

  1. Test 1: Benchmark the current micromod to identify issues in the current design provide design recommendations for a future micromod prototype

  2. Test 2: Perform A/B testing on 2 prototypes to recommend what design should move forward

Methods Used: Usability Testing (Benchmark and A/B), Survey

Tools: Framer, Overflow

Timeframe: 5 months

Test 1: Benchmark

Task Scenarios. In order to develop the task scenarios for the benchmark test, I worked closely with the Consumer Insights (CI) team in order to determine common use cases for this product.

A major feature and selling point on the micromod is "precision cooking." This feature users pre-set cooking algorithms made by food scientists at GE with the goal of perfectly cooking your meals. An issue observed by the CI team was that users did not trust the automated nature of precision cooking.

Therefore, I made tasks that utilized the precision cooking feature, along with testing common user cases.

Procedure. I recruited users that were primarily responsible for cooking in their homes and would be willing to pay the price-point for the micromod. We introduced users to the micromod had them preform 4 scenarios:

  1. Use the Microwave to Reheat Leftovers

  2. Use the Oven to Bake and Broil

  3. Use Microwave and Oven at the Same Time

  4. Use Precision Cooking to Make Soup

Lastly, in a post-task survey we had users complete the System Usability Scale (SUS) and Net-Promoter Score (NPS).

Metrics. To evaluate the product's usability overall, we look at the survey results from the SUS and NPS. We also looked at usability within tasks with the subjective easiness score, time on tasks and pass/fail rates.

Results. The current micromod scored poorly on the SUS in comparison to GE and competitor products. Because of the result of the benchmark test, I developed design recommendations to improve the UX of the product, some of which are soon below:

Test 2: A/B Test

Prototype Design. After the results of Test 1, I worked with the design team to create prototypes for a new iteration for the micromod. We worked closely to ensure that the new design incorporated the recommendations that were suggested in Test 1. Below are some prototypes that were designed during our initial brainstorm:

Procedure. The A/B test used many of the task scenarios and metrics as in the benchmark test. We did this to ensure that we could see whether the new prototypes improved on the current production micromod by directly comparing their SUS scores.

Stimuli. Participants were randomly assigned to two conditions (Prototype A or Prototype B).

As this was a digital prototype, users interacted with the prototype on a touchscreen monitor. I ensured that the dimension of the prototype on the monitor remained the same as it would be in the actual prototype.

As this was a digital prototype, users interacted with the prototype on a touchscreen monitor. I ensured that the dimension of the prototype on the monitor remained the same as it would be in the actual prototype.

 
 
To increase the sense of realism with the prototype, we also had the current production micromod present, and had users imagine they were interacting with the actual product while they were completing the tasks on the computer monitor.

To increase the sense of realism with the prototype, we also had the current production micromod present, and had users imagine they were interacting with the actual product while they were completing the tasks on the computer monitor.

Readout & Results

Overall, both prototypes improved on the current production micromod by approximately 12 points on the SUS. We ultimately recommended Prototype A to participants, based on the results from the SUS score and feedback on the prototype from participants. This design achieved a higher SUS score, and when shown Prototype A and Prototype B, they favored Prototype A's design. 

It is easier to have all the buttons in front of me [in Prototype A]. There’s less steps to getting to the options than Prototype B.
— Participant

After recommending Prototype A to the stakeholders, I then provided design recommendations to further improve on the design for future iterations, working closely with design to ensure the recommendations were implemented, and provided feedback for future iterations of the micromod design.