Ethnographic Evaluation of Internal CMS

Context

During the summer of 2020, the IBM Marketing implemented Seismic, a sales enablement platform. The goal of Seismic was to support all IBM Sellers by saving time as a single destination for all things sales and providing an easy say to collaborate and personalize content.

The implementation of a new software across a huge organization like IBM does come with some downfalls:

  • Prior to my my joining, there was no research conducted before implementing this software onto the organization and thus there were many complaints around the tool. For example, people rated the performance and ease-of-use to be poor via surveys, but since no one had talked to any actual users, they weren’t aware what specific complaints they had with the tool.

  • Additionally, the tool was lacking a full understanding its users. So, we knew that our users were generally content managers and sellers, but we didn’t really have any insights about the users groups and their pain points

Therefore, the objective of this foundational project was twofold:

  1. Add to the collective understanding of key users, their pain-points and provide opportunities for enabling a more effective set of tools

  2. Evaluate the design of the tool for various user groups

The deliverables for the project were to:

  1. Create personas around the roles of the IBM Content Manager and Sellers

  2. Preform contextual inquires of the product from the perspective of the Content Manager and Seller

The following case study will focus on the second deliverable: Contextual Inquiry

The second deliverable is outlined in the Personal Development for Internal Marketing Organization

Methods Used: Contextual Inquiry, Interview, Quantitative Analysis

Tools: Mural

Timeframe: 6 weeks

Research Progression

Develop Protocol

Developed interview questions and a standardized testing procedure, with feedback from stakeholders.

 

Collect Data

Interviewed participants over the course of 2 weeks in 2-part sessions.

 
 

Analyze Data

The team used the method of affinity diagraming to identify themes in our user groups.

 
 
 

Deliverables

The result from testing was the development of 2 contextual inquiries for the IBM Content Manager and Seller’s experience with Seismic.

Developing the Protocol

 

Part 1: Questions were developed in order to determine Content Manager and Seller’s previous experience with the tool

  • E.g., What goals are you generally trying to accomplish with the tool Provide a scenario on when you would use the tool. Provide a painful experience when using the tool.

Part 2: Users were then asked to go through common tasks on the tool in order to determine their pain points with the tool

  • Walkthrough Tasks Included:

    • Content Manager: Show me how you would access a piece of content.

    • Sellers: Show me how you would search for items on the tool.

I recruited 6 IBM Content Managers and 8 Sellers to participate virtually in our interviews.

Analysis

To analyze the interviews data, I used open coding where I analyzed the data and found themes across participant responses and made affinity diagrams on Mural.

We already had quantitive data from the NPS and UMUX lite from Usabilia, indicating that users were unsatisfied with Seismic, but those surveys did not provide enough detail into what we were interested in: Why were they are unsatisfied with Seismic?

Each colored sticky represents a different participant’s answers.

Each colored sticky represents a different participant’s answers.

 
 
I then made affinity diagrams and identified themes (black circles) across all my participants for each interview question that I asked.

I then made affinity diagrams and identified themes (black circles) across all my participants for each interview question that I asked.

Here is the full affinity diagram for all our with themes clustered for all the IBM Content Managers.

Here is the full affinity diagram for all our with themes clustered for all the IBM Content Managers.

Triangulating Data

Intercept data from Usabilia was pulled in order to see how satisfied Seismic users were currently with the tool.

It was seen that overall, the data from both groups, the interview participants and the intercept data were similar. It was then a question of why the ratings were less than ideal.

UMUX-lite Ratings for Content Managers

We compared the 6 Content Manager’s UMUX-lite ratings to the intercept data. We found that Content Managers were less satisfied than users overall in terms of all the dimensions on the UMUX-lite.

UMUX-lite Ratings for Sellers

We compared the 8 Seller’s UMUX-lite ratings to the intercept data. We found that Sellers were less satisfied than users overall in terms ease of use and capabilities, but were more satisfied in terms of accuracy of information and performance.

Results

Results from the contextual inquiry were split into 3 parts:

  • Interface Issues: GUI or interface issues that have relatively simple solutions that can be done quickly via software changes.

  • Positive Themes: Areas of the software were participants felt like their needs were meet or exceeded.

  • Negative Themes: More complex issues that will require more effort and resources to address.

Content Managers

Screen Shot 2021-03-13 at 12.53.38 PM.png
Screen Shot 2021-03-13 at 12.55.40 PM.png
Screen Shot 2021-03-13 at 12.53.48 PM.png

Sellers

Screen Shot 2021-03-13 at 1.54.28 PM.png
Screen Shot 2021-03-13 at 1.54.18 PM.png
Screen Shot 2021-03-13 at 1.54.41 PM.png

Readout

At the conclusion of the project:

  1. I presented design recommendations to the product owners and associated teams from Seismic

  2. Teams were committed to implementing the recommendations and developed a “by item” response plan to address the issues

  3. I was able to see many of those recommendations implemented in the coming months (see below!)

Picture1seismic.png
 
 
Picture2seismic.png