Case Study  ⸻

Observability at Segment



Enabling customers to monitor and debug their data pipelines, and maintain pipeline health more efficiently.







Role
Senior Product Designer
Company
Segment
Duration
6 months (October 2020 - March 2021)




My Role

I collaborated with another seasoned Product Designer to dive deep into the problem space and produce an MLP prototype. During the process, I drove a Customer Journey Map workshop to synthesize research findings, visualized the map and end user personas as final artifacts to be socialized across the org. I was able to utilize the CJM to bring more user empathy to the team and propel the project progress.




Business Objectives

Reduce customer churn and support costs per year by enabling customers to self-service implement & maintain all Segment products more efficiently.




Digging into Customer Pain Points

My team (a product designer and a researcher) drove 28 user interview/ observation sessions with customers and SAs. I contributed to a few of these technical research calls as I on-boarded to the project.

We selectively interviewed customers who have comparably more mature data models:
  • Larger business tier customers
    • Customers early in their implementation (<2 months since contract start)
    • Who are further along in their implementation
    • Who have purchased our other products
  • Customers that have not worked with an SA but are at the high end of ARR for non-SA supported customers
    • Customers early in their implementation (<2 months since contract start)
    • Customers that have renewed for 1+ year


Some of our enterprise customers are having trouble detecting issues and taking actions. We revealed a Segment implementation journey from implementation difficulties → lost trust → churn.






Customer journey map

I workshopped with the team to produce a customer journey map, that captures customer pain points through out the end to end journey from implementing Segment to maintaining data pipeline health using Segment.



Read more about CJM in my blog post ︎︎︎

With CJM visualization, we understand there are different pain points and user touch points we can potentially solve for. With it and business analysis, we revealed that a lack of implementation feedback, monitoring, and debugging capabilities is at the heart of the majority of ‘implementation difficulties.’ Thus, they are what we prioritized to tackle first.

We also revealed a customer’s internal collaboration model that tend to yield good Segment implementation. The collaboration model results in matured and stabled data management process.






Before

Currently, we display the delivery success rate and potential root causes. However, users have to visit different pages, and use different tools that we provide in order to solve an issue. It is time and energy consuming. Most of our enterprise users have to rely on SAs for answering those questions and debugging issues for them. 





After 



Vision: Enable Segment Implementation Tech Leads to self-service implement & maintain all Segment products.

We envision that users will be able to monitor and debug their data pipelines efficiently. Users will have a high level view of the data journey from Source (where they collect data) to Destination (tools they send data to), and have the ability to filter and drill down into details.




Process

There are a few things that can happen to the data being sent from Source to Destination:
  1. Data does not meet with the privacy standard and is blocked from arriving at Segment
  2. Poor data is filtered and blocked at Segment Source
  3. Poor data is filtered and blocked at Segment Destination
  4. Data does not get sent to external tools successfully.

Now when an issue occurs, it is nearly impossible to understand where it gets dropped off, let alone debugging the issue successfully.

In order to solve this problem, we also envisioned how to improve the underlying architecture besides UI changes.



Meanwhile, we aligned on a high level jobs-to-be-done



With the knowledge and our vision, we started to sketch out a 3 year out UI.


I backcasted from our vision UI and cut some features that are less important to the current problem we’re working on. And simplified the UI to be like this:



In order to figure out the most impactful features that we should ship first, I focused on the JTBDs and used one of the scenarios to align our team on a user flow. The plan is to continue thinking through different scenarios and ruthlessly prioritize. Also, with new forces joining to the team, we are better equipped to progress faster.



Our next step is to user test with MLP prototype (Figma file), while we build out a working prototype with our engineer.



Visual Explorations




Explorations of metrics display


Final MLP metrics hover state



UI exploration for data pipeline visualization



Final MLP data pipeline visualization






Impact

  1. User-centricity: CJM allows teams to start thinking about various product problems from users’ point of view instead of “what we think where users typically run into problems”.
  2. Strategy: CJM allows teams to zoom out and view customer experiences on a high level, to think about an overall product experience, and to reframe customer problems.
  3. Alignment: The vision prototype we built is a tangible experience that allows stakeholders to understand and align on a vision that can help solve several customer pain points.


︎     ︎     ︎  

© QIAO HUANG. ALL RIGHTS RESERVED