In this session from DevRelCon London 2019, Hamza Alam shows how the Twitter team enabled and acted on developer feedback.
Hamza: “Developer experience is really important to us”. “Our documentation is top tier.” “Developers love us”. Do you ever hear companies say these things and think to yourself, do developers really love you? Rest assured I’m not here to tell you any of the above. Rather I’m here to tell you that your documentation sucks to at least somebody. And guess what? That’s actually a good thing.
Today I’m going to tell you how I found that it’s in the negative feedback that companies can find real value. So who am I and why am I telling you this? Well my name is Hamza, hello. And I have a name that’s not the easiest to remember. I’m sure some of you, if not most, even as soon as today have gone into Starbucks, ordered a drink, told them your name. Then afterwards when you check the cup it’s a completely different name. Raise your hands if this has happened to you. Okay, and keep your hand raised if you actually told the barista afterwards that they had made a mistake. Two brave people – three. If you’re anything like me you’ll probably just tweet about these mistakes. Here’s some of my favorites. Hansel. I guess that’s close to Hamza, can’t really complain. My personal favorite is Handa. I guess that’s a cute mash up of Hamza plus Panda equals Handa.
But besides from being called Handa at Starbucks, I’m a developer advocate at Twitter based in New York. I’m constantly working on and looking for ways to better serve the unique and diverse communities that build with the Twitter API. And on Twitter’s DevRel team I actually head up developer satisfaction, commonly referred to as CSAT, or customer satisfaction in the industry. Now developers come up to us all the time and aren’t shy about letting us know how they feel. They can come up to us at events like this, send us DMs on Twitter, or even post tweets on Twitter also letting us know what’s wrong. Don’t get me wrong, this is awesome. We love developer feedback. But how do we scale all of this feedback and create a single source for creating all this?
If I take a few steps back, before I actually started working at Twitter I was on the documentation site itself. And no offense to the previous people on the team who had come before me, I found it so frustrating because I couldn’t understand certain things on the page. I felt like maybe I was just doing something wrong. Or maybe I just wasn’t smart enough to understand.
During my final interview I put together a portfolio. Photoshop’s one of my hobbies. it was bound, super-duper official. And I proposed creating a button on the bottom of our docs to capture developer feedback. As a developer myself, I wanted to have feedback right at the source. Little did I know that I’d been presenting this today one and a half years later.
Also, I want to have some fun with you so I have a deck of Handa Cards, patent pending, which I’ll pull from each time you see one of these wild Handas floating on my deck. The cards just contain brain teasers, trivia, and other fun stuff. So let’s just jump right into the first one. So this one is a poll, or I guess I should say hashtag poll, since I work for Twitter. Now I’m going to ask you a question and ask you to choose would you rather. So would you rather give someone else negative feedback or would you rather receive negative feedback from someone else? So once again, would you rather give someone else negative feedback, or would you rather receive negative feedback? If you’d rather receive negative feedback, raise your hands. Okay, and if you’d rather give negative feedback? I’m in your camp, so don’t worry. Interesting.
At Twitter on the DevRel team, we faced a few problems which I’m sure other companies have also experienced. We lacked an outside developer perspective. We wanted to have all the developer feedback, but we didn’t have a single reliable source to filter all of this through. When you’re responsible for hundreds of thousands of developers who build products that affect the millions of people that use Twitter everyday, I mean millions and millions, it’s not really easy to understand where you need to prioritize your efforts. You don’t have this tangible data, or at least we didn’t, to understand where we needed to prioritize most of our work.
We decided that we needed to do something completely drastic which we had never done before, something completely new. We needed to create a completely new channel for how developers spoke to us. As developer advocates can probably attest to this, we’re pretty spread thin. The amount of work that we do is sometimes quite a lot. Companies can often overlook the value we bring when it comes to measuring output because we’re not 100% developers and we’re not 100% marketing. But we do wear many hats.
When I went into this project I had one very clear goal, to improve the satisfaction of our developers and our online docs. Now that the project had started, I got to work on the fun part. I guess it was fun for me because I liked to Photoshop things. I got to work on the design on how the button should look and how it should work. If any of you have worked on a cross-functional team, no disrespect to anyone, you’ll know how slow things can chug if you’re not careful. Things were on hold for weeks because of over-listening to feedback. And that’s the first and last time I’ll say that on my talk about feedback. But things were on hold because we couldn’t decide on certain things.
When you’re building something completely brand new, everyone has a valid opinion which is great, but also good at slowing things down if you’re not careful. Here’s the first design of the CSAT buttons. I think it’s fun, smiley, you know what you’re going to get, happy or sad. But hold on, are these too childish? Will developers take this seriously? Going back to the drawing board, made some changes based on feedback. Came back with this option, thumbs up, thumbs down. It’s still fun and it kind of gets to the point. But hold on, is that not offensive in some cultures to give a thumbs up? At Twitter aren’t we in different markets around the world? The last thing we want to do is offend anyone, which is a completely valid reason. So back to the drawing board again. After many, many revisions came up with this. Very simple, good or bad. I couldn’t foresee any issues. But hold on. Was this too English-speaking centric? Do developers only speak English? No, at Twitter we’re probably in every single country, not to sound like that. Our developers don’t just speak English, that’s what I’m trying to say. After many revisions, blood, sweat, and tears we came back with the final version – of our first choice. The lesson here is don’t overthink the small things.
Now, we have the design settled. We got to work understanding what types of feedback we’d like to learn from developers who choose the unhappy face. The feedback would ideally be actionable so we could take quick action. And the key points were from accuracy, navigation, clarity, and reliability translated into these points. We also know that when developers leave feedback there’s probably something very specific they want us to know. We decided to give an optional comment box, should they feel inclined to give a reason.
And now it’s time for Handa time. Okay so another poll, I’m going to read two comments from the CSAT feedback that were actually left by an outside developer or a member of the DevRel team. We also use this, too. And I’d like you to guess who said what. The first comment is “Please add more cURL examples”. One more time, “Please add more cURL examples”. And the second comment, all caps, “You need to fix the spacing at bottom, exclamation mark, exclamation mark, exclamation mark”. And this goes on five more times. So who thinks the first comment was the outside developer? Raise your hands. Okay, and who thinks it was a member of the DevRel team? Okay and the second comment, “You need to fix the spacing at bottom, exclamation mark continued”. Who thinks that was the outside developer? Okay, now I’m going to guess the other half think otherwise.
The people who said that the first comment, please add cull examples was the outside developer are correct. Sometimes they’re more polite than us. So around this time of year, last year actually, we flipped the switch and immediately started receiving a lot of data. We were all biting our nails, nervous and completely unaware of what the world had to say. If you’ve ever been in this position you’ll know it’s extremely humbling. And it turns out we actually did okay. In our first full quarter of measuring, our doc score landed at 74.21%. That number represents how many people clicked on the happy face. And this was from a pool of 5,000 unique developers who left feedback.
That meant about 25% of developers chose the unhappy face. That’s 1,000 developers telling us they’re not happy. Literally they clicked on the unhappy face. Of those 1,000, 900 left a reason telling us why they’re unhappy and 400 left additional feedback in the form of a comment – not only telling us that we sucked, but exactly where and what in. This data was from 200 unique pages across all of our docs. This was awesome.
Before, I mentioned that we lacked an outside developer perspective. This was now no longer an issue. And for our team this was huge. We’d never had a way to funnel this specific feedback right into one place, right at the source, on our docs. So to recap, I mentioned a lot of numbers. There were 200 different pages which received feedback, 400 comments that developers left. 900 reasons why they chose the unhappy face. 1000 unhappy faces, 4,000 happy faces, bringing a total of 5,000 total responses. Now if you’re just one person running this, it’s pretty overwhelming seeing all of this data. Yep, it was overwhelming. Or as Yoda would probably say, overwhelming is this.
Now we had all of this data and this amazing CSAT in place, we needed to figure out what we were going to do next. We’re going to do a Handa Card. Okay so this one actually involves prizes. I have cut out Handas. They’re not a sticker, they’re just laminated. You’ll probably throw it in the trash afterwards, but if you get the correct answer, or at least are close to it, then you’ll get one. So the question is, “In a survey conducted by Twitter, according to respondents, two top reasons a way a brand can be more relevant is through…?” And I’ll just read that one more time. “In a survey conducted by Twitter this year, according to respondents, the top two reasons a way a brand can be more relevant is through…?” Anyone want to guess?
Audience Member 1: More hashtags?
Hamza: The lady said more hashtags…
Audience Member 2: Gifs.
Hamza: Gifs? You’re getting close.
Audience Member 3: Memes.
Audience Member 4: Baby Yoda.
Hamza: I’ll give you a prize, that was good. But the top two reasons were actually giving back to the community, and then putting developers first. So yeah, I guess with memes you are thinking of them in some way.
After receiving all of this data, I was pretty overwhelmed. I spent many weeks, literally many weeks trying to understand how I could understand our CSAT data. I was waiting for my “aha” moment. But it was only after a conversation with Stephen Compston, the head of Twitter’s DevRel team that I actually found my “aha” moment. He didn’t realize this until I messaged him two weeks ago reminding him. But I told him, I said Stephen I’m trying to understand how we can measure and view CSAT scores. To which he replied, “Sweet, our docs’ health are important”. That was it. As soon as he said health, I knew that was what we needed to do. We had all of this amazing data which we could look at through the lens of a health score. Coincidentally, my mom who’s a doctor and had always wanted me to go into the healthcare field was really pleased when I told her this.
We had a baseline on our scores of 74.21% in Q1. I knew that we wanted to raise this score by at least a few percent, so I defined pages as healthy if they scored 80% or higher. If they scored less I considered them weak. If they scored less than 70% I considered them unhealthy. We also made sure to account for pages that might have only received one or two responses to make sure that didn’t skew for the health score for that page. Now, once we aggregate this raw data that comes in from the CSAT tour we can see which pages are are performing unhealthy, shown in red, and go through the feedback to understand why this page is performing bad, and do what we need to do to nurse it back to life, to give it some TLC if you will.
Previously I mentioned one of our problems being prioritization; as developer advocates we have so many things going on. It’s hard to understand where we need to focus more of our time. This was now not much of an issue because we had this tangible data and a way of understanding the way that we respond. In our first quarter, based on the CSAT score and the health score process we were able to identify 55 actionable items that lead to either getting fixed immediately or being added to our roadmap. We created Jira tickets. We knew where our priorities lie. The results of listening to developer feedback are really showing. All of our documentation decisions are now based or informed by the CSAT data. In Q1 our score was 74.21%. After responding to the feedback, we saw that they actually went up, which is awesome. And in Q3 it went up to 77.87%. This is actually super awesome. Not only are we listening to what developers are telling us, we’re listening to what they want.
You’re probably wondering how does this data actually come into play? How does it work day-to-day? I have an example to show you. This is one of the things that the CSAT has really helped us rebuild. Our authentication docs, if you are familiar with the Twitter API, I fixed thes. When we looked at the health score for each of these pages we saw that they performed over 10% lower than average. That was no good. When we looked at the negative feedback, I’m not going to read them all, we were able to find common themes which developers were telling us. And we boiled them down into three simple reasons: They were unclear, confusing, and lacking examples.
Because developers told us that these were unclear we identified that there were actually 33 pages related to authentication living on our docs. They spanned multiple sections of our docs, which was not good. So, we decided to move all of these pages onto one single unified section to make it more clear, because developers told us that the pages were confusing. At the beginning of each page, we begin with the why. Why does this page matter? Why do I need to know this information? What will reading this help me solve? Because developers told us that these pages lacked examples, we moved the examples into places where you would actually expect them to be. In places that made more sense. Now let me show you.
And just so you guys know, this is a top special unveiling. These aren’t actually live but I did get the okay from marketing. But they should be coming, but you are the first ones to see them. This is our old authentication docs… I don’t even want to say homepage, but this is the page that you get taken to. It kind of just throws you in. This is what this one does, this is what this one does. And if you’re like a Twitter employee and you know these things, the way that it’s laid out is really confusing, even to myself. This is the new authentication homepage. Clean, simple, and easy to navigate. Now you know why to do something as opposed to only know how to do something.
All of our authentication docs were redone based on feedback. But some changes were not as drastic as a brand new homepage. For example, our API reference page was redone. This is the old API reference page. If you’re a minimalist, this is okay, but if you need to click on each different page to understand what each API does, you’d want to find an easier solution. This is the new API reference page. We let the developer know, on the overview, exactly what API does what so they can understand and find things related to what matters to them. It just makes it so much easier.
Had there not been someone on the outside telling us that these pages sucked we wouldn’t have known where our priorities lie. It’s humbling and restores trust in your relationship with developers if you want to grow with them. It shows them that you care more by listening to what they have to say. And that is the feedback. That is the value that negative feedback provides for you. Thank you.
Can you make good release notes by collating your commit messages? Eva Parish argues not.
Can negative feedback from customer satisfaction surveys have a positive impact on your developer experience?