Metrics discussion panel

As part of DevRelCon Earth 2020, Bitergia’s Ana Jiminez Santamaria, Orbit’s Josh Dzielak, and Phil “AAARRRP” Leggetter got together to discuss their approaches to measuring developer relations.

Transcript

Matthew: If you had been in the UK earlier this year, you might have seen the developer relations meetup in London. We were planning to do a panel on metrics there but unfortunately, COVID meant that couldn’t happen.

So we’ve regathered the people that we’re going to take part. Ana, hello.

Ana: Hi, guys.

Matthew: You join us from Madrid, is that right?

Ana: Yes.

Matthew: We have Josh joining us from somewhere in France?

Josh: Yes, from Paris. Hi, Matthew. Hi everyone. Hi, Ana.

Matthew: Okay. Hey, how are you doing? And so Josh is co-founder of Orbit and we’ll talk a bit about that in a moment. And Phil Leggetter is in Scotland, Dollar in Scotland. So, you know, Phil, you spent four years running quite a sizable team at a big company. Josh, you’ve run teams but now you have a company where you’re building a tool to apply a very specific model of measuring developer communities.

And Ana, you also work for a company where you are measuring the effectiveness I guess of open source projects and how you Marshal developers within those projects. So, Phil, as you’re coming at it from, perhaps the point of view that many people watching this have, I’d like to just get a quick overview of what your developer relations metrics philosophy is.

Phil: A good question. Yes, so, I mean, I left Vonage next… a month ago now and I’m still missing the team and having to watch Ben before this was quite hard. I missed Ben in particular. Yeah, I mean, Vonage was a… I mean, at Vonage, Vonage was obviously listed on the New York Stock Exchange. Actually, then it moved to NASDAQ.

There were shareholders. There’s a board who report to the shareholders. There’s, you know, a CEO who reports to the board, who reports to the shareholders. And, you know, the hierarchy is as such and therefore, the goals of the organization filter down from that leadership team. So, you know, the role of a developer relations team within such an organization is to understand those goals and translate them into activities and actions that help drive those top-level goals for the business.

And that’s, you know, exactly what kind of, you know, the art framework that I did a while back, is about understanding the goals from the business. It’s about identifying activities and it’s not really about measuring. You know, absolutely you know, Josh’s Orbit framework that I read prior to this call, actually fits in very well with it because it’s specifically about measuring the community side of things there and another tooling as well.

So my philosophy is to, you know, basically try and deliver activities and, you know, work from a Developer Relations team that helps the business address those goals. And I think if some parts of Developer Relations that I’m particularly into, so, you know, definitely supporting communities, I like the involvement of a Developer Relations team within the product and building developer experience.

If those activities and that work doesn’t map to the business needs, then I wouldn’t be involved in it. So that’s kind of the overall philosophy and then how I look for my involvement in that.

Matthew: Thanks, okay. So, Josh, next Tuesday, the seventh you’ll actually be giving a talk about the Orbit model. But just give us an overview of what that is, but also why you felt it was necessary to come up with a new analogy for Developer Relations measurement?

Josh: Yeah, absolutely. I think when I was working at Algolia, we wanted to find a terminology that described the way that our team thought about the community. And we worked very closely with the marketing team. But when we tried to use the marketing funnel language to work with that team, we ran into some issues. Some things that were a little too direct like how many leads came from this meetup, which is a question a lot of us have been exposed to.

And we said, “Well, maybe that’s not the right thing to measure. We’re not exactly doing a funnel here when we’re building the community. So is there another metaphor that we can try?” And then that was really the idea for this thing we call the Orbit model was continued there.

It was born a little bit earlier in my career at a company called Keen IO, where we just started to kind of play with the idea but it’s about organizing your developers, and the people in your community into concentric circles based on how close they are to you, how much of a role they play in the community, how engaged they are, and then designing programs that differ based on that.

So it’s not like we have a program for developers in this stage of the funnel, this stage of the funnel. We say this part of the orbit, this part of the orbit. So there’s the kind of philosophy and metaphor piece about it, which is just a solar system instead of a funnel. And it lets us do things like keep developers in certain places without having to force them to, let’s say, the bottom of the funnel but we can actually, you know, have different programs that work for people who are very close to us like our ambassadors, and then people who are maybe just reading our content wanting to check something out.

Maybe they’ll be an ambassador someday but, you know, we want to meet them where they are right now. So that’s, you know…I will be giving a talk about the Orbit model, and we’ll get more into data and how we measure some of these things, and I’m happy to take questions about that too. But kind of, at a basic level, that’s what the Orbit model is all about.

Matthew: Okay, thanks. And Ana, much of Developer Relations as practiced, I believe, is really focused on or at least the measurements that executives and managers seem to care about are focused on the point at which someone gets an API key or makes a download, or maybe makes their first API call, or some kind of essentially, you know, getting people down the funnel and counted as a win somehow.

Whereas the work that you do at Bitergia is more about what happens after that. And I know that, you know, in an open source sense, you know, a lot of the time there aren’t the same commercial incentives, but you have an involvement in in the source as well, don’t you? So tell us what are the things that you measure then in open source communities that help to show whether they’re healthy or not.

Ana: Okay. So well, in Bitergia, what we are measuring is software development analytics. So as you said, we are not so focused on the first level that maybe developers might be looking for that our users, our most valuable users, but a little bit beyond that. I guess as Josh was saying, users and most valuable users might be more involved in these marketing metrics, marketing goals, but then there is beyond that.

So we usually work with customers that sometimes are community moderators or even DevRels in the open source ecosystems. They also want to know about contributors, not only users, or even maintainers, so those people that actually are asking questions on GitHub, on GitLab, on mailing lists, on Slack, but they are…and then also started to provide code, let’s send pull request, and maybe converting like transitioning from those initial most valuable users, your contributors and then to maintainers.

So that’s basically what we do.

Matthew: Okay, thanks. So, Jason in the Slack has got straight down to business and got to the fundamental question that a lot of DevRel skeptic executives are asking. You know, I’m not saying that that’s what Jason is.

I’m saying that this is the question that gets asked. And it’s the…you know, revenue is the thing that people care about, or the businesses care about. Ultimately, you know, if Developer Relations or the community is not pulling its weight financially, a good CEO will direct resources elsewhere.

So how do you translate, Jason asks, the activity of DevRel teams into metrics that speak to that ultimate company goal of revenue generation. Josh, I want to ask you that because I’m interested to see how the Orbit model fits in here.

Josh: Yeah, yeah, absolutely. So first I’ll say that the Orbit model is mostly concerned with how we measure and grow the community itself. Because I love to use metaphors… If you think about an ice cream cone, you can think about Orbit on top and the funnel on the bottom. And when we’re building a community and making it really, really highly engaged, lots of people buzz, lots of activity, we’re growing the size of those orbits.

And the question then becomes, how do we get all of that activity from the orbits to eventually drip some ice cream down into the funnel? And in my metaphor, that’s how I think about, how do we get that activity that’s really a very active, very happy community doing…you know, getting into the things that… like purchasing the product, like moving on to next steps in the funnel?

And so for us, we think about where are the points in the orbits where a community member is most likely to want to move on to the next step? And then we count when those things happen. A lot of stuff in the Orbit model is just about counting like, “Count the number of activities that people in your community do.”

And one of the things that you can count, and this comes to Mary’s great idea of DevRel qualified leads is counting the number of times that you hand someone off from one of the orbits down into the funnel at the right place. So we can count that and see. But I’ll say…

Yeah, I think actually that’s mostly it. The orbit model itself, we’re usually thinking just about how do we get more and more people into those orbits? And then I think we also look for inspiration for other frameworks to see like Mary’s and like Pirate Metrics, things like that to see where are the best touchpoints that we should be counting and then which one of those?

Because if we know those, then we can actually go back up into the orbits and look at our programs and see is this program for these people in this orbit actually dripping people down the funnel at some point or not?

Matthew: So Phil, when you were in your previous role, and you know, you’ve led teams in other companies, what…you know, you’d have to go to present to the senior leadership team on what you’ve been spending all this money on and, you know, why there are members of staff tweeting about airport lounges and things like that.

What is it that you fundamentally say to them about the value that you’re generating?

Phil: Well, I mean, I think we did map some of the work we did through to revenue, and we’d had discussions and ultimately took ownership at Vonage, for revenue that came from self-serve because the work that we did and the ownership of the assets on the platform that we had were about enabling customers to self-serve.

So yes, there were… We had two types of accounts. We had managed accounts, so an account that has an account manager, so someone that has, you know, someone hand-holding them through the on-boarding process and through, you know, their active usage of the account. And those that don’t have an account manager, so self-served or unmanaged accounts.

So we’d had an agreement from the business that we would be, you know, held accountable for the revenue and benefit…from the revenue that comes from those self-serve, those umanaged accounts. So one of our goals was to grow that unmanaged revenue. Now, don’t get me be wrong. There are many points across the business that those people interact with, but we felt that there were a number of activities that we did and things that we owned.

So we did directly map some of the work that we did and we’re held accountable to a revenue figure. There’s also, you know, the cost of a managed account. There’s conversations around pricing, reductions in price. There’s the cost of an account manager. There’s more costs in a managed account. So the self-served account also provided a better gross margin.

So your operating costs take away revenue. So the benefit there of these accounts, you know, for a business such as an API platform is clear that you want to grow gross margin. And that’s a figure that you report to the street, the stock market. And then there were some other things that, you know, I do think, you know, to the question to look at revenue alone isn’t the way that a business should operate, and that business will likely fail because you do need to look at multiple things.

Yes, there’s the more monetary side of things. There’s revenue, there’s gross margin, there’s customer satisfaction, and there’s employee engagement. Those are the four things that as an organization, we particularly tracked on and then we mapped through to those things as well. So, you know, to answer the question directly, we did map to revenue. But I think it’s very important to acknowledge that a company cannot succeed on revenue alone because that costs as an example could be more than the revenue.

So that’s no use.

Matthew: But okay, we’re in a time where we’re probably..where we’re already seeing some economic turmoil and budgets being cut and so on. So do you think it washes to say to an executive who wants to cut your program, “Ah, but let’s not just think about revenue?

You know, we’re building goodwill or…” you know, how do you get around…? And I’m not saying I disagree with you, Phil. But I think a lot of executives might… How do you skirt around that? Because ultimately, all the good stuff you do has to affect the financial health of the company, whether it’s an ability to raise more investment, or if, you know, you’re actually running as a normal company to show a profit.

Phil: Yeah, I mean, I think the gross margin is a really good example of that. You know, the self-serve individual developers that activate and okay. You know, you want to get as many developers through to using your product as possible and certainly an API platform wants to do that. And thousands of developers giving you $100 a month at a very low cost is really valuable to the business.

And I think, you know, an executive understands that. Don’t get me wrong. I mean, we’ve seen this, right? We’ve seen companies shift from being very developer-focused to very enterprise-focused and then back again. But you’ve got to get that balance right. And then I think to some of the more… I mean, I can give specific examples.

You know, the Vonage top relations team now just recently ran an internal global hackathon. Now, what’s the revenue benefit from that? It’s very difficult to measure. But what that does is that makes everybody work together, increases collaboration, and improves a sense of culture.

Certainly, the way that that team approached it, it encouraged a diverse type of person. You know, so we had people across the business whether, they were technical or not, collaborative collaborating or working together on that. And the benefit there is that you’re maintaining a workforce that wants to work together and improving collaboration there which will then, you know, if you apply some logic to that will help you, you know, deliver better business outcomes to take a phrase of the exiting CEO of Vonage.

Matthew: Ana, what is it that… Because I think it’s different clearly from the point of view of measuring open source communities and developing communities. But do you find that people use the measurements that your software provides to then report back on how that affects maybe not revenue but perhaps the cost base of the business?

Ana: Okay. So in this case, I will say that having only even focus on revenue is more like marketing and sales prospective. And I totally get that because a lot of DevRels report to marketing and sales. But I think it’s important to let them know, to let that department know that what we are here is for build relationships and to create value for developers.

And in this case, what we need to be focused on, I think the main objective will be more related with the relationships and the community. And that’s why we need to add these metrics related with community. And really, really important, I will say that really main objective will be like how are we transitioning?

Like, okay, we have users that are using our software, but from an open source perspective, for us, this is not enough. And an objective for us will be, okay, so this user has been converting, has been upgraded to a maintainer because they move to Discord, maybe to Git and then Git decided to ask, “Can you to open an issue?”

And that’s a high level of engagement. So now we were talking about metrics engagement. And then we can go a higher level. And now this developer is being more confident. And now they are starting to add code, like adding pull requests. So I think it’s important to look at transitions and where the developers are coming from, and that’s also really, really valuable for the business.

It’s not only okay, this is revenue, because revenue will come later. I think it’s an important thing to know is that DevRels are not going to get revenue in just one year or six months. It’s a long-term process. And marketing, and I come from marketing. So I know that the metrics are completely different.

And it doesn’t make sense just being on top of my revenue and just start. So having that, okay, where are my developer journeys and where are the different transitions and the different touchpoints as Josh says that my developers are taking in order to build those valuable communications with them on each of the different steps process or yeah, steps.

Matthew: Okay, thanks. So frequently, or at least, you know, every time we’ve done a DevRelCon, I’ve either seen a talk or heard a corridor conversation where people are arguing that we shouldn’t measure Developer Relations at all.

You know, it just is. It’s a known good. Are there any arguments in favor of that, would you say? Or is it, you know, is it…? Is it possible to exist as an effective Developer Relations organisation without strict measurements?

And even is it desirable because can you focus on the right things that you’re not measuring? What do you think, Josh?

Josh: Yeah, I think that the right style of measurement is very important. I think that certainly as a team gets bigger and fights for more budget, it’s just impossible to do that without metrics. That just becomes an increasingly harder battle. And if you’re a decision-maker, you want to have the kind of data anyway to know where you should be investing.

So if you’re not measuring anything you don’t know which sub-community should we be investing in, what location? What features do we have for developers that are the most interesting? I mean, it becomes very, very hard north of one human brain holding all of that information to actually make, you know, make smart decisions and then be able to communicate what’s going on. But I think some of the allergy to measurement in DevRel come from trying to measure it with tools that come from other disciplines like sales and marketing.

Certainly some marketing metrics absolutely work well for DevRel, but I think there’s sometimes, especially for the people coming to DevRel through engineering, there’s some unfamiliarity with how those metrics are, or how they work or their general like, “Oh, it’s marketing so I don’t want anything to do with it.” And I think that’s wrong, but I think we need to have some set of metrics that we feel are simple and clear enough where even if it’s not someone’s primary job as a DevRel data analyst, which I don’t think that’s anyone’s primary job, they can still track things, get enough data to make decisions, get enough data to communicate how the program is doing, or what the community is up to, and then ultimately, the attribution question, like which of the activities that we’re investing in are actually leading to that revenue.

So I think, it’d be crazy to try to scale a DevRel program without measuring some things about how it’s doing, but you don’t have to measure it in terms of metrics that you don’t think are a good fit. I think it’s okay to come up with other ones. Of course, I’d say that because we’re kind of doing that at Orbit coming up with some different metrics to think about these things.

But ultimately people will measure what they think is useful and trying to push them to measure things that they don’t really want to is not ultimately very effective.

Ana: Can I add something?

Matthew: Yeah, please do, yeah.

Ana: Yeah, I just wanted to highlight the need of having a standard ways of metric. So I think that’s one of the things that DevRels are missing right now. Because in marketing and sales, we have the sales funnel, we have the marketing funnel. And even though we’re in a company and we go to another company is like, “Hey, we’re going to follow this.”

But in DevRel maybe in your company, you convince the marketing department and the sales department that, “Hey, this is my objective. This is the value of DevRel.” But then you move to another company, and it’s like, “Oh, my God, then I need to do again the same effort to explain to marketing people and salespeople that their metrics are not enough.

So I think it’s really important that all the DevRel community starts to looking for, “Hey, these are the metrics we should be following in order to then show to other people that are not DevRel and are not familiar with what we are doing, show them our value.”

Phil: Yeah, I’ll make a few points. It’s interesting what Ana is saying there. I mean, I think even if you’re not moving companies get ready to repeat the same thing over and over and over again throughout the organization to be continuously sharing, you know, what you do, why you do it, and the value that you bring.

You know, other departments will change, new people will come in, people need to be reminded. So I think that’s really important whether or not you change your company or within the same organization. And I think Josh, you know, nailed it in terms of if you want to scale this, you want to scale Developer Relations, and you want more investment you have…like anything, you have to justify that investment. I think there’s an interesting phase for a lot of companies.

And when that is already dependent on probably the founders, right, or the stakeholders in Developer Relations because businesses that understand the importance of the developer will invest in Developer Relations from the off and probably invest for longer and more because they ultimately believe.

But even within that sort of company, eventually there’ll become a point at which there are other stakeholders and the need to justify is there. You know, I love that phase of…and it’s only happened to me, you know, once really or twice now with my new stealth mode thing. There’s a natural belief in the Developer Relations side of things, you know, the importance of developers, the importance of community.

But obviously, I mean, once at Pusher and now this company where there’s investment already in that. But yeah, at a point there will be, “Okay, well, let’s really get down look at the metrics and justify where we should be putting this funding.” Should it be towards a more traditional acquisition, you know, marketing? Should it be investment in community?

Should it be on something else?

Matthew: Okay, thanks. This is really flown by. I wish it had been a bit longer slot. But let’s close with a question from Tamao who has a question that I’m really interested in because I think, you know, probably there is a lot to be said about trying to come up with a formula for input x results and output y.

So, Tamao says, “Have any of you needed to provide estimates for revenue when justifying a new hire and why that hire whether it’s a dev advocate or a community manager is worth the expected salary?” Such as we are missing an opportunity for x revenue because we don’t have someone doing this exact type of work.

Phil: I mean, never in a single hire. One is there was a yearly cycle of, you know, requesting headcount and then putting models in place to… I mean, examples would be, right, okay, hiring someone in a certain location. But often we did it as, you know, an additional… I mean, you saw the growth from 3 to 40 something people in four years so there was you know, quite…

So what would happen is then during the negotiations and, you know, just sometimes just some arbitrary cuts because we needed to find money, you would then work away but you… And then you would kind of go, “Okay. Well, we’ve taken that away therefore we expect this drop in the revenue wherein our predictions have changed.” So not on an individual headcount, but yes, on a, you know, a yearly plan, the revenue we expect to bring with this many hires.

Matthew: Josh, did, you know, when you were in Keen and Algolia, did you come across that kind of situation?

Josh: I came across something I think that’s pretty close, which is trying to fight for a hire on your team that might go somewhere else. So I know I had questions that came to me around, “Well, why should we hire a developer advocate who costs this much when maybe we could just hire a marketer or someone else who costs this much?”

And, you know, tell me why we should spend more money hiring a developer advocate for this position or to do this kind of growth. And then that becomes the conversation or the debate that you have to make as the DevRel leader in that position. Like, why is spending money on developer advocate for the business goals that we have, for the community goals that we have better than maybe bringing in a different…opening up a different kind of role?

And so I think in someone’s head, especially leadership, they’ve got like a number in their head, like, “What’s the difference in the cost that I’m going to expect if I hire this person or this person?” and they want you to justify that. So if it’s this many thousand dollars different for these salaries, you know, I think it’s kind of the implicit question or probably explicit for some people like, “Tell me how that’s going to translate to that much more value for the company if we do that.”

So I’ve had to make that argument once or twice.

Matthew: And, Ana, I guess from the point of view of companies who invest in open source communities and projects, this must come up at some point, you know, the opportunity cost of doing that versus something else.

Ana: Yes. We have had customers that actually had kind of same problem. They were customers that let us know, like, “Hey, we usually had only users. And now we’re starting our community building strategy.” Normally, DevRel is involved in that community building. And I think from the open source point of view it’s easier to show that value because as if those software developer, those developers are providing code, are becoming contributors and maintainers they’re being more healthy, the communities is being healthy and their technology is improving, is being more innovative.

So that’s the value they will save by having contributors and maintainers.

Matthew: Okay, thanks a lot. We’re over-running. I’m really sorry. I feel like we barely got started talking. Ana, where can people find out more about Bitergia?

Ana: Okay. Well, I think you can follow Bitergia, Twitter. Also, you can ask me any questions on my Twitter, and also bitergia.com. We have a lot of information, webinars, and things you might like. And yes, that’s all.

Matthew: Cool. Okay, Josh, where can we learn more about Orbit?

Josh: You can head to orbit.love, or follow us on Twitter, @orbitmodel.

Matthew: And, Phil you’re being quiet about what you’re doing so we’re not going to talk about it. Oh, @leggetter, yeah, right. There’s your thing. Okay, and I’m really sorry. You’ve got a question in slack that is do you feel like people ever ask this question about people ops or HR or other must-haves?

I think maybe if someone can give a 30-second answer to that question, we can close on that.

Phil: I’ve seen HR significantly under invested in and it’s fundamental to the successful running of a business. So yes, it definitely helps with that… It definitely happens with other departments.

Matthew: Okay. All right. Well, thank you so much all three of you for joining me. I’m sorry that it was over so quickly. So, goodbye. Thank you for being part of DevRelCon.