Made by
Made by
Avital Tzubeli
What does it take to create a great developer portal? Avital Tzubeli shares Kaltura’s journey that has led them to support 30 programming languages with first-class client libraries.
All right, let’s imagine another scenario. Let’s imagine that in some high rise in San Francisco on the 27th floor, sits a developer named James.
Now at about 10:45, 11:00, on a Monday morning he, you know, rolls into the office wearing his favorite sweatshirt, maybe a cool set of headphones. And I’m sorry for the stereotypes so to make this fun let’s say he doesn’t like beer. He’s a Ruby on Rails developer, because Ruby’s not dead, for a tech company in the recruitment scene.
And this is the cool kind, the kind where you as the candidate create a profile on the app and instead of desperately submitting your resume to a bunch of companies, they, in fact, find you. Do you know those?
And so James is surrounded by all of the coolest swag and the coolest snacks and a bunch of posters on the wall that say cheesy things like, “Love what you do,” and, “Mondays are a reason to celebrate.” At about noon, James scrolls through his first task of the day. Product has requested that there be a video option in the app so that candidates could record themselves introducing themselves.
And so James does what any of us would do in this case which is Google. He searches “Video API for Application.” Now let’s remember that this is a made-up scenario, I have not in fact conducted this Google search, I’m not making any implications about the results.
But in this scenario, James opens the first three results, or the first four results because he’s an overachiever, in separate tabs. And the first result, as you can imagine, is a big, popular, flashy site, super overwhelming. James clicks around and can’t actually find their dev portal.
Now you and I all know that that’s just not going to cut it so he closes that tab immediately. The second tab seems like a pretty good video solution and he finds their dev portal quite soon and they have examples for how to access their API with stream requests. Great, right, examples are great.
But he’s a Ruby on Rails developer, he knows that file handling can be a pain in the butt, especially everything around video and so he leaves that tab open and continues searching. Tab number three looks awesome. He finds their dev portal, he finds downloadable client libraries, especially one in Ruby.
This is awesome, this is going great. But he searches a little more and sees that those libraries have not been updated since 2017 which means that their documentation’s probably just as abandoned and James has zero patience for code that is out-of-date and so he obliterates that tab. We’re on tab number four and James realizes it’s going to be a long day. Now this is a made-up scenario but you are not.
And in this scenario, you belong to a video company that is providing exactly what James needs and you are tab number four. Now how are how you going to make sure to give him what he wants? How are you going to make sure that you can give him a Ruby library or any other library he’s searching for?
How are you going to make sure that library is always up-to-date, that the documentation is always up-to-date, and how are you going to make sure that James, or his co-worker, Susie, or the developer down the street, how are you going to make sure that they pick you? By making it easy, of course. My name is Avital Tzubeli and I’m here to talk to you about automating client library generation.
Not for the fact itself but as a powerful tool for making the lives of developers infinitely easier. Its a powerful tool for connecting with developers and marketing to developers. As a tool to make them feel like regular people who just have opinions and preferences for specific languages and tools and all of the methodologies and tools that come with the languages that they love.
I’ll speak to you about automating documentation, that means interactive documentation because you know this slogan, you’ve seen this before. You know that nobody deserves to be clicking around your dev portal trying to find answers. And for free I’ll throw in this little tidbit. This is not just for external developers, this is for internal developers as well and I’ll tell you a story about a real person who actually exists, a developer named Henri, who sits four floors above me in the office in Tel Aviv.
He is a JavaScript lover, he has one of those obnoxiously loud, red backlit keyboards. He doesn’t like coffee, he does like gummy worms, I’ve discovered. He’s a fan of Radiohead and he’ll tell me all about Red Dead Redemption 2. At Kaltura, he works on our media space or our private YouTube, if you may.
And to do that he uses the Kaltura PHP client library. He says that he can’t imagine life without it. So I said, “Try. What would you do if the library didn’t exist?” And he said, “Um, I guess I would write a little library that included all of the calls that I make on a daily basis.”
And I said, “Okay, great, that sounds cool.What would you do if the API changes and your library breaks?” And he looks at me and says, “This is just a hypothetical situation, right, you’re not taking away my library?I guess I would just have to fix it every time it breaks.” But luckily for Henri, his PHP library will likely always be up-to-date with our API and very easy to use. So how is that?
How is that that Kaltura has over 30 languages, has over 30 client libraries used internally and externally that are always up-to-date with our API? To explain that I’ll back up a little. It’s 2007, Kaltura’s just beginning as an open source video platform. I am still in high school, not part of the story. We know a few things.
Well, first of all, at the time, we have the player, the widgets for uploading videos and the KMC, the Kaltura Management Console. We know a few things. We know that we’re going to build all of our products on top of the same API and because of that the API’s going to become really huge, really complicated, and have lots of objects being thrown around. At this point 758, but who’s counting?
We know that string requests might end up looking something like this. And we know that if we write a client library and then change the API and break that library if this is our development team, 40% of the development team would have to spend their time fixing that library to match the API. So what did we do?
At the time all of these products, all of those tools are using a flex library, X amount to access the API and we’re about to get into the business of writing a whole bunch of other client libraries, so what did we do? We wrote an XML schema that entirely describes the API, complete services, classes, and objects. And then, an XML generator that automatically creates the XML from the API.
And so, for whichever language or function we wanted a library, we just wrote a generator that would parse the latest XML and create that library. Basically, code that writes code. Who doesn’t love that? And so in 2008, we wrote a batch client library that handles all the slow stuff that happens behind the scenes like transcoding the video, a suggestion.
And then a PHP 4 library because at the time, API was still in PHP. And then in 2009, we wrote the testing library. And then the guy who’s writing all the libraries decides to write a JavaScript library just for fun because it’s fun, right? And he says it was awful but it got really popular and I don’t know if that’s a testament to his standards or to how much the community loves JavaScript.
And then the community starts to ask for access. They say, “We don’t want your entire out-of-the-box solution, we just want bits and pieces of your API.” Now generally, this might be a problem but we had everything in place and so we were able to very easily give them a PHP 5 library, something that we didn’t anticipate at first, but why not? And in 2012, the community makes more demands so we give them Python, Ruby, Java, and C Sharp.
Then 2017 was a huge year for client libraries at Kaltura. We created a testing library that powers the interface that we use internally and externally for trying out the API, and then the new AJAX library as well as multi-spreading libraries for Mogul, Milton, and Java. This past year, we rewrote our legacy, Kaltura Management Console, and did so in Angular.
Decided to write a typescript library along the way and why am I making this sound so easy? Because relatively, it was. All we need to do is write a library for…Typewrite and generator for the specific language for which we want a library.
And because of automation with the XML in place, they’re always using the latest version of the API. You can see it visually in this diagram. So the logo, the Kultura logo, that’s the server logic or the API. And from there we have the introspection layer, that’s where the XML is being created. It’s a script that literally just writes the API and I’ll show that to you in a minute.
And then CI, and then the libraries are generated and CI continues integration, puts them in GitHub where they’re tested and any libraries that pass are officially public repo. As well as package managers for PHP, Ruby, Java, and Python. And the simple yet most important part is that a script runs every night that generates the XML from the API and the client libraries from the XML.
Which means that developers are always getting the latest and greatest version of our API. And all of this is happening in no-click at all. How many of you are developers in this room? Just a few. So the cool thing is that this is all part of an open source project that you can use to generate your own client libraries from a schema if you have one.
And in that project, I’m not going to get too technical, but in that project, you can actually see a link to our XML which looks something like this. So this is literally a schema of our entire API. It contains enums, classes, services, plugins, and all of the descriptions around those things. And I’m quickly going to scroll through what you get if you download the open source project.
You basically get three template files. One is an abstract class that helps you with regenerating, another is the wrappers group that runs the generator and tests all the libraries and saves files to disks. And then this is the part that actually parses the XML and generates code. And we’ll get to that in a minute.
I’ve got a bunch of libraries in different languages. Now I’ve been standing up on stages around the world and talking about this because I’m super passionate about this. And I’ve been saying, “Okay, everybody should do this, it’s super easy. Here is this project where you can see what it looks like,” and I said, “Maybe I should practice what I preach first.” And so I had this great idea to create a generator for Go because there had been a lot of demand for a Go library.
Now I’m also a developer evangelist and so since I’ve had this idea, I’ve been on like 20 flights. So as you can see, Go is not up there. But along the way, I learned a lot of things and it gave me insight a little bit more on how the client generators are being created. And so every client library generator is made up of two parts. There’s the source, that’s the step that always stays.
So this is the code that’s written in the target language. For example, what are we looking at here? For example, that would be where the HTTP requests are being created, stuff that never changes unless you change the entire interface. Oh, so this is an example of that in Go, of me writing a request to the Kaltura server.
And then the second part, this is the interesting part, the meat of the generator, this is where you’re actually writing code that writes code. And so, in the Python example, you see how it takes bits and pieces of the XML that I showed you before and this is an example of writing service where it takes bits and pieces of that service.
So the way our API is laid out is that each API endpoint is a service and all of the objects on that service are considered actions. And so it’s literally going through each of those services and creating functions for each of those actions with the name of the actions.
And for those of you who know Go, a function signature in Go is actually one of its like more interesting things. Actually, we want to see that. So that’s what a function signature looks like in Go. And I’m literally taking bits and pieces of the information that I have from the XML and having the code write Go code.
Now, this was not…I didn’t finish it obviously but this was not as easy as I thought it was. And so here is the good news, here’s the good news, you don’t have to do this part from scratch. You don’t have to do any of this from scratch. There are plenty of out-of-the-box tools that will automate this process for you. For example, Swagger, which you’ve probably heard of, has a bunch of solutions that can take your API and turn it into a Swagger spec, an open API spec, which is the standard for describing or visualizing your rest API.
What you end up with is something like this in JSON or something like this in YAML. And similar to the XML schema that I showed you before, that entirely describes your API and it does so either by calling your API endpoints or just by using the code directly. And after you figured that out, the next thing is to create libraries in any which language you choose using that open API spec.
And again, you don’t have to do this from scratch. Things like Swagger exist that have plenty of tools that will do that for you. And what you end up with are client libraries that are always accurate, always up-to-date, and always properly versioned. Now as you know, generated code is not always the best solution. It wasn’t the best solution for us also because this was a while back.
But as we see more and more of API as a product, these tools are becoming more and more popular. In fact, on the topic of automation, do you remember this diagram? Another really cool thing we do is we generate our entire developer portal directly from the XML that I showed you before with the help of LucyBot, which is a project that we sponsor.
The main developer at LucyBot actually wrote this handy little tool for us that converts our XML to an open API spec and then we feed it to LucyBot, which takes all of the contents of the spec and puts it inside nice HTML pages for documentation, workflows, and a try-it-out console. We’ve all seen a developer portal before, we’ve been speaking about it all day.
But this is literally all created from our XML using LucyBot as you can see all of the available endpoints. And then once you get around to playing with it, you can select any of the languages that we have client libraries available for and everyone’s favorite thing to do, copy-paste code. More exciting are the workflows which is super useful in the case of, you know, big operations that require a lot of steps.
For example, uploading a video in the case of Kaltura which is something super common, requires creating an upload token and then actually uploading the file using an upload token, and then creating an entry with all of the video detail, then connecting the entry to the file and lastly, viewing the video in the player.
And the cool thing about these workflows is that they’re super easy to create. All you need to do is select the API endpoint that you want from each step and all of this information is brought in by LucyBot. Select the endpoint that you want, which parameters are required, which values to pass on to the next step, and any text you want to add to that and you’ve got yourself a workflow.
And I get really excited about this because it’s not just for external developers, it’s for internal developers as well. I refer back to this quite often even as an evangelist. And so whether you generate half of this process and write the rest by hand or whether you generate the entire process, it doesn’t matter. What’s important is that you have this in place because developers who aren’t writing your API, don’t need to know about the intricacies of your API.
And so if there’s one thing you take away from this talk, it’s this, define your API. Find a way to create a spec of your API. And if you don’t have an API, go spec first. And once you have that definition, do stuff with it because that definition is now the source of truth between the API and the developers.
The frozen shot of your API will help you see design patterns and flaws. It’ll help you and your developers using the API, or building the API, make decisions about the API. Because if you’ve got multiple products all using your API, you can create internal libraries that your developers are using internally on separate teams to work on those separate products.
And because you’ll be able to add your API to a marketplace or things of that nature and get exposure that allows people to play around with your API and collect their favorite endpoints which I just found out is a thing. And lastly, because your developer portal is your relationship with your developers, give them libraries and languages that they love.
Give them interactive documentation, give them copy/paste code, give them snippets they can use, give them a sandbox and let them play so that they fall in love with your platform while they’re still in your portal. Because you’re tab number four and it is your responsibility to make sure that within three seconds, James knows what your API does and within 30 seconds he finds your dev portal.
And three minutes later he’s up and running with code that works. You’re tab number four and you want to be the reason that James leans back and says, “Hey, Monday’s aren’t so bad after all.” Thank you. Dan from Twitter inspired me to start tweeting today so I’ve added my handle. I have like three tweets now which I realize is a theme for the day.
Thank you so much.