Our series on getting the most ROI out of your API has looked at all of the key decisions that need to be made as a business or agency creates APIs, publishes them, and fosters an active community of API developer-consumers. We started by looking at the key customer segments and use cases that would need to be supported by an API strategy, aligned that with a business model, created a team, and did the technical work of choosing an API design and tech stack. In the last couple of chapters, we've looked at how to build great developer experiences (with documentation and a developer portal) so that an API strategy nurtures the developer segments identified at the start of our decision-making process.
In this final part of our series, we look at how to ensure that all the decisions you've made are the right ones. How do you measure whether your API strategy is successful?
ProgrammableWeb founder John Musser has produced one of the most in-depth looks at how to create metrics for APIs. Janet Wagner, a ProgrammableWeb writer, comprehensively summarized his presentation deck.
A key point Musser makes in his overview is that deciding what metrics matter will depend on the overall strategy goals and business model that aligns with your APIs. This is why defining the business model was one of the first steps in your decision series. It was the guiding driver for all of the subsequent decisions. Now we need to make sure our evaluation framework matches back up to what we set out to do.
API Metrics that Match with the Wider Business Context
Because an API strategy must align with your overall business goals, you need to have some metrics that feed up from the API strategy to the wider business perspective.
If you've built internal support (see part two), you'll want to encourage business units to maintain support and alignment with your API strategy. Make sure you're collecting metrics that feed into their business unit reporting processes, which is essential to maintaining organizational support for your API strategy. For example, in Part Five of our series, we looked at how to product manage your API. Melissa Jurkoic from Amadeus Hospitality described how she noticed that at the C-level, some members of management were focused on the public engagement of their overall business plan. She was able to gain support for her API strategy by ensuring there was a media component to her plan and that she was able to gain traction externally through press releases describing the API project milestones. In her case, she may need to keep a metric focused on regular press coverage and ensuring external discussions take place around Amadeus Hospitality's API strategy for it to continue receiving support internally.
Paul Rohan
Banking API industry thought leader Paul Rohan argues that you can use the metrics themselves to generate internal support for an API strategy. (That's why we first discussed metrics in Part Two, when we spoke with Kirsten Hunter on building internal support.) Rohan says that if banks seem reluctant to progress an open banking platform agenda, beginning to report metrics on any existing APIs may be an effective way to foster internal support. After all, banking more than most industries is used to counting things.
He lists a range of metrics that could be relevant to a banking API strategy (these are listed in the image below):
Revenue metrics
Scott Ling, founder of API serverless architecture startup InstantAPI says that you need two types of analytics to measure API growth.
The first type relates to the day-to-day management and governance of your API. This includes analytics that measure and throttle any rate limits needing implementation and metrics like the number of active and inactive API users (for example, those who've registered for an API key versus those who are actively participating in user forums or integrating your API).
The second type of analytics, according to Ling, are those that relate to your business model and monetization plan. You need analytics here to show current revenue being generated from APIs, alongside forecasted revenue and growth trends. "You can start to show churn, the growth of API uses across plans, any conversion rate from free to prepaid plans, and compare this against the costs of the dedicated API infrastructure," says Ling. "This is what enterprises really need to know. Even if you have an open API, from a marketing budget you still have to show the active users, who's on the best paid plans....This helps you make strategic pricing decisions. We spend a lot of time working with enterprises on that."
Developer satisfaction metrics: Time to first Hello World
Jakub Nešetřil, co-founder and CEO for API design tool Apiary (recently acquired by Oracle), says metrics should help measure and reduce the amount of time it takes for developers to adopt your API. "The time to the first Hello World is an indicator of conversion," says Nešetřil. According to Musser's work, time to first Hello World is the length of time it takes for a new developer to become familiar enough with your API that they can make a simple call that reflects a genuine use case. Having identified key developer segments and their corresponding customer journeys and use cases as part of your first decisions in an API strategy, you can now use this to create a metric around what a time to first Hello World would look like for each developer customer segment.
Some API providers are turning to established SaaS metrics to measure their API strategy effectiveness. This can include the churn rate that, in this case, you would measure by looking at the number of developers who sign up for an API key versus the number who are actively creating integrations or apps using your APIs. The Net Promoter Score (NPS) is standard SaaS metric that's used increasingly to measure developer satisfaction with an API.
Musser says that one of the keys to measuring developer engagement is to map the developer funnel. To do this you:
- Identify how developers will discover your API
- How long they stay engaged with documentation
- How long they take to make their first time to Hello World
- How they integrate your API into their products, services, and workflows
- The value they gain from using your API.
Ideally, a metrics system for your API strategy will measure each stage of the developer funnel and identify opportunities to improve levels of engagement.
Modeling the API Ecosystem
Musser also recommends mapping the ecosystem around an API as a way to measure the size of active engagement and to measure the impact value of your API strategy. Based on Musser's design, I created a developer ecosystem measurement approach for ProgrammableWeb's 8 Real World API Strategies and The Keys To Their Success series.
In the following example, the Edmunds APIs ecosystem is measured in terms of:
- Developers active on social media
- Those who are visiting the developer portal and participating in support forums
- Those that downloaded the level of SDKs and Ruby Gems
- The number of developers certified and listed in the Edmunds marketplace
- The known number of apps built using Edmunds APIs
- The size of the customer base for some of those apps
Modeling the Edmunds API Ecosystem
For API strategies where growing an ecosystem and moving to a platform-based business model is a key driver, mapping the ecosystem around an API is an important metric. You can also map the ecosystems of your key competitors and compare this against your own in order to show you the size of mindshare among all developers. This may also help you identify ways to help support your developer consumers to grow their businesses further.
This is the final part in our API Decisions Series. We are keen to hear your feedback on any particular chapter or your thoughts overall on how these discussions have helped guide your own decision-making.
Mark Boyd would like to thank all interviewees who contributed their time and thought leadership for this series. In particular, I would like to thank James Higginbotham, Kristen Womack, Daniel Cerecedo, Steven Willmott, Mehdi Medjaoui, Lorinda Brandon, John Musser, Ashley Hathaway, and Kin Lane for their ongoing support and feedback in discussing API best practices. Thanks to my editors David Berlind, Lisa Nadile and Wendell Santos for guiding me through the research and writing of this series.