Whether in tighter times or not, companies looking to scale have always had be creative in how they attract and retain tech talent while also achieving urgent business goals.
As developer experience, platform engineering and productivity have emerged as priorities for organizations of all sizes, one startup is focusing on the companies that don’t have time or budget for a DevEx researcher — those that might not have time to read the latest research and don’t see DORA metrics fitting their needs.
Quotient, founded in 2022, builds tools to make engineering teams more productive, using a mix of systems data and three-minute monthly developer surveys. It leverages AI to surface priority areas for improvement, which it pairs with research-backed recommended actions. The goal is to help users learn from the best engineering cultures.
The New Stack sat down for an exclusive interview with Lizzie Matusov, co-founder and CEO of Quotient, to talk about research-driven engineering leadership and the psychology of effective engineering teams — and, of course, how to measure it all.
Developer Productivity Drivers Wanted Now
“In a market like this, companies are focusing on how they can deliver more revenue with either the same or sometimes even less resources than they had before,” Matusov said. “So they turn to developer productivity as a way to do that.”
While touting the contributions of companies like Spotify, Microsoft and Google to current developer productivity research, she said, “what we found is that when you move into the tier of companies who don’t have the budget to have a million-dollar spun-up [research and development] arm around productivity, they’re just applying tools in the space to make it work for them.”
For those organizations, she added, “there’s this real need to get a clear, comprehensive view of productivity and also empower engineering teams to take the right action.”
Quotient has found its niche with smaller engineering organizations, where the engineering management layers are just being established. Though developer productivity matters to their companies, they are racing to market and don’t usually have the budget for a full-fledged productivity engineering team to write and analyze data from exhaustive quarterly surveys.
For these startups, engineering and product can make up more than 70% of the company’s budget, Matusov said, so removing bottlenecks can have immense impact.
Making Sense of Productivity Data
It’s not just about measuring developer perspective. It’s about what you do with that information.
“Traditionally, developer productivity as a market has been around sharing dashboards and then requiring teams to be the experts on how to analyze and act on data,” Matusov said. Engineering leadership then fixates on a single metric, like velocity, because it’s easier to understand.
“While attractive, focusing on a single metric has been proven overwhelmingly to cause more damage to productivity,” she said. “For example, focusing on velocity without also thinking about developer experience can lead teams to deliver less stable software or wrong customer value, all while increasing burnout. None of those dimensions are covered in a single metric.”
Google will be the first to admit that most engineering teams are doing DORA metrics wrong, when they just focus on the four key measures: deployment frequency, lead time for changes, change failure rate and failed delivery recovery time.
But that happens, Matusov argued, because otherwise collecting and analyzing dozens of metrics across the annual DORA report, SPACE framework, DevEx metrics and more becomes overwhelming.
“Developer productivity is a unique field because you need enough high-quality information to show you what is happening on a team, as well as an actionable digest of how to turn information to action,” she said. “Over the last few years, what we’ve learned with the way that software is changing is that simplicity and analysis are critical to delivering value in software tools.”
This is an area where generative AI can prove useful: It allows us to skim and contextualize information faster than we could without it.
Quotient collects data from developer systems and developer surveys to get a comprehensive view of the team’s productivity. Then, Matusov said, it leverages AI to “take the sea of metrics and turn it into a high signal subset for teams to act on.” The tool then outputs three things:
- A summary of developer perceptions, calling out key movers at the team level.
- A metrics view of the developer experience. Any engineer can see all the results, and Quotient highlights what it recommends as top priorities for each team.
- Research-backed recommended actions for teams to take.
The product also uses AI to help analyze month-over-month results, as well as continuous systems data.
Productivity Metrics Impact Psychological Safety
In contemporary developer experience, you rarely force developers to do anything, which is why adoption rate is an essential metric for platform engineering and DevEx success.
Quotient boasts an average survey response rate of more than 80%, which Matusov attributes to “a culture of our tool where data is anonymized, aggregated and made transparent to the team.”
In addition, when actions are done based on developer responses, that creates a positive feedback loop that encourages more developers to participate. Internal marketing activities like demos, roadmap sharing and new feature launches, delivered in response to developer surveys, help increase the future survey response rate.
Quotient, Matusov said, is driven in part by the founders’ experiences as individual contributors, seeing their work unfairly or incorrectly represented in developer productivity data, and also, she said, “at a leadership engineering executive level, not having a clear picture of what was happening on teams and how we could take action.”
Technical questions are typically easier for organizations to keep track of with existing tooling. Often, the nontechnical questions are much harder to answer.
Some of the most interesting data around developer productivity is about how developers feel about their work. The so-called “sentiment metrics” can be as insightful as systems ones, such as: How productive do you feel with these tools? What do you think is holding you back?
For example, Quotient includes measures of psychological safety, and the team worked with Amy Edmondson, a psychologist and organizational behavior expert, to adapt her psychological safety scale to a software development environment.
“If engineers don’t feel comfortable experimenting with their different ideas,” Matusov said, “that has a massive impact on how quickly they can use trial and error to come to the right solution as a team to build high-quality software.”
Psychological safety considerations include:
- People in this team are eager to share information about what doesn’t work, as well as what does work.
- How do developers feel about their work?
- How do developers feel about the team they’re working with?
As developer experience is inherently sociotechnical, a combination of metrics is key. Quotient combines data from developer systems and perspectives, including data about how teams work with popular collaboration tools, including GitHub, Jira and Slack. This team-level data includes:
- How do teams distribute code reviews?
- How complex, on average, are code reviews on the team?
- What types of issues are teams spending the most time on?
Quotient then combines that data with its monthly developer surveys. Both sets of numbers, Matusov said, are driven by the SPACE developer productivity framework, with a co-author of the framework, Jenna Butler, serving as a key adviser at Quotient.
The Weight of Technical Debt
Technical debt is often the weight on a team’s shoulders that isn’t noticed until it’s removed. And it’s something that has an impact across the organization.
When asked “what’s your biggest inhibitor to shipping software faster?” in an April TNS VoxPop survey, by far the largest share of participants (43%) said “complicated codebase and technical debt.”
Technical debt is another bottleneck that doesn’t affect only one team.
"One of the cornerstones of Quotient's actions is that smaller, iterative actions lead to outsized productivity gains for the team,” said Matusov “Instead of suggesting a team immediately pay down technical debt — which is not something they can immediately control as a team — we give teams the right tools to take ownership over their productivity directly"
One recommendation from Quotient would be to catalog and communicate technical debt. “That way, we give teams the opportunity to figure out the biggest drivers in our technical debt and how to prioritize that,” Matusov said. “Now that teams have that information, it’s much more likely that the team level can take those steps toward improving their situation.”
Technical debt is one of those clear examples when business influences priorities. The pressure’s on to create value for customers. But dealing with tech debt has an impact on deadlines, which are set in collaboration between the engineering and product
A recent Quotient customer, according to Matusov, saw team autonomy and the impact of technical debt improve significantly. How are these related? With technical debt reduced, she said, “the team is no longer wrangling fires with their technical debt, which means they get more opportunity to work on the things that they think matter. And that’s really critical for productivity.”
Measure Teams, not Individual Performance
“We don’t report individual data about engineers at all anywhere on the platform,” Matusov said, although team-level reporting is enabled.
“We’re very opinionated that it’s hard to do both productivity and individual performance on a single platform without potentially breaking trust along the way. And trust is the center of what we do.”
This is, after all, the whole point of DevOps, too, where the emphasis is on removing blockers to team flow. This was not represented in the notorious developer productivity framework that McKinsey released last year, which lends importance to lines of code written. Recent developer productivity research including from SPACE and DevEx metrics supports Quotient’s focus on the team.
The product also incorporates free text fields, if an individual wishes to provide additional anonymous information.
Of course, “team” remains a nebulous term. Quotient works with each customer’s engineering leadership to define what team means. Sometimes it is a five- to nine-person scrum team, but at another company Matusov worked with, it’s the entire AI group, which has a unique way of working even though it’s scattered throughout the organization.
And then as organizations look to unlock gains to help it scale fast — what she calls “when 1 + 1 = 5” — Quotient gives leadership a global view, which can be used to drive early platform engineering initiatives. The tool includes insights and recommended actions at the organizational level, backed by research from both industry and academia.
Documentation Helps — If Everyone Pitches In
Across all developer productivity tooling and surveys, documentation remains a bottleneck — developers always want more and better, but don’t want to take the time to contribute to it. When docs inevitably surface as a priority, Quotient recommends moving to a continuous documentation culture.
“Teams have a challenge where documentation is their top area of concern,” Matusov contended, because “there’s a culture within a team of waiting until the product is done before writing up all the docs, which is very normal. We’ve all been in this situation.”
But if your team adopts a culture of continuous documentation, she said, “you write docs with each task you do or each ticket that you write, instead of at the end of the project, when it’s a giant two-week endeavor.”
The result is less pain. But docs-driven productivity wins could come at a cost.
“There’s an overwhelming amount of research about how you can incorrectly use data around developer actions and potentially use that to hurt diversity and impact diversity on your teams,” Matusov warned.
For example, 2023’s Google DevOps Research and Assessment Group (DORA) study — also known as the Accelerate State of DevOps Report — found that, across the board, teams with higher quality documentation outperformed teams with lower quality documentation.
“That’s great; it improves productivity. However, they also noticed a trend where, on teams that had improved documentation, their underrepresented groups also reported a higher burden,” she said, meaning women and team members from minority groups were taking on more responsibility for documentation than their peers.
“This underscores how important it is to measure developer experience in a high-trust, comprehensive way. If you miss these important signals, you may be unintentionally hurting your team’s productivity overall.”
As the industry sees big wins in unlocking developer productivity, it’s important that this doesn't come at the cost of certain team members bearing more burden of work. A simple way to fix this could be defining a task as "done" if it has been documented, and making that criteria apply to everyone who deploys software to production.
Psychological safety, technical debt and distribution of work are three examples of areas that demand a nuanced understanding of an engineering organization’s developer experience in order to make improvements that lift the team’s productivity overall.
“The magic of developer productivity is that when you focus on empowering teams and align with their goals of delivering high-value solutions to their customers, everyone wins,” Matusov said. “The engineers, team culture and the customers all gain immense value from improved productivity.”
The post How Do You Measure Developer Experience? appeared first on The New Stack.
A DevEx startup, Quotient, uses AI to discern which productivity metrics matter as organizations scale, and how to turn data into priorities.