Build what's next on GitHub, the place for anyone from anywhere to build anything.
Join us October 28-29 in San Francisco or online for GitHub Universe, our flagship developer event uniting people, agents, and the world's code.
When the GitHub Copilot Technical Preview launched just over one year ago, we wanted to know one thing: Is this tool helping developers? The GitHub Next team conducted research using a combination of surveys and experiments, which led us to expected and unexpected answers.
Everyday, we use tools and form habits to achieve more with less. Software development produces such a high number of tools and technologies to make work efficient, to the point of inducing decision fatigue. When we first launched a technical preview of GitHub Copilot in 2021, our hypothesis was that it would improve developer productivity and, in fact, early users shared reports that it did. In the months following its release, we wanted to better understand and measure its effects with quantitative and qualitative research. To do that, we first had to grapple with the question: what does it mean to be productive?
When it comes to measuring developer productivity, there is little consensus and there are far more questions than answers. For example:
In a 2021 study, we found that developers’ own view of productivity has a twist–it’s more akin to having a good day. The ability to stay focused on the task at hand, make meaningful progress, and feel good at the end of a day’s work make a real difference in developers’ satisfaction and productivity.
This isn’t a one-off finding, either. Other academic research shows that these outcomes are important for developers [5] and that satisfied developers perform better [6, 7]. Clearly, there’s more to developer productivity than inputs and outputs.
Because AI-assisted development is a relatively new field, as researchers we have little prior research to draw upon. We wanted to measure GitHub Copilot’s effects, but what are they? After early observations and interviews with users, we surveyed more than 2,000 developers to learn at scale about their experience using GitHub Copilot. We designed our research approach with three points in mind:
Let’s dig in and see what we found!
Through a large-scale survey, we wanted to see if developers using GitHub Copilot see benefits in other areas beyond speeding up tasks. Here’s what stood out:
What the charts show, and how to read them
📝 Helpful context: In the survey, we included statements that covered all dimensions of the SPACE framework. SPACE is the acronym for a multidimensional framework that describes several aspects of developer productivity: (1) Satisfaction and well-being; (2) Performance; (3) Activity; (4) Communication and collaboration; and (5) Efficiency and flow. In this post, we focus on the results from the dimensions of “Satisfaction and well-being” and “Efficiency and flow.” Both dimensions are critical to developer productivity and are often overlooked. We’re also preparing more publications of our findings and data, so we can’t share everything today!
📊 What the charts show: The charts below show the percentage of respondents who agreed with the corresponding statement (strongly agree + agree). The >2,000 responses we received came from developers that were signed up in the Technical Preview. They were primarily professional developers (~60%), though we also received responses from students (~30%), and developers who identified as hobbyists (~7%).
If you want a refresher on how we also correlated the productivity results to how much developers use GitHub Copilot, check out our previous blog post, and our academic paper.
All questions were modeled off of the SPACE framework.
Developers see GitHub Copilot as a productivity aid, but there’s more to it than that. One user described the overall experience:
(With Copilot) I have to think less, and when I have to think it’s the fun stuff. It sets off a little spark that makes coding more fun and more efficient.
- Senior Software Engineer
The takeaway from our qualitative investigation was that letting GitHub Copilot shoulder the boring and repetitive work of development reduced cognitive load. This makes room for developers to enjoy the more meaningful work that requires complex, critical thinking and problem solving, leading to greater happiness and satisfaction.
In the survey, we saw that developers reported they complete tasks faster when using GitHub Copilot, especially repetitive ones. That was an expected finding (GitHub Copilot writes faster than a human, after all), but >90% agreement was still a pleasant surprise. Developers overwhelmingly perceive that GitHub Copilot is helping them complete tasks faster—can we observe and measure that effect in practice? For that we conducted a controlled experiment.
We recruited 95 professional developers, split them randomly into two groups, and timed how long it took them to write an HTTP server in JavaScript. One group used GitHub Copilot to complete the task, and the other one didn’t. We tried to control as many factors as we could–all developers were already familiar with JavaScript, we gave everyone the same instructions, and we leveraged GitHub Classroom to automatically score submissions for correctness and completeness with a test suite. We’re sharing a behind-the-scenes blog post soon about how we set up our experiment!
In the experiment, we measured—on average—how successful each group was in completing the task and how long each group took to finish.
There’s more to uncover! We’re conducting more experiments and a more thorough analysis of the experiment data we already collected—looking into heterogeneous effects, or potential effects on the quality of code—and we are planning further academic publications to share our findings.
We’re here to support developers while they build software—that includes working more efficiently and finding more satisfaction in their work. In our research, we saw that GitHub Copilot supports faster completion times, conserves developers’ mental energy, helps them focus on more satisfying work, and ultimately find more fun in the coding they do.
We’re also hearing that these benefits are becoming material to engineering leaders in companies that ran early trials with GitHub Copilot. When they consider how to keep their engineers healthy and productive, they are thinking through the same lens of holistic developer wellbeing and promoting the use of tools that bring delight.
The engineers’ satisfaction with doing edgy things and us giving them edgy tools is a factor for me. Copilot makes things more exciting.
- CTO, Large Engineering Org
With the advent of GitHub Copilot, we’re not alone in exploring the impact of AI-powered code completion tools! In the realm of productivity, we recently saw an evaluation with 24 students, and Google’s internal assessment of ML-enhanced code completion. More broadly, the research community is trying to understand GitHub Copilot’s implications in a number of contexts: education, security, labor market, as well as developer practices and behaviors. We are all currently learning by trying GitHub Copilot in a variety of settings. This is an evolving field, and we’re excited for the findings that the research community — including us — will uncover in the months to come.
We are very grateful to all the developers who participated in the survey and experiments–we would be in the dark without your input! GitHub Next conducted the experiment in partnership with the Microsoft Office of the Chief Economist, and specifically in collaboration with Sida Peng and Aadharsh Kannan.
