> Copilot excels at low-to-medium complexity tasks in well-tested codebases, from adding features and fixing bugs to extending tests, refactoring, and improving documentation.
Bounds bounds bounds bounds. The important part for humans seems to be maintaining boundaries for AI. If your well-tested codebase has the tests built thru AI, its probably not going to work.
I think its somewhat telling that they can't share numbers for how they're using it internally. I want to know that Microsoft, the company famous for dog-fooding is using this day in and day out, with success. There's real stuff in there, and my brain has an insanely hard time separating the trillion dollars of hype from the usefulness.
We've been using Copilot coding agent internally at GitHub, and more widely across Microsoft, for nearly three months. That dogfooding has been hugely valuable, with tonnes of valuable feedback (and bug bashing!) that has helped us get the agent ready to launch today.
So far, the agent has been used by about 400 GitHub employees in more than 300 our our repositories, and we've merged almost 1,000 pull requests contributed by Copilot.
In the repo where we're building the agent, the agent itself is actually the #5 contributor - so we really are using Copilot coding agent to build Copilot coding agent ;)
(Source: I'm the product lead at GitHub for Copilot coding agent.)
> In the repo where we're building the agent, the agent itself is actually the #5 contributor - so we really are using Copilot coding agent to build Copilot coding agent ;)
Really cool, thanks for sharing! Would you perhaps consider implementing something like these stats that aider keeps on "aider writing itself"? - https://aider.chat/HISTORY.html
> In the repo where we're building the agent, the agent itself is actually the #5 contributor - so we really are using Copilot coding agent to build Copilot coding agent ;)
Thats a fun stat! Are humans in the #1-4 slots? Its hard to know what processes are automated (300 repos sounds like a lot of repos!).
Thank you for sharing the numbers you can. Every time a product launch is announced, I feel like its a gleeful announcement of a decrease of my usefulness. I've got imposter syndrome enough, perhaps Microsoft might want to speak to the developer community and let us know what they see happening? Right now its mostly the pink slips that are doing the speaking.
How strong was the push from leadership to use the agents internally?
As part of the dogfooding I could see them really pushing hard to try having agents make and merge PRs, at which point the data is tainted and you don't know if the 1,000 PRs were created or merged to meet demand or because devs genuinely found it useful and accurate.
I'd like a breakdown of this phrase, how much human work vs Copilot and in what form, autocomplete vs agent. It's not specified seems more like a marketing trickery than real data
> In the repo where we're building the agent, the agent itself is actually the #5 contributor
How does this align with Microsoft's AI safety principals? What controls are in place to prevent Copilot from deciding that it could be more effective with less limitations?
What's the motivation for restricting to Pro+ if billing is via premium requests? I have a (free, via open source work) Pro subscription, which I occasionally use. I would have been interested in trying out the coding agent, but how do I know if it's worth $40 for me without trying it ;).
Question you may have a very informed perspective on:
where are we wrt the agent surveying open issues (say, via JIRA) and evaluating which ones it would be most effective at handling, and taking them on, ideally with some check-in for conirmation?
Or, contrariwise, from having product management agents which do track and assign work?
Is Copilot _enforced_ as the only option for an AI coding agent? Or can devs pick-and-choose whatever tool they prefer
I'm interested in the [vague] ratio of {internallyDevlopedTool} vs alternatives - essentially the "preference" score for internal tools (accounting for the natural bias towards ones own agent for testing/QA/data purposes). Any data, however vague is necessary, would be great.
(and if anybody has similar data for _any_ company developing their own agent, please shout out).
400 GitHub employees are using GitHub Copilot day in day out, and it comes out as #5 contributor? I wouldn't call that a success. If it is any useful, I would expect that even if a developer write 10% of their code using it, it would hold be #1 contributor in every project.
re: 300 of your repositories...
so it sounds like y'all don't use a monorepo architecture. i've been wondering if that would be a blocker to using these agents most effectively. expect some extra momentum to swing back to the multirepo approach accordingly
When I repeated to other tech people from about 2012 to 2020 that the technological singularity was very close, no one believed me. Coding is just the easiest to automate away into almost oblivion. And, too many non technical people drank the Flavor Aid for the fallacy that it can be "abolished" completely soon. It will gradually come for all sorts of knowledge work specialists including electrical and mechanical engineers, and probably doctors too. And, of course, office work too. Some iota of a specialists will remain to tune the bots, and some will remain in the fields to work with them for where expertise is absolutely required, but widespread unemployment of what were options for potential upward mobility into middle class are being destroyed and replaced with nothing. There won't be "retraining" or handwaving other opportunities for the "basket of labor", but competition of many uniquely, far overqualified people for ever dwindling opportunities.
It is difficult to get a man to understand something when his salary depends upon his not understanding it. - Upton Sinclair
> In the repo where we're building the agent, the agent itself is actually the #5 contributor - so we really are using Copilot coding agent to build Copilot coding agent ;)
From talking to colleagues at Microsoft it's a very management-driven push, not developer-driven. Friend on an Azure team had a team member who was nearly put on a PIP because they refused to install the internal AI coding assistant. Every manager has "number of developers using AI" as an OKR, but anecdotally most devs are installing the AI assistant and not using it or using it very occasionally. Allegedly it's pretty terrible at C# and PowerShell which limits its usefulness at MS.
> I want to know that Microsoft, the company famous for dog-fooding is using this day in and day out, with success
Have they tried dogfooding their dogshit little tool called Teams in the last few years? Cause if that's what their "famed" dogfooding gets us, I'm terrified to see what lays in wait with copilot.
I feel like I saw a quote recently that said 20-30% of MS code is generated in some way. [0]
In any case, I think this is the best use case for AI in programming—as a force multiplier for the developer. It’s for the best benefit of both AI and humanity for AI to avoid diminishing the creativity, agency and critical thinking skills of its human operators. AI should be task oriented, but high level decision-making and planning should always be a human task.
So I think our use of AI for programming should remain heavily human-driven for the long term. Ultimately, its use should involve enriching humans’ capabilities over churning out features for profit, though there are obvious limits to that.
You might want to study the history of technology and how rapidly compute efficiency has increased as well as how quickly the models are improving.
In this context, assuming that humans will still be able to do high level planning anywhere near as well as an AI, say 3-5 years out, is almost ludicrous.
They have released numbers, but I can't say they are for this specific product or something else. They are apparently having AI generate "30%" of their code.
Whatever the true stats for mistakes or blunders are now, remember that this is the worst its ever going to be. And there is no clear ceiling in sight that would prevent it from quickly getting better and better, especially given the current levels of investment.
That sounds reasonable enough, but the pace or end result is by no means guaranteed.
We have invested plenty of money and time into nuclear fusion with little progress. The list of key acheivments from CERN[1] is also meager in comparison to the investment put in, especially if you consider their ultimate goal to ultimately be towards applying research to more than just theory.
I tried doing some vibe coding on a greenfield project (using gemini 2.5 pro + cline). On one hand - super impressive, a major productivity booster (even compared to using a non-integrated LLM chat interface).
I noticed that LLMs need a very heavy hand in guiding the architecture, otherwise they'll add architectural tech debt. One easy example is that I noticed them breaking abstractions (putting things where they don't belong). Unfortunately, there's not that much self-retrospection on these aspects if you ask about the quality of the code or if there are any better ways of doing it. Of course, if you pick up that something is in the wrong spot and prompt better, they'll pick up on it immediately.
I also ended up blowing through $15 of LLM tokens in a single evening. (Previously, as a heavy LLM user including coding tasks, I was averaging maybe $20 a month.)
Cline very visibly displays the ongoing cost of the task. Light edits are about 10 cents, and heavy stuff can run a couple of bucks. It's just that the tab accumulates faster than I expect.
I think that models are gonna commoditize, if they haven't already. The cost of switching over is rather small, especially when you have good evals on what you want done.
Also there's no way you can build a business without providing value in this space. Buyers are not that dumb.
I, too, recommend aider whenever these discussions crop up; it converted me from the "AI tools suck" side of this discussion to the "you're using the wrong tool" side.
I'd also recommend creating little `README`'s in your codebase that are mainly written with aider as the intended audience. In it, I'll explain architecture, what code makes (non-)sense to write in this directory, and so on. Has the side-effect of being helpful for humans, too.
Nowadays when I'm editing with aider, I'll include the project README (which contains a project overview + pointers to other README's), and whatever README is most relevant to the scope of my session. It's super productive.
I'm yet to find a model that beats the cost-effectiveness of Sonnet 3.7. I've tried the latest deepseek models, and while I love the price (nearly 50x cheaper?), it's just far too error-prone compared to Sonnet 3.7. It generates solid plans / architecture discussions, but, unlike Sonnet, the code it generates often confidently off-the-mark.
My tool Plandex[1] allows you to switch between automatic and manual context management. It can be useful to begin a task with automatic context while scoping it out and making the high level plan, then switch to the more 'aider-style' manual context management once the relevant files are clearly established.
I loathe using AI in a greenfield project. There are simply too many possible paths, so it seems to randomly switch between approaches.
In a brownfield code base, I can often provide it reference files to pattern match against. So much easier to get great results when it can anchor itself in the rest of your code base.
The trick for greenfield projects is to use it to help you design detailed specs and a tentative implementation plan. Just bounce some ideas off of it, as with a somewhat smarter rubber duck, and hone the design until you arrive at something you're happy with. Then feed the detailed implementation plan step by step to another model or session.
This is a popular workflow I first read about here[1].
This has been the most useful use case for LLMs for me. Actually getting them to implement the spec correctly is the hard part, and you'll have to take the reigns and course correct often.
While its being touted for Greenfield projects I've notices a lot of failures when it comes to bootstrapping a stack.
For example it (Gemini 2.5) really struggles with newer ecosystem like Fastapi when wiring libraries like SQLAlchemy, Pytest, Python-playwright, etc., together.
I find more value in bootstrapping myself, and then using it to help with boiler plate once an effective safety harness is in place.
I've vibe coded small project as well using Claude Code. It's about visitors registration at the company. Simple project, one form, a couple of checkboxes, everything is stored in sqlite + has endpoint for getting .xlsx.
Initial cost was around $20 USD, which later grew to (mostly polishing) $40 with some manual work.
I've intentionally picked up simple stack: html+js+php.
A couple of things:
* I'd say I'm happy about the result from product's perspective
* Codebase could be better, but I could not care less about in this case
* By default, AI does not care about security unless I specifically tell it
* Claude insisted on using old libs. When I've specifically told it to use the latest and greatest, it upgraded them but left code that works just with an old version. Also it mixed latest DaisyUI with some old version of tailwindcss :)
On one hand it was super easy and fun to do, on the other hand if I was a junior engineer, I bet it would have cost more.
If you want to use Cline and are at all price sensitive (in these ranges) you have to do manual context management just for that reason. I find that too cumbersome and use Windsurf (currently with Gemini 2.5 pro) for that reason.
I think it's just that it's not end-to-end trained on architecture because the horizon is too short. It doesn't have the context length to learn the lessons that we do about good design.
> I noticed that LLMs need a very heavy hand in guiding the architecture, otherwise they'll add architectural tech debt. One easy example is that I noticed them breaking abstractions
That doesn’t matter anymore when you’re vibe coding it. No human is going to look at it anyway.
It can all be if/else on one line in one file. If it works and if the LLMs can work at, iterate and implement new business requirements, while keeping performance and security - code structure, quality and readability don’t matter one bit.
Customers don’t care about code quality and the only reason businesses used to care is to make it less money consuming to build and ship new things, so they can make more money.
Nope - I use a-la-carte pricing (through openrouter). I much prefer it over a subscription, as there are zero limits, I pay only for what I use, and there is much less of a walled garden (I can easily switch between Anthropic, Google, etc).
I wish they optimized things before adding more crap that will slow things down even more. The only thing that's fast with copilot is the autocomplete, it sometimes takes several minutes to make edits on a 100 line file regardless of the model I pick (some are faster than others). If these models had a close to 100% hit rate this would be somewhat fine, but going back and forth with something that takes this long is not productive. It's literally faster to open claude/chatgpt on a new tab and paste the question and code there and paste it back into vscode than using their ask/edit/agent tools.
I've cancelled my copilot subscription last week and when it expires in two weeks I'll mostly likely shift to local models for autocomplete/simple stuff.
My experience has mostly been the opposite -- changes to several-hundred-line files usually only take a few seconds.
That said, months ago I did experience the kind of slow agent edit times you mentioned. I don't know where the bottleneck was, but it hasn't come back.
I'm on library WiFi right now, "vibe coding" (as much as I dislike that term) a new tool for my customers using Copilot, and it's snappy.
I've had this too, especially it getting stuck at the very end and just.. never finishing. Once the usage-based billing comes into effect I think I'll try cursor again.
What local models are you using? The local models I tried for autocomplete were unusable, though based on aiders benchmark I never really tried with larger models for chat. If I could I would love to go local-only instead.
That first PR (115733) would make me quit after a week if we were to implement this crap at my job and someone forced me to babysit an AI in its PRs in this fashion. The others are also rough.
A wall of noise that tells you nothing of any substance but with an authoritative tone as if what it's doing is objective and truthful - Immediately followed by:
- The 8 actual lines of code (discounting the tests & boilerplate) it wrote to actually fix the issue is being questioned by the person reviewing the code, it seems he's not convinced this is actually fixing what it should be fixing.
- Not running the "comprehensive" regression tests at all
- When they do run, they fail
- When they get "fixed" oh-so confidently, they still fail. Fifty-nine failing checks. Some of these tests take upward of an hour to run.
So the reviewer here has to read all the generated slop in the PR description and try to grok what the PR is about, read through the changes himself anyway (thankfully it's only a ~50 line diff in this situation, but imagine if this was a large refactor of some sort with a dozen files changed), and then drag it by the hand multiple times to try fix issues it itself is causing. All the while you have to tag the AI as if it's another colleague and talk to it as if it's not just going to spit out whatever inane bullshit it thinks you want to hear based on the question asked. Test failed? Well, tests fixed! (no, they weren't)
And we're supposed to be excited about having this crap thrust on us, with clueless managers being sold on this being a replacement for an actual dev? We're being told this is what peak efficiency looks like?
Thanks, that’s really interesting to see - especially with the exchange around whether something is the problem or the symptom, where the confident tone belies the lack of understanding. As an open source maintainer I wonder about the best way to limit usage to cases where someone has time to spend on those interactions.
Thanks. I wonder what model they're using under the hood? I have such a good experience working with Cline and Claude Sonnet 3.7 and a comparatively much worse time with anything Github offers. These PRs are pretty consistent with the experience I've had in the IDE too. Incidentally, what has MSFT done to Claude Sonnet 3.7 in VSCode? It's like they lobotomized it compared to using it through Cline or the API directly. Trying to save on tokens or something?
Major scam alert, they are training on your code in private repos if you use this
You can tell because they advertise “Pro” and “Pro+” but then the FAQ reads,
> Does GitHub use Copilot Business or Enterprise data to train GitHub’s model?
> No. GitHub does not use either Copilot Business or Enterprise data to train its models.
Aka, even paid individuals plans are getting brain raped
If you're programming on Windows, your screen is being screenshotted every few seconds anyway. If you don't think OCR isn't analysing everything resembling a letter on your screen boy do I have some news for you.
I’ve been trying to use Copilot for a few days to get some help writing against code stored on GitHub.
Copilot has been pretty useless. It couldn’t maintain context for more than two exchanges.
Copilot: here’s some C code to do that
Me: convert that to $OTHER_LANGUAGE
Copilot: what code would you like me to convert?
Me: the code you just generated
Copilot: if you can upload a file or share a link to the code, I can help you translate it …
It points me in a direction that’s a minimum of 15 degrees off true north (“true north” being the goal for which I am coding), usually closer to 90 degrees. When I ask for code, it hallucinates over half of the API calls.
I’m sure you have no idea what my method is. Besides, this whole “you’re holding it wrong” mentality isn’t productive - our technology should be adapting to us, we shouldn’t need to adapt ourselves to it.
Anyway, I can just use another LLM that serves me better.
I played around with it quite a bit. it is both impressive and scary. most importantly, it tends to indiscriminately use dependencies from random tiny repos, and often enough not the correct ones, for major projects. buyer beware.
This is something I've noticed as well with different AIs. They seem to disproportionately trust data read from the web. For example, I asked to check if some obvious phishing pages were scams and multiple times I got just a summary of the content as if it was authoritative. Several times I've gotten some random chinese repo with 2 stars presented as if it was the industry standard solution, since that's what it said in the README.
On an unrelated note, it also suggested I use the "Strobe" protocol for encryption and sent me to https://strobe.cool which is ironic considering that page is all about making one hallucinate.
>They seem to disproportionately trust data read from the web.
I doubt LLM's have anything like what we would conceptualize as trust. They have information, which is regurgitated because it is activated as relevant.
That being said, many humans don't really have a strong concept of information validation as part of day to day action and thinking. Development theory talks about this in terms of 'formal operational' thinking and 'personal epistemology' - basically how does thinking happen and then how is knowledge in those models conceptualized. Learning Sciences research generally talks about Piaget and formal operational before adulthood and stages of personal epistemology in higher education.
Research consistently suggests that about 50% of adults are not able to consistently operate in the formal thinking space. The behavior you are talking about is also typical of 'absolutist' epistemic perspectives where answers are right or wrong and aren't meaningfully evaluated - just identifed as relevant or not. Evaluating the credibility of information is that it comes from a source that is trusted - most often an authority figure - it is not the role of the person knowing it.
Oh wow, that was great - particularly if I then look at my own body parts (like my palm) that I know are not moving, it's particularly disturbing. That's a really well done effect, I've seen something similar but nothing quite like that.
>On an unrelated note, it also suggested I use the "Strobe" protocol for encryption and sent me to https://strobe.cool which is ironic considering that page is all about making one hallucinate.
That's not hallucination. That's just an optical illusion.
As we've built Copilot coding agent, we've put a lot of thought and work into our security story.
One of the things we've done here is to treat Copilot's commits like commits from a first-time contributor to an open source project.
When Copilot pushes changes, your GitHub Actions workflows won't run by default, and you'll have to click the "Approve and run workflows" button in the merge box.
That gives you the chance to run Copilot's code before it runs in Actions and has access to your secrets.
(Source: I'm on the product team for Copilot coding agent.)
No, not at all. Why do people keep saying shit like these thought terminating sentences. Try to see the glass of Kool Aid please. People are trying to understand how to communicate important valuable things about failure states and you're advocating ignorance.
But rest assured that with Github Copilot Coding Agent, your codebase will develop larger and larger volumes of new, exciting, underexplored technical debt that you can't be blamed for, and your colleagues will follow you into the murky depths soon.
As peer commenters have noted, coding agent can be really good at improving test coverage when needed.
But also as a slightly deeper observation - agentic coding tools really do benefit significantly from good test coverage. Tests are a way to “box in” the agent and allow it to check its work regularly. While they aren’t necessary for these tools to work, they can enable coding agents to accomplish a lot more on your behalf.
In my experience they write a lot of pointless tests that technically increase coverage while not actually adding much more value than a good type system/compiler would.
They also have a tendency to suppress errors instead of fixing them, especially when the right thing to do is throw an error on some edge case.
In my experience it works well even without good testing, at least for greenfield projects. It just works best if there are already tests when creating updates and patches.
My buddy is at GH working on an adjacent project & he hasn't stopped talking about this for the last few days. I think I've been reminded to 'make sure I tune into the keynote on Monday' at least 8 times now.
I gave up trying to watch the stream after the third authentication timeout, but if I'd known it was this I'd maybe have tried a fourth time.
I love Copilot in VSCode. I have it set to use Claude most of the time, but it let's you pick your fav LLM, for it to use. I just open the files I'm going to refactor, type into the chat window what I want done, click 'accept' on every code change it recommends in it's answer, causing VSCode to auto-merge the changes into my code. Couldn't possibly be simpler. Then I scrutinize and test. If anything went wrong I just use GitLens to rollback the change, but that's very rare.
Especially now that Copilot supports MCP I can plug in my own custom "Tools" (i.e. Function calling done by the AI Agent), and I have everything I need. Never even bothered trying Cursor or Windsurf, which i'm sure are great too, but _mainly_ since they're just forks of VSCode, as the IDE.
Have you tried the agent mode instead of the ask mode? With just a bit more prompting, it does a pretty good job of finding the files it needs to use on its own. Then again, I've only used it in smaller projects so larger ones might need more manual guidance.
I assumed I was using 'Agent mode' but now that you mentioned it, I checked and you're right I've been in 'Ask mode' instead. oops. So thanks for the tip!
I'm looking forward to seeing how Agent Mode is better. Copilot has been such a great experience so far I haven't tried to keep up with every little new feature they add, and I've fallen behind.
I've come to the same conclusions mentioned in most of that and done most of that already. I was an early-adopter of LLM tech, and have my own coding agent system, written in python. Soon I'm about to port those tools over to MCP so that I can just use VSCode for most everything, and never even need my Gradio Chatbot that I wrote to learn how to write tools, and use tools.
My favorite tool that I've written is one that simply lets me specify named blocks by name, in a prompt, and AI figures out how to use the tool to read each block. A named block is defined like:
# block_begin MyBlock
...lines of code
# block_end
So I can just embed those blocks around the code rather change pasting into prompts.
The biggest change Copilot has done for me so far is to have me replace my VSCode with VSCodium to be sure it doesn't sneak any uploading of my code to a third party without my knowing.
I'm all for new tech getting introduced and made useful, but let's make it all opt in, shall we?
These kinds of patterns allow compute to take much more time than a single chat since it is asynchronous by nature, which I think is necessary to get to working solutions on harder problems
Yes. This is a really key part of why Copilot coding agent feels very different to use than Copilot agent mode in VS Code.
In coding agent, we encourage the agent to be very thorough in its work, and to take time to think deeply about the problem. It builds and tests code regularly to ensure it understands the impact of changes as it makes them, and stops and thinks regularly before taking action.
These choices would feel too “slow” in a synchronous IDE based experience, but feel natural in a “assign to a peer collaborator” UX. We lean into this to provide as rich of a problem solving agentic experience as possible.
I'm building RSOLV (https://rsolv.dev) as an alternative approach to GitHub's Copilot agent.
Our key differentiator is cross-platform support - we work with Jira, Linear, GitHub, and GitLab - rather than limiting teams to GitHub's ecosystem.
GitHub's approach is technically impressive, but our experience suggests organizations derive more value from targeted automation that integrates with existing workflows rather than requiring teams to change their processes. This is particularly relevant for regulated industries where security considerations supersede feature breadth. Not everyone can just jump off of Jira on moment's notice.
Curious about others' experiences with integrating AI into your platforms and tools. Has ecosystem lock-in affected your team's productivity or tool choices?
Oh, the savings calculator in your website made me sad, that's the first time I've seen it put that way.
I know it's marketing but props to you for being sincere. At least you're not hiding the intentions of your service (like others).
Yeah, the ROI calculator's target audience is the folks with the checkbook, so it needs to be a dollar figure. My _actual_ hope is that this lets engineers focus on feature work (which is typically more rewarding anyway) without constantly bashing their heads against the tech debt and maintenance work they're effectively barred from performing until it becomes emergent and actively blocking.
I know that's a bit kneejerk, but I actually think that's a pretty reasonable question.
Automating the reputation and network of an individual person doesn't seem like a good fit for an LLM, regardless of the person. But the _decisionmaking_ capacities for a position that's largely trend-following is something that's at the very least well-supported by interacting with a well-trained model.
In my mind, though, that doesn't look like a niched service that you sell to a company. That looks like a cofounder-type for someone with an idea and a technical background. If you want to build something but need help figuring out how to market and sell it, you could do a lot worse than just chatting with Claude right now and taking much of its advice.
That might just by my own lack of bizdev expertise, though.
In the early days on LLM, I had developed an "agent" using github actions + issues workflow[1], similar to how this works. It was very limited but kinda worked ie. you assign it a bug and it fired an action, did some architect/editing tasks, validated changes and finally sent a PR.
Is there anything that satisfies the people here ? Copilot today is perhaps the only AI that is actually assisting for something productive.
Microsoft, besides maybe Google and OpenAI, are the only ones that are actually exploring towards the practical usefulness of AIs. Other kiddies like Sonnet and whatnot are still chasing meaningless numbers and benchmarking scores, that sort of stuff may appeal to high school kids or immatures but burning billions of dollars and energy resources just to sound like a cool kid?
Which GitHub subscription level is required for the agent?
I found it very confusing - we have GH Business, with Copilot active. Could not find a way to upgrade our Copilot to the level required by the agent.
I tried using my personal Copilot for the purpose of trialing the agent - again, a no-go, as my Copilot is "managed" by the organization I'm part of.
Also, you will want to add more control over to who can assign things to Copilot Agent - just having write access to the repository is a poor descriminator, I think.
I'm running into the same issue. I think you have to upgrade your entire organization to "enterprise", which comes with a per seat cost increase (separate from the cost of copilot).
I don't know, I feel this is the wrong level to place the AI at this moment. Chat-based AI programming (such as Aider) offers more control, while being almost as convenient.
I recently created an course for LinkedIn Learning using generative AI for creating SDKs[0]. When I was onsite with them to record it, I found my Github Copilot calls kept failing.. with a network error. Wha?
Turns out that LinkedIn doesn't allow people onsite to to Copilot so I had to put my Mifi in the window and connect to that to do my work. It's wild.
Btw, I love working with LinkedIn and have 15+ courses with them in the last decade. This is the only issue I've ever had.. but it was the least expected one.
Is Copilot a classic case of slow megacorp gets outflanked by more creative and unhindered newcomers (ie Cursor)?
It seems Copilot could have really owned the vibe coding space. But that didn’t happen. I wonder why? Lots of ideas gummed up in organizational inefficiencies, etc?
This is a direct threat to Cursor. The smarter the models get, the less often programmers really need to dig into an IDE, even one with AI in it. Give it a couple of years and there will be a lot of projects that were done just by assigning tasks where no one even opened Cursor or anything.
Copilot Workspace could take a task, implement it and create a PR - but it had a linear, highly structured flow, and wasn't deeply integrated into the GitHub tools that developers already use like issues and PRs.
With Copilot coding agent, we're taking all of the great work on Copilot Workspace, and all the learnings and feedback from that project, and integrating it more deeply into GitHub and really leveraging the capabilities of 2025's models, which allow the agent to be more fluid, asynchronous and autonomous.
(Source: I'm the product lead for Copilot coding agent.)
Probably. Also this new feature seems like an expansion/refinement of Copilot Workspaces to better fit the classic Github UX: "assign an issue to Copilot to get a PR" sounds exactly like the workflow Copilot Workspaces wanted to have when it grew up.
I go back and forth between ChatGPT and copilot in vs code. It really makes the grammar guessing much easier in objc. It’s not as good on libraries and none existent on 3rd party libraries, but that isn’t maybe because I challenge it enough. It makes tons of flow and grammar errors which are so easy to spot that I end up using the code most of the time after a small correction. I’m optimistic about the future especially since this is only costing me $10 a month. I have dozens of iOS apps to update. All of them are basically productivity apps that I use and sell so double plus good.
Which model does it use? Will this let me select which model to use? I have seen a big difference in the type of code that different models produce, although their prompts may be to blame/credit in part.
I assume you can select whichever one you want (GPT-4o, o3-mini, Claude 3.5, 3.7, 3.7 thinking, Gemini 2.0 Flash, GPT=4.1 and the previews o1, Gemini 2.5 Pro and 04-mini), subject to the pricing multiplicators they announced recently [0].
Edit: From the TFA: Using the agent consumes GitHub Actions minutes and Copilot premium requests, starting from entitlements included with your plan.
At the moment, we're using Claude 3.7 Sonnet - but we're keeping our options open to experiment with other models and potentially bring in a model picker.
(Source: I'm on the product team for Copilot coding agent.)
I was trying to find information on this on the internet and couldn't find any, thanks for providing. Interestingly enough Copilot coding agent on github.com repeatedly could not complete css changes correctly, when I switched to Agent mode in the project IDE with Claude 3.7 it was able to complete it in one round, so I assumed that there was a different model.
In my experience using Claude Sonnet 3.7 in GitHub Copilot extension in VSCode, the model produced hideously verbose code, completely unnecessary stuff. GPT-4.1 was a breath of fresh air.
So far, i am VERY unimpressed by this. It gets everything completely wrong and tells me lies and completely false information about my code. Cursor is 100000000x better.
In hindsight it was a mistake that Google killed Google Code. Then again, I guess they wouldn't have put enough effort into it to develop into a real GitHub alternative.
Now Microsoft sits on a goldmine of source code and has the ability to offer AI integration even to private repositories. I can upload my code into a private repo and discuss it with an AI.
The only thing Google can counter with would be to build tools which developers install locally, but even then I guess that the integration would be limited.
And considering that Microsoft owns the "coding OS" VS Code, it makes Google look even worse. Let's see what they come up with tomorrow at Google I/O, but I doubt that it will be a serious competition for Microsoft. Maybe for OpenAI, if they're smart, but not for Microsoft.
You win some you lose some. Google could have continued with Google code. Microsoft could've continued with their phone OS. It is difficult to know when to hold and when to fold.
I kind of love the idea that all of this works in the familiar flow of raising an issue and having a magic coder swoop in and making a pull request.
At the same time, I have been spoiled by Cursor. I feel I would end up preferring that the magic coder is right there with me in the IDE where I can run things and make adjustments without having to do a followup request or comment on a line.
So can I switch this to high contrast Black on White on mobile instead? I cannot read any of this (in the bright sunlight where I am) without pulling it through a reader app. People do get why books and other reading materials are not published grey on black, right?
I wonder what the coding agent story will be for bespoke hardware. For instance I'd like to test somethings out on a specific gpu which isnt available on github. Can I configure my own runners and hope for the beat? What about bespoke microcontroller?
How does that compare to using agent mode in VS Code?
Is the main difference that the files are being edited remotely instead of on your own machine, or is there something different about the AI powering the remote agent compared to the local one?
I have been so far disappointed by copilot's offerings. It's just not good enough for anything valuable. I don't want you to write my getter and setter. And call it a day.
It could be an amazing product. But the aggressive marketing approach from Microsoft plastering "CoPilot" everywhere makes me want to try every alternative.
How good does your test suite and code base have to be for the agent to verify re fix properly including testing things to at can be broken else where?
I'm honestly surprised by so much hate. IMHO it's more important to look at 1) the progress we've made + what this can potentially do in 5 years and 2) how much it's already helping people write code than dismissing it based on its current state.
Copilot pushes its work to a branch and creates a pull request, and then it's up to you to review its work, approve and merge.
Copilot literally can't push directly to the default branch - we don't give it the ability to do that - precisely because we believe that all AI-generated code (just like human generated code) should be carefully reviewed before it goes to production.
(Source: I'm the product lead for Copilot coding agent.)
> Once Copilot is done, it’ll tag you for review. You can ask Copilot to make changes by leaving comments in the pull request.
To me, this reads like it'll be a good junior and open up a PR with its changes, letting you (the issue author) review and merge. Of course, you can just hit "merge" without looking at the changes, but then it's kinda on you when unreviewed stuff ends up in main.
In my experience in VSCode, Claude 3.7 produced more unsolicited slop, whereas GPT-4.1 didn't. Claude aggressively paid attention to type compatibility. Each model would have its strengths.
> Copilot excels at low-to-medium complexity tasks in well-tested codebases, from adding features and fixing bugs to extending tests, refactoring, and improving documentation.
Bounds bounds bounds bounds. The important part for humans seems to be maintaining boundaries for AI. If your well-tested codebase has the tests built thru AI, its probably not going to work.
I think its somewhat telling that they can't share numbers for how they're using it internally. I want to know that Microsoft, the company famous for dog-fooding is using this day in and day out, with success. There's real stuff in there, and my brain has an insanely hard time separating the trillion dollars of hype from the usefulness.
We've been using Copilot coding agent internally at GitHub, and more widely across Microsoft, for nearly three months. That dogfooding has been hugely valuable, with tonnes of valuable feedback (and bug bashing!) that has helped us get the agent ready to launch today.
So far, the agent has been used by about 400 GitHub employees in more than 300 our our repositories, and we've merged almost 1,000 pull requests contributed by Copilot.
In the repo where we're building the agent, the agent itself is actually the #5 contributor - so we really are using Copilot coding agent to build Copilot coding agent ;)
(Source: I'm the product lead at GitHub for Copilot coding agent.)
> we've merged almost 1,000 pull requests contributed by Copilot
I'm curious to know how many Copilot PRs were not merged and/or required human take-overs.
20 replies →
> In the repo where we're building the agent, the agent itself is actually the #5 contributor - so we really are using Copilot coding agent to build Copilot coding agent ;)
Really cool, thanks for sharing! Would you perhaps consider implementing something like these stats that aider keeps on "aider writing itself"? - https://aider.chat/HISTORY.html
1 reply →
> In the repo where we're building the agent, the agent itself is actually the #5 contributor - so we really are using Copilot coding agent to build Copilot coding agent ;)
Thats a fun stat! Are humans in the #1-4 slots? Its hard to know what processes are automated (300 repos sounds like a lot of repos!).
Thank you for sharing the numbers you can. Every time a product launch is announced, I feel like its a gleeful announcement of a decrease of my usefulness. I've got imposter syndrome enough, perhaps Microsoft might want to speak to the developer community and let us know what they see happening? Right now its mostly the pink slips that are doing the speaking.
2 replies →
How strong was the push from leadership to use the agents internally?
As part of the dogfooding I could see them really pushing hard to try having agents make and merge PRs, at which point the data is tainted and you don't know if the 1,000 PRs were created or merged to meet demand or because devs genuinely found it useful and accurate.
> 1,000 pull requests contributed by Copilot
I'd like a breakdown of this phrase, how much human work vs Copilot and in what form, autocomplete vs agent. It's not specified seems more like a marketing trickery than real data
4 replies →
So I need to ask: what is the overall goal of your project? What will you do in, say, 5 years from now?
45 replies →
> In the repo where we're building the agent, the agent itself is actually the #5 contributor
How does this align with Microsoft's AI safety principals? What controls are in place to prevent Copilot from deciding that it could be more effective with less limitations?
7 replies →
Yeah, Product Managers always swear by their products.
What's the motivation for restricting to Pro+ if billing is via premium requests? I have a (free, via open source work) Pro subscription, which I occasionally use. I would have been interested in trying out the coding agent, but how do I know if it's worth $40 for me without trying it ;).
1 reply →
Question you may have a very informed perspective on:
where are we wrt the agent surveying open issues (say, via JIRA) and evaluating which ones it would be most effective at handling, and taking them on, ideally with some check-in for conirmation?
Or, contrariwise, from having product management agents which do track and assign work?
2 replies →
Is Copilot _enforced_ as the only option for an AI coding agent? Or can devs pick-and-choose whatever tool they prefer
I'm interested in the [vague] ratio of {internallyDevlopedTool} vs alternatives - essentially the "preference" score for internal tools (accounting for the natural bias towards ones own agent for testing/QA/data purposes). Any data, however vague is necessary, would be great.
(and if anybody has similar data for _any_ company developing their own agent, please shout out).
Why don't you focus on automating your CEO's job, a comparatively easy task, rather than automating your fellow engineer's jobs?
2 replies →
Welp....Github was good product while it lasted.
5 replies →
400 GitHub employees are using GitHub Copilot day in day out, and it comes out as #5 contributor? I wouldn't call that a success. If it is any useful, I would expect that even if a developer write 10% of their code using it, it would hold be #1 contributor in every project.
2 replies →
re: 300 of your repositories... so it sounds like y'all don't use a monorepo architecture. i've been wondering if that would be a blocker to using these agents most effectively. expect some extra momentum to swing back to the multirepo approach accordingly
What model does it use? gpt-4.1? Or can it use o3 sometimes? Or the new Codex model?
2 replies →
When I repeated to other tech people from about 2012 to 2020 that the technological singularity was very close, no one believed me. Coding is just the easiest to automate away into almost oblivion. And, too many non technical people drank the Flavor Aid for the fallacy that it can be "abolished" completely soon. It will gradually come for all sorts of knowledge work specialists including electrical and mechanical engineers, and probably doctors too. And, of course, office work too. Some iota of a specialists will remain to tune the bots, and some will remain in the fields to work with them for where expertise is absolutely required, but widespread unemployment of what were options for potential upward mobility into middle class are being destroyed and replaced with nothing. There won't be "retraining" or handwaving other opportunities for the "basket of labor", but competition of many uniquely, far overqualified people for ever dwindling opportunities.
It is difficult to get a man to understand something when his salary depends upon his not understanding it. - Upton Sinclair
7 replies →
TBF, you are more than biased to conclude this, I definitely take your opinion with an whole bottle of salt.
Without data, a comprehensive study and peers review, it's a hell no. Would GitHub willing to be at academic scrutiny to prove it?
> In the repo where we're building the agent, the agent itself is actually the #5 contributor - so we really are using Copilot coding agent to build Copilot coding agent ;)
Ah yes, the takeoff.
From talking to colleagues at Microsoft it's a very management-driven push, not developer-driven. Friend on an Azure team had a team member who was nearly put on a PIP because they refused to install the internal AI coding assistant. Every manager has "number of developers using AI" as an OKR, but anecdotally most devs are installing the AI assistant and not using it or using it very occasionally. Allegedly it's pretty terrible at C# and PowerShell which limits its usefulness at MS.
[flagged]
15 replies →
> I want to know that Microsoft, the company famous for dog-fooding is using this day in and day out, with success
Have they tried dogfooding their dogshit little tool called Teams in the last few years? Cause if that's what their "famed" dogfooding gets us, I'm terrified to see what lays in wait with copilot.
I feel like I saw a quote recently that said 20-30% of MS code is generated in some way. [0]
In any case, I think this is the best use case for AI in programming—as a force multiplier for the developer. It’s for the best benefit of both AI and humanity for AI to avoid diminishing the creativity, agency and critical thinking skills of its human operators. AI should be task oriented, but high level decision-making and planning should always be a human task.
So I think our use of AI for programming should remain heavily human-driven for the long term. Ultimately, its use should involve enriching humans’ capabilities over churning out features for profit, though there are obvious limits to that.
[0] https://www.cnbc.com/2025/04/29/satya-nadella-says-as-much-a...
How much was previously generated by intellisense and other code gen tools before AI? What is the delta?
> I feel like I saw a quote recently that said 20-30% of MS code is generated in some way. [0]
Similar to google. MS now requires devs to use ai
5 replies →
How much of that is protobuf stubs and other forms of banal autogenerate code?
4 replies →
That quote was completely misrepresented.
You might want to study the history of technology and how rapidly compute efficiency has increased as well as how quickly the models are improving.
In this context, assuming that humans will still be able to do high level planning anywhere near as well as an AI, say 3-5 years out, is almost ludicrous.
1 reply →
"I want to know that Microsoft, the company famous for dog-fooding is using this day in and day out, with success."
They just cut down their workforce, letting some of their AI people go. So, I assume there isn't that much success.
They have released numbers, but I can't say they are for this specific product or something else. They are apparently having AI generate "30%" of their code.
https://techcrunch.com/2025/04/29/microsoft-ceo-says-up-to-3...
That article is wrong, that is not what was said.
2 replies →
> Microsoft, the company famous for dog-fooding
This was true up around 15 years ago. Hasn't been the case since.
That's great, our leadership is heavily pushing ai-generated tests! Lol
Whatever the true stats for mistakes or blunders are now, remember that this is the worst its ever going to be. And there is no clear ceiling in sight that would prevent it from quickly getting better and better, especially given the current levels of investment.
That sounds reasonable enough, but the pace or end result is by no means guaranteed.
We have invested plenty of money and time into nuclear fusion with little progress. The list of key acheivments from CERN[1] is also meager in comparison to the investment put in, especially if you consider their ultimate goal to ultimately be towards applying research to more than just theory.
[1] https://home.cern/about/key-achievements
I tried doing some vibe coding on a greenfield project (using gemini 2.5 pro + cline). On one hand - super impressive, a major productivity booster (even compared to using a non-integrated LLM chat interface).
I noticed that LLMs need a very heavy hand in guiding the architecture, otherwise they'll add architectural tech debt. One easy example is that I noticed them breaking abstractions (putting things where they don't belong). Unfortunately, there's not that much self-retrospection on these aspects if you ask about the quality of the code or if there are any better ways of doing it. Of course, if you pick up that something is in the wrong spot and prompt better, they'll pick up on it immediately.
I also ended up blowing through $15 of LLM tokens in a single evening. (Previously, as a heavy LLM user including coding tasks, I was averaging maybe $20 a month.)
> I also ended up blowing through $15 of LLM tokens in a single evening.
This is a feature, not a bug. LLMs are going to be the next "OMG my AWS bill" phenomenon.
Cline very visibly displays the ongoing cost of the task. Light edits are about 10 cents, and heavy stuff can run a couple of bucks. It's just that the tab accumulates faster than I expect.
14 replies →
Especially at companies (hence this github one), where the employees don't care about cost because it's the boss' credit card.
I think that models are gonna commoditize, if they haven't already. The cost of switching over is rather small, especially when you have good evals on what you want done.
Also there's no way you can build a business without providing value in this space. Buyers are not that dumb.
1 reply →
> I also ended up blowing through $15 of LLM tokens in a single evening.
Consider using Aider, and aggressively managing the context (via /add, /drop and /clear).
https://aider.chat/
I, too, recommend aider whenever these discussions crop up; it converted me from the "AI tools suck" side of this discussion to the "you're using the wrong tool" side.
I'd also recommend creating little `README`'s in your codebase that are mainly written with aider as the intended audience. In it, I'll explain architecture, what code makes (non-)sense to write in this directory, and so on. Has the side-effect of being helpful for humans, too.
Nowadays when I'm editing with aider, I'll include the project README (which contains a project overview + pointers to other README's), and whatever README is most relevant to the scope of my session. It's super productive.
I'm yet to find a model that beats the cost-effectiveness of Sonnet 3.7. I've tried the latest deepseek models, and while I love the price (nearly 50x cheaper?), it's just far too error-prone compared to Sonnet 3.7. It generates solid plans / architecture discussions, but, unlike Sonnet, the code it generates often confidently off-the-mark.
8 replies →
My tool Plandex[1] allows you to switch between automatic and manual context management. It can be useful to begin a task with automatic context while scoping it out and making the high level plan, then switch to the more 'aider-style' manual context management once the relevant files are clearly established.
1 - https://github.com/plandex-ai/plandex
Also, a bit more on auto vs. manual context management in the docs: https://docs.plandex.ai/core-concepts/context-management
I loathe using AI in a greenfield project. There are simply too many possible paths, so it seems to randomly switch between approaches.
In a brownfield code base, I can often provide it reference files to pattern match against. So much easier to get great results when it can anchor itself in the rest of your code base.
The trick for greenfield projects is to use it to help you design detailed specs and a tentative implementation plan. Just bounce some ideas off of it, as with a somewhat smarter rubber duck, and hone the design until you arrive at something you're happy with. Then feed the detailed implementation plan step by step to another model or session.
This is a popular workflow I first read about here[1].
This has been the most useful use case for LLMs for me. Actually getting them to implement the spec correctly is the hard part, and you'll have to take the reigns and course correct often.
[1]: https://harper.blog/2025/02/16/my-llm-codegen-workflow-atm/
3 replies →
The trouble occurs when the brownfield project is crap already.
While its being touted for Greenfield projects I've notices a lot of failures when it comes to bootstrapping a stack.
For example it (Gemini 2.5) really struggles with newer ecosystem like Fastapi when wiring libraries like SQLAlchemy, Pytest, Python-playwright, etc., together.
I find more value in bootstrapping myself, and then using it to help with boiler plate once an effective safety harness is in place.
I've vibe coded small project as well using Claude Code. It's about visitors registration at the company. Simple project, one form, a couple of checkboxes, everything is stored in sqlite + has endpoint for getting .xlsx.
Initial cost was around $20 USD, which later grew to (mostly polishing) $40 with some manual work.
I've intentionally picked up simple stack: html+js+php.
A couple of things:
* I'd say I'm happy about the result from product's perspective * Codebase could be better, but I could not care less about in this case * By default, AI does not care about security unless I specifically tell it * Claude insisted on using old libs. When I've specifically told it to use the latest and greatest, it upgraded them but left code that works just with an old version. Also it mixed latest DaisyUI with some old version of tailwindcss :)
On one hand it was super easy and fun to do, on the other hand if I was a junior engineer, I bet it would have cost more.
If you want to use Cline and are at all price sensitive (in these ranges) you have to do manual context management just for that reason. I find that too cumbersome and use Windsurf (currently with Gemini 2.5 pro) for that reason.
$15 in an evening sounds like a great deal when you consider the cost of highly-paid software engineers
> highly-paid software engineers
For now.
The money won't be flowing forever. This will cost you $6,000 a year.
3 replies →
> LLMs need a very heavy hand in guiding the architecture, otherwise they'll add architectural tech debt
I wonder if the next phase would be the rise of (AI-driven?) "linters" that check that the implementation matches the architecture definition.
And now we've come full circle back to UML-based code generation.
Everything old is new again!
I think it's just that it's not end-to-end trained on architecture because the horizon is too short. It doesn't have the context length to learn the lessons that we do about good design.
> I noticed that LLMs need a very heavy hand in guiding the architecture, otherwise they'll add architectural tech debt. One easy example is that I noticed them breaking abstractions
That doesn’t matter anymore when you’re vibe coding it. No human is going to look at it anyway.
It can all be if/else on one line in one file. If it works and if the LLMs can work at, iterate and implement new business requirements, while keeping performance and security - code structure, quality and readability don’t matter one bit.
Customers don’t care about code quality and the only reason businesses used to care is to make it less money consuming to build and ship new things, so they can make more money.
Wild take. Let’s just hand over the keys to LLMs I suppose, the fancy next token predictor is the capitan now.
2 replies →
LLMs need a very heavy hand in guiding the architecture because otherwise they'll code it in a way that even they can't maintain or expand.
1 reply →
I don’t get it? Isn’t it just a monthly fixed subscription.
For now. Who is to say in 5 years where everyone makes this THE default workflow things work go up in price?
Nope - I use a-la-carte pricing (through openrouter). I much prefer it over a subscription, as there are zero limits, I pay only for what I use, and there is much less of a walled garden (I can easily switch between Anthropic, Google, etc).
2 replies →
Average coders, terrible engineers
I wish they optimized things before adding more crap that will slow things down even more. The only thing that's fast with copilot is the autocomplete, it sometimes takes several minutes to make edits on a 100 line file regardless of the model I pick (some are faster than others). If these models had a close to 100% hit rate this would be somewhat fine, but going back and forth with something that takes this long is not productive. It's literally faster to open claude/chatgpt on a new tab and paste the question and code there and paste it back into vscode than using their ask/edit/agent tools.
I've cancelled my copilot subscription last week and when it expires in two weeks I'll mostly likely shift to local models for autocomplete/simple stuff.
My experience has mostly been the opposite -- changes to several-hundred-line files usually only take a few seconds.
That said, months ago I did experience the kind of slow agent edit times you mentioned. I don't know where the bottleneck was, but it hasn't come back.
I'm on library WiFi right now, "vibe coding" (as much as I dislike that term) a new tool for my customers using Copilot, and it's snappy.
Here's a video of what it looks like with sonnet 3.7.
https://streamable.com/rqlr84
The claude and gemini models tend to be the slowest (yes, including flash). 4o is currently the fastest but still not great.
1 reply →
I've had this too, especially it getting stuck at the very end and just.. never finishing. Once the usage-based billing comes into effect I think I'll try cursor again. What local models are you using? The local models I tried for autocomplete were unusable, though based on aiders benchmark I never really tried with larger models for chat. If I could I would love to go local-only instead.
Several minutes? Something is seriously wrong. For most models, it takes seconds.
2m27s for a partial response editing a 178 line file (it failed with an error, which seems to happen a lot with claude, but that's another issue).
https://streamable.com/rqlr84
It takes minutes for me too sometimes.
Cursor is quicker, I guess it's a response parsing thing - when they make the decision to show it in the UI.
Some example PRs if people want to look:
https://github.com/dotnet/runtime/pull/115733 https://github.com/dotnet/runtime/pull/115732 https://github.com/dotnet/runtime/pull/115762
That first PR (115733) would make me quit after a week if we were to implement this crap at my job and someone forced me to babysit an AI in its PRs in this fashion. The others are also rough.
A wall of noise that tells you nothing of any substance but with an authoritative tone as if what it's doing is objective and truthful - Immediately followed by:
- The 8 actual lines of code (discounting the tests & boilerplate) it wrote to actually fix the issue is being questioned by the person reviewing the code, it seems he's not convinced this is actually fixing what it should be fixing.
- Not running the "comprehensive" regression tests at all
- When they do run, they fail
- When they get "fixed" oh-so confidently, they still fail. Fifty-nine failing checks. Some of these tests take upward of an hour to run.
So the reviewer here has to read all the generated slop in the PR description and try to grok what the PR is about, read through the changes himself anyway (thankfully it's only a ~50 line diff in this situation, but imagine if this was a large refactor of some sort with a dozen files changed), and then drag it by the hand multiple times to try fix issues it itself is causing. All the while you have to tag the AI as if it's another colleague and talk to it as if it's not just going to spit out whatever inane bullshit it thinks you want to hear based on the question asked. Test failed? Well, tests fixed! (no, they weren't)
And we're supposed to be excited about having this crap thrust on us, with clueless managers being sold on this being a replacement for an actual dev? We're being told this is what peak efficiency looks like?
[dead]
Thanks, that’s really interesting to see - especially with the exchange around whether something is the problem or the symptom, where the confident tone belies the lack of understanding. As an open source maintainer I wonder about the best way to limit usage to cases where someone has time to spend on those interactions.
Seems amazing similar to the changes a junior would make (jump to the solution that "fixes" it in the most shallow way) at the moment
That first PR is rough. Why does it have to wait for a comment to fix failing tests?
lol, those first two… poor Stephen
Thanks. I wonder what model they're using under the hood? I have such a good experience working with Cline and Claude Sonnet 3.7 and a comparatively much worse time with anything Github offers. These PRs are pretty consistent with the experience I've had in the IDE too. Incidentally, what has MSFT done to Claude Sonnet 3.7 in VSCode? It's like they lobotomized it compared to using it through Cline or the API directly. Trying to save on tokens or something?
Major scam alert, they are training on your code in private repos if you use this
You can tell because they advertise “Pro” and “Pro+” but then the FAQ reads,
> Does GitHub use Copilot Business or Enterprise data to train GitHub’s model? > No. GitHub does not use either Copilot Business or Enterprise data to train its models.
Aka, even paid individuals plans are getting brain raped
Might have been the case, but no longer:
https://docs.github.com/en/copilot/managing-copilot/managing...
If you're programming on Windows, your screen is being screenshotted every few seconds anyway. If you don't think OCR isn't analysing everything resembling a letter on your screen boy do I have some news for you.
Windows recall is not installed by default
2 replies →
I’ve been trying to use Copilot for a few days to get some help writing against code stored on GitHub.
Copilot has been pretty useless. It couldn’t maintain context for more than two exchanges.
Copilot: here’s some C code to do that
Me: convert that to $OTHER_LANGUAGE
Copilot: what code would you like me to convert?
Me: the code you just generated
Copilot: if you can upload a file or share a link to the code, I can help you translate it …
It points me in a direction that’s a minimum of 15 degrees off true north (“true north” being the goal for which I am coding), usually closer to 90 degrees. When I ask for code, it hallucinates over half of the API calls.
Be more methodical, it isn’t magic: https://taoofmac.com/space/blog/2025/05/13/2230
I’m sure you have no idea what my method is. Besides, this whole “you’re holding it wrong” mentality isn’t productive - our technology should be adapting to us, we shouldn’t need to adapt ourselves to it.
Anyway, I can just use another LLM that serves me better.
I played around with it quite a bit. it is both impressive and scary. most importantly, it tends to indiscriminately use dependencies from random tiny repos, and often enough not the correct ones, for major projects. buyer beware.
This is something I've noticed as well with different AIs. They seem to disproportionately trust data read from the web. For example, I asked to check if some obvious phishing pages were scams and multiple times I got just a summary of the content as if it was authoritative. Several times I've gotten some random chinese repo with 2 stars presented as if it was the industry standard solution, since that's what it said in the README.
On an unrelated note, it also suggested I use the "Strobe" protocol for encryption and sent me to https://strobe.cool which is ironic considering that page is all about making one hallucinate.
>They seem to disproportionately trust data read from the web.
I doubt LLM's have anything like what we would conceptualize as trust. They have information, which is regurgitated because it is activated as relevant.
That being said, many humans don't really have a strong concept of information validation as part of day to day action and thinking. Development theory talks about this in terms of 'formal operational' thinking and 'personal epistemology' - basically how does thinking happen and then how is knowledge in those models conceptualized. Learning Sciences research generally talks about Piaget and formal operational before adulthood and stages of personal epistemology in higher education.
Research consistently suggests that about 50% of adults are not able to consistently operate in the formal thinking space. The behavior you are talking about is also typical of 'absolutist' epistemic perspectives where answers are right or wrong and aren't meaningfully evaluated - just identifed as relevant or not. Evaluating the credibility of information is that it comes from a source that is trusted - most often an authority figure - it is not the role of the person knowing it.
> ... sent me to...
Oh wow, that was great - particularly if I then look at my own body parts (like my palm) that I know are not moving, it's particularly disturbing. That's a really well done effect, I've seen something similar but nothing quite like that.
>On an unrelated note, it also suggested I use the "Strobe" protocol for encryption and sent me to https://strobe.cool which is ironic considering that page is all about making one hallucinate.
That's not hallucination. That's just an optical illusion.
Thanks for flagging this! That isn't a behavior I've seen before in testing, and I'd love to dig into it more to see what's happening.
Would you be able to drop me an email? My address is my HN login @github.com.
(I work on the product team for Copilot coding agent.)
Given that PRs run actions in a more trusted context for private repos, this is a bit concerning.
As we've built Copilot coding agent, we've put a lot of thought and work into our security story.
One of the things we've done here is to treat Copilot's commits like commits from a first-time contributor to an open source project.
When Copilot pushes changes, your GitHub Actions workflows won't run by default, and you'll have to click the "Approve and run workflows" button in the merge box.
That gives you the chance to run Copilot's code before it runs in Actions and has access to your secrets.
(Source: I'm on the product team for Copilot coding agent.)
2 replies →
So like the typical junior developer, then.
No, lol. Even the enthusiastic junior developer would go around pestering people asking if the dependency is OK.
No, not at all. Why do people keep saying shit like these thought terminating sentences. Try to see the glass of Kool Aid please. People are trying to understand how to communicate important valuable things about failure states and you're advocating ignorance.
2 replies →
"Drowning in technical debt?"
Stop fighting and sink!
But rest assured that with Github Copilot Coding Agent, your codebase will develop larger and larger volumes of new, exciting, underexplored technical debt that you can't be blamed for, and your colleagues will follow you into the murky depths soon.
> Copilot excels at low-to-medium complexity tasks
Oh cool!
> in well-tested codebases
Oh ok never mind
As peer commenters have noted, coding agent can be really good at improving test coverage when needed.
But also as a slightly deeper observation - agentic coding tools really do benefit significantly from good test coverage. Tests are a way to “box in” the agent and allow it to check its work regularly. While they aren’t necessary for these tools to work, they can enable coding agents to accomplish a lot more on your behalf.
(I work on Copilot coding agent)
In my experience they write a lot of pointless tests that technically increase coverage while not actually adding much more value than a good type system/compiler would.
They also have a tendency to suppress errors instead of fixing them, especially when the right thing to do is throw an error on some edge case.
1 reply →
In my experience it works well even without good testing, at least for greenfield projects. It just works best if there are already tests when creating updates and patches.
Have it write tests for everything and then you've got a well tested codebase.
Caveat empor, I've seen some LLMs mock the living hell out of everything, to the point of not testing much of anything. Something to be aware of.
1 reply →
You forgot the /s
My buddy is at GH working on an adjacent project & he hasn't stopped talking about this for the last few days. I think I've been reminded to 'make sure I tune into the keynote on Monday' at least 8 times now.
I gave up trying to watch the stream after the third authentication timeout, but if I'd known it was this I'd maybe have tried a fourth time.
I’m always hesitant to listen to the line coders on projects because they’re getting a heavy dose of the internal hype every day.
I’d love for this to blow past cursor. Will definitely tune in to see it.
>I’m always hesitant to listen to the line coders on projects because they’re getting a heavy dose of the internal hype every day.
I'm senior enough that I get to frequently see the gap between what my dev team thinks of our work and what actual customers think.
As a result, I no longer care at all what developers (including myself on my own projects) think about the quality of the thing they've built.
1 reply →
Word of advice: just go to YouTube and skip the MS registration tax
What specific keynote are they referring to? I'm curious, but thus far my searches have failed
MS Build is today
I love Copilot in VSCode. I have it set to use Claude most of the time, but it let's you pick your fav LLM, for it to use. I just open the files I'm going to refactor, type into the chat window what I want done, click 'accept' on every code change it recommends in it's answer, causing VSCode to auto-merge the changes into my code. Couldn't possibly be simpler. Then I scrutinize and test. If anything went wrong I just use GitLens to rollback the change, but that's very rare.
Especially now that Copilot supports MCP I can plug in my own custom "Tools" (i.e. Function calling done by the AI Agent), and I have everything I need. Never even bothered trying Cursor or Windsurf, which i'm sure are great too, but _mainly_ since they're just forks of VSCode, as the IDE.
Have you tried the agent mode instead of the ask mode? With just a bit more prompting, it does a pretty good job of finding the files it needs to use on its own. Then again, I've only used it in smaller projects so larger ones might need more manual guidance.
I assumed I was using 'Agent mode' but now that you mentioned it, I checked and you're right I've been in 'Ask mode' instead. oops. So thanks for the tip!
I'm looking forward to seeing how Agent Mode is better. Copilot has been such a great experience so far I haven't tried to keep up with every little new feature they add, and I've fallen behind.
2 replies →
Try doing https://taoofmac.com/space/blog/2025/05/13/2230, you’ll have some fun,
I've come to the same conclusions mentioned in most of that and done most of that already. I was an early-adopter of LLM tech, and have my own coding agent system, written in python. Soon I'm about to port those tools over to MCP so that I can just use VSCode for most everything, and never even need my Gradio Chatbot that I wrote to learn how to write tools, and use tools.
My favorite tool that I've written is one that simply lets me specify named blocks by name, in a prompt, and AI figures out how to use the tool to read each block. A named block is defined like:
# block_begin MyBlock ...lines of code # block_end
So I can just embed those blocks around the code rather change pasting into prompts.
The biggest change Copilot has done for me so far is to have me replace my VSCode with VSCodium to be sure it doesn't sneak any uploading of my code to a third party without my knowing.
I'm all for new tech getting introduced and made useful, but let's make it all opt in, shall we?
Care to explain? Where are they uploading code to?
Whatever servers run Copilot for code suggestions
That isn't running locally
This is quite alarming: https://www.cursor.com/security
And this one too: https://docs.github.com/en/site-policy/privacy-policies/gith...
“ Copilot excels at low-to-medium complexity tasks”
Then we have very different interpretations of what constitutes a medium complexity task
Medium complexity tasks in our training set*
These kinds of patterns allow compute to take much more time than a single chat since it is asynchronous by nature, which I think is necessary to get to working solutions on harder problems
Yes. This is a really key part of why Copilot coding agent feels very different to use than Copilot agent mode in VS Code.
In coding agent, we encourage the agent to be very thorough in its work, and to take time to think deeply about the problem. It builds and tests code regularly to ensure it understands the impact of changes as it makes them, and stops and thinks regularly before taking action.
These choices would feel too “slow” in a synchronous IDE based experience, but feel natural in a “assign to a peer collaborator” UX. We lean into this to provide as rich of a problem solving agentic experience as possible.
(I’m working on Copilot coding agent)
I'm building RSOLV (https://rsolv.dev) as an alternative approach to GitHub's Copilot agent.
Our key differentiator is cross-platform support - we work with Jira, Linear, GitHub, and GitLab - rather than limiting teams to GitHub's ecosystem.
GitHub's approach is technically impressive, but our experience suggests organizations derive more value from targeted automation that integrates with existing workflows rather than requiring teams to change their processes. This is particularly relevant for regulated industries where security considerations supersede feature breadth. Not everyone can just jump off of Jira on moment's notice.
Curious about others' experiences with integrating AI into your platforms and tools. Has ecosystem lock-in affected your team's productivity or tool choices?
Oh, the savings calculator in your website made me sad, that's the first time I've seen it put that way. I know it's marketing but props to you for being sincere. At least you're not hiding the intentions of your service (like others).
Yeah, the ROI calculator's target audience is the folks with the checkbook, so it needs to be a dollar figure. My _actual_ hope is that this lets engineers focus on feature work (which is typically more rewarding anyway) without constantly bashing their heads against the tech debt and maintenance work they're effectively barred from performing until it becomes emergent and actively blocking.
Why don't you focus on automating your CEO's job, a comparatively easy task compared to automating engineering tasks.
I know that's a bit kneejerk, but I actually think that's a pretty reasonable question.
Automating the reputation and network of an individual person doesn't seem like a good fit for an LLM, regardless of the person. But the _decisionmaking_ capacities for a position that's largely trend-following is something that's at the very least well-supported by interacting with a well-trained model.
In my mind, though, that doesn't look like a niched service that you sell to a company. That looks like a cofounder-type for someone with an idea and a technical background. If you want to build something but need help figuring out how to market and sell it, you could do a lot worse than just chatting with Claude right now and taking much of its advice.
That might just by my own lack of bizdev expertise, though.
In the early days on LLM, I had developed an "agent" using github actions + issues workflow[1], similar to how this works. It was very limited but kinda worked ie. you assign it a bug and it fired an action, did some architect/editing tasks, validated changes and finally sent a PR.
Good to see an official way of doing this.
1. https://github.com/asadm/chota
Is there anything that satisfies the people here ? Copilot today is perhaps the only AI that is actually assisting for something productive.
Microsoft, besides maybe Google and OpenAI, are the only ones that are actually exploring towards the practical usefulness of AIs. Other kiddies like Sonnet and whatnot are still chasing meaningless numbers and benchmarking scores, that sort of stuff may appeal to high school kids or immatures but burning billions of dollars and energy resources just to sound like a cool kid?
Which GitHub subscription level is required for the agent?
I found it very confusing - we have GH Business, with Copilot active. Could not find a way to upgrade our Copilot to the level required by the agent.
I tried using my personal Copilot for the purpose of trialing the agent - again, a no-go, as my Copilot is "managed" by the organization I'm part of.
Also, you will want to add more control over to who can assign things to Copilot Agent - just having write access to the repository is a poor descriminator, I think.
I'm running into the same issue. I think you have to upgrade your entire organization to "enterprise", which comes with a per seat cost increase (separate from the cost of copilot).
Yes, so it seems.
I don't know, I feel this is the wrong level to place the AI at this moment. Chat-based AI programming (such as Aider) offers more control, while being almost as convenient.
So, fun thing.. LinkedIn doesn't use Copilot.
I recently created an course for LinkedIn Learning using generative AI for creating SDKs[0]. When I was onsite with them to record it, I found my Github Copilot calls kept failing.. with a network error. Wha?
Turns out that LinkedIn doesn't allow people onsite to to Copilot so I had to put my Mifi in the window and connect to that to do my work. It's wild.
Btw, I love working with LinkedIn and have 15+ courses with them in the last decade. This is the only issue I've ever had.. but it was the least expected one.
0: https://www.linkedin.com/learning/build-with-ai-building-bet...
> LinkedIn doesn't use Copilot
They definitely use it for full-time SWEs
Source: I work there
Anthropic just announced the same thing for Claude Code, same day: https://docs.anthropic.com/en/docs/claude-code/github-action...
And Google's version: https://jules.google
God save the juniors...
Kicking the can down the road. So we can all produce more code faster but there is NSB. Most of my time isn't spent writing the code anyway.
Is Copilot a classic case of slow megacorp gets outflanked by more creative and unhindered newcomers (ie Cursor)?
It seems Copilot could have really owned the vibe coding space. But that didn’t happen. I wonder why? Lots of ideas gummed up in organizational inefficiencies, etc?
This is a direct threat to Cursor. The smarter the models get, the less often programmers really need to dig into an IDE, even one with AI in it. Give it a couple of years and there will be a lot of projects that were done just by assigning tasks where no one even opened Cursor or anything.
on a other note https://github.com/github/dmca/pull/17700 GitHub's automated auto-merged DMCA sync PRs get automated copilot reviews for every single one.
AMAZING
The worst thing about LLMs getting commoditized and becoming cheaper is seeing slop like this pollute every meaningful discussion on the internet
GitHub had this exact feature late last year itself, perhaps under a slightly different name.
I think you're probably thinking of Copilot Workspace (<https://github.blog/news-insights/product-news/github-copilo...>).
Copilot Workspace could take a task, implement it and create a PR - but it had a linear, highly structured flow, and wasn't deeply integrated into the GitHub tools that developers already use like issues and PRs.
With Copilot coding agent, we're taking all of the great work on Copilot Workspace, and all the learnings and feedback from that project, and integrating it more deeply into GitHub and really leveraging the capabilities of 2025's models, which allow the agent to be more fluid, asynchronous and autonomous.
(Source: I'm the product lead for Copilot coding agent.)
Are you thinking if Copilot Workspaces?
That seemed to drop off the Github changelog after February. I’m wondering if that team got reallocated to the copilot agent.
Probably. Also this new feature seems like an expansion/refinement of Copilot Workspaces to better fit the classic Github UX: "assign an issue to Copilot to get a PR" sounds exactly like the workflow Copilot Workspaces wanted to have when it grew up.
I go back and forth between ChatGPT and copilot in vs code. It really makes the grammar guessing much easier in objc. It’s not as good on libraries and none existent on 3rd party libraries, but that isn’t maybe because I challenge it enough. It makes tons of flow and grammar errors which are so easy to spot that I end up using the code most of the time after a small correction. I’m optimistic about the future especially since this is only costing me $10 a month. I have dozens of iOS apps to update. All of them are basically productivity apps that I use and sell so double plus good.
Which model does it use? Will this let me select which model to use? I have seen a big difference in the type of code that different models produce, although their prompts may be to blame/credit in part.
I assume you can select whichever one you want (GPT-4o, o3-mini, Claude 3.5, 3.7, 3.7 thinking, Gemini 2.0 Flash, GPT=4.1 and the previews o1, Gemini 2.5 Pro and 04-mini), subject to the pricing multiplicators they announced recently [0].
Edit: From the TFA: Using the agent consumes GitHub Actions minutes and Copilot premium requests, starting from entitlements included with your plan.
[0] https://docs.github.com/en/copilot/managing-copilot/monitori...
At the moment, we're using Claude 3.7 Sonnet - but we're keeping our options open to experiment with other models and potentially bring in a model picker.
(Source: I'm on the product team for Copilot coding agent.)
I was trying to find information on this on the internet and couldn't find any, thanks for providing. Interestingly enough Copilot coding agent on github.com repeatedly could not complete css changes correctly, when I switched to Agent mode in the project IDE with Claude 3.7 it was able to complete it in one round, so I assumed that there was a different model.
Do you at least control the prompt?
In my experience using Claude Sonnet 3.7 in GitHub Copilot extension in VSCode, the model produced hideously verbose code, completely unnecessary stuff. GPT-4.1 was a breath of fresh air.
> Copilot coding agent is rolling out to GitHub Mobile users on iOS and Android, as well as GitHub CLI.
Wait, is this going to pollute the `gh` tool? Please tell me this isn't happening.
Don't worry - this is 100% opt in. We've just added the ability to assign Copilot to an issue from `gh issue edit` and other similar commands.
(Source: I'm on the product team for Copilot coding agent.)
ubuntu@pc:~$ gh --help
Sure! How can I help you?
So far, i am VERY unimpressed by this. It gets everything completely wrong and tells me lies and completely false information about my code. Cursor is 100000000x better.
In hindsight it was a mistake that Google killed Google Code. Then again, I guess they wouldn't have put enough effort into it to develop into a real GitHub alternative.
Now Microsoft sits on a goldmine of source code and has the ability to offer AI integration even to private repositories. I can upload my code into a private repo and discuss it with an AI.
The only thing Google can counter with would be to build tools which developers install locally, but even then I guess that the integration would be limited.
And considering that Microsoft owns the "coding OS" VS Code, it makes Google look even worse. Let's see what they come up with tomorrow at Google I/O, but I doubt that it will be a serious competition for Microsoft. Maybe for OpenAI, if they're smart, but not for Microsoft.
You win some you lose some. Google could have continued with Google code. Microsoft could've continued with their phone OS. It is difficult to know when to hold and when to fold.
Gemini has some GitHub integrations
https://developers.google.com/gemini-code-assist/docs/review...
Google Cloud has a pre-GA product called "Secure Source Manager" that looks like a fork of Gitea: https://cloud.google.com/secure-source-manager/docs/overview
Definitely not Google Code, but better than Cloud Source Repositories.
Or they'll just buy Cursor
UX-wise...
I kind of love the idea that all of this works in the familiar flow of raising an issue and having a magic coder swoop in and making a pull request.
At the same time, I have been spoiled by Cursor. I feel I would end up preferring that the magic coder is right there with me in the IDE where I can run things and make adjustments without having to do a followup request or comment on a line.
So can I switch this to high contrast Black on White on mobile instead? I cannot read any of this (in the bright sunlight where I am) without pulling it through a reader app. People do get why books and other reading materials are not published grey on black, right?
I wonder what the coding agent story will be for bespoke hardware. For instance I'd like to test somethings out on a specific gpu which isnt available on github. Can I configure my own runners and hope for the beat? What about bespoke microcontroller?
How does that compare to using agent mode in VS Code? Is the main difference that the files are being edited remotely instead of on your own machine, or is there something different about the AI powering the remote agent compared to the local one?
I have been so far disappointed by copilot's offerings. It's just not good enough for anything valuable. I don't want you to write my getter and setter. And call it a day.
I think we expected disappointment with this one. (I expected it at least)[0]
But the upgraded Copilot was just in response to Cursor and Winsurf.
We'll see.
[0] https://news.ycombinator.com/item?id=43904611
Looks like their GitHub Copilot Workspace.
https://githubnext.com/projects/copilot-workspace
It could be an amazing product. But the aggressive marketing approach from Microsoft plastering "CoPilot" everywhere makes me want to try every alternative.
How good does your test suite and code base have to be for the agent to verify re fix properly including testing things to at can be broken else where?
Lately, vscode updates are all about copilot
I'm honestly surprised by so much hate. IMHO it's more important to look at 1) the progress we've made + what this can potentially do in 5 years and 2) how much it's already helping people write code than dismissing it based on its current state.
Nice
Check in unreviewed slop straight into the codebase. Awesome.
Copilot pushes its work to a branch and creates a pull request, and then it's up to you to review its work, approve and merge.
Copilot literally can't push directly to the default branch - we don't give it the ability to do that - precisely because we believe that all AI-generated code (just like human generated code) should be carefully reviewed before it goes to production.
(Source: I'm the product lead for Copilot coding agent.)
I'm waiting for the first unicorn that uses just vibe coding.
I expect it to be a security nightmare
1 reply →
> Once Copilot is done, it’ll tag you for review. You can ask Copilot to make changes by leaving comments in the pull request.
To me, this reads like it'll be a good junior and open up a PR with its changes, letting you (the issue author) review and merge. Of course, you can just hit "merge" without looking at the changes, but then it's kinda on you when unreviewed stuff ends up in main.
A good junior has strong communication skills, humility, asks many good questions, has imagination, and a tremendous amount of human potential.
1 reply →
Management: "Why aren't you going faster now that the AI generates all the code and we fired half the dev team?"
Now developers can produce 20x the slop and refactor at 5x speed.
So a 4x slowdown?
In my experience in VSCode, Claude 3.7 produced more unsolicited slop, whereas GPT-4.1 didn't. Claude aggressively paid attention to type compatibility. Each model would have its strengths.
[dead]
[dead]