GitHub Copilot vs Cloud Code: Full Breakdown & Transcript

[00:01] about how I switched from Cursor as my daily driver coding ID into GitHub Copilot, and I have to say that video didn't really age well. So, I promised you an update when June comes around, and this is that update. So, here's the

[00:16] video I was talking about, and as I said in the pinned comment there, I actually recorded this video a couple of weeks before it was set to be published. And lucky for me, Microsoft, or rather GitHub, announced a very significant

[00:29] pricing change to GitHub Copilot. So, a lot of you asked, "What about the changes when June comes around?" So, let's actually open GitHub Copilot and talk about this some more. So, here's what we are going to do. I'm going to

[00:42] take GitHub Copilot and put it to the test, and I'll compare it to Cloud Code. So, actually, before recording this video, I had used up my entire Copilot allowance for the month of June. I believe I get about 7,000 credits, and

[00:56] right now Copilot is on a credit-based system where each operation consumes a given number of credits. Previously, it was on a per request basis or something insane usage with Copilot, and was the sole reason I was using it for months

[01:09] because I could just get so much usage out of it in the monthly allowance. Now, come the month of June, this changes, and I took it for a spin one afternoon, and was able to spend 100% of my credits in a couple of hours of, let's say,

[01:21] fairly serious coding, what you would do in your typical day job. Now, I was just video and show you some stuff I was working with Cloud Code. However, when I open it today, I see that my credits have been reset. So, now we have a good

[01:35] opportunity to compare it to Cloud Code. So, here's what we're going to do. I'm going to give the same prompt to GitHub Copilot and to Cloud Code. I'll be using the latest model from Anthropic, which is Claude 3 5. It's their most capable

[01:49] model and released a couple of days ago. I'm going to set the effort to max just so that we can see how much tokens we can burn with Copilot. So, let me type out the prompt, and here's actually what I want to build. This is an application

[02:01] that uses Dapr, the distributed application runtime, and Dapr has this very cool feature called workflows, where you can create a workflow that here's what it looks like. Now, although it seems that these actually execute

[02:15] sequentially, Dapr actually turns these into durable functions. So, these are going to execute asynchronously one by one, and Dapr is going to manage this video about this at some point in the future, probably, but this is going to

[02:30] be useful enough for our example. So, in this workflow, we are processing the payment for the customer in step number two, and reserving the stock in step number three. So, what I want to do inside of our prompt is just implement

[02:44] some sort of compensating action when charging the customer completes, but updating the inventory fails for whatever reason. And here's the prompt, it, "I want you to implement a compensating action in the order

[02:58] processing workflow when step number two completes, but step number three fails. We need to refund the order's payment. You should also write test cases for properly." So, I'm going to send this off to Copilot on Cloud Fable 5. I'm

[03:13] using agent mode, and I'm using autopilot so that it doesn't ask any end to end. So, let's send this off, and while this is working, I'll switch to a different window. And here, we're going to give the same prompt to Cloud Code

[03:27] been using it for the most part. I don't know, I just like the VS Code UI a little bit better, and I'm definitely not a CLI guy, and the Cloud Code desktop app isn't just clunky, if you ask me. So, I'm setting this to use

[03:40] Fable, max effort, I'm using auto mode so everything completes to end, send off the same prompt, and just let them work, and we're going to compare the changes once they are done. Now, I do want to comment a bit on the usage that you can

[03:54] get with Cloud Code. So, I subscribed to their default $20 a month plan, which is very similar to Copilot, and you definitely get more usage out of the of anything. I know somebody's going to write this in the comment. I'm just

[04:08] sharing my experience with these tools and what's working for me right now or not. I did contemplate going back to Cursor, but that bad experience really take a look at Cloud Coding?" And and boy was I surprised when I started using

[04:22] it. You could say the AI mind virus has got to me yet again. Let's check up on Copilot and see what's going on. So, it's still working. Now, some commentary on Cloud Fable 5 while this is going on. First of all, it's incredibly capable.

[04:35] It's going to implement basically any feature that you throw at it end to end, write test cases, it's going to debug everything, run the app, and smoke test that everything works correctly. It can spin off sub-agents to do some spikes if

[04:47] it gets stuck. Just an amazing model to work with. However, it's freaking expensive, and you're probably going to burn through your $20 a month Cloud Code burn through your $20 a month Cloud Code plan in a couple of hours, which is why

[04:59] I upgraded to the $100 a month plan. And I was actually using it for a while also going to end up spending all of the limits there. So, I promptly updated to the $200 a month Cloud Code plan. And right now, I think I get enough usage

[05:14] out of it for what I'm currently working on. Now, if you don't believe me, here's my usage on the $200 a month plan, and Anthropic actually did reset all of the limits after they released Fable 5. So, this is just my usage in the couple of

[05:27] days after Fable was released. I ended up using 58% of my weekly allowance, and If we take a look at my last 7 days of usage, you can see I spent about 18

[05:39] million tokens while working on a secret project that I'm going to reveal more about in some upcoming videos. But for now, I'm going to keep it a secret while I polish everything up. Most of this usage was on Fable 5, so about, let me

[05:54] see, 13 million tokens. And then a huge majority of that was like 2 days ago when I really ran it through the roof. So, before that, my average spend was might be interesting, and let me know in the comments, I might consider making a

[06:07] separate video on how you can burn so much tokens while not producing slop and making sure that your application still works as expected. Going back to GitHub Copilot and VS Code, we can see it made some changes, and it looks like Copilot

[06:22] went on some tangent exploring something about the workflow context, which is an internal data type to Dapr Workflows, so it's probably not all too relevant to what I was asking it to build. And in the meantime, Cloud Code seems to have

[06:36] done the work. So, let's take a look at what changes it made. So, inside of the order processing workflow, you can see it added another step here. So, when step number three fails and throws a workflow task failed exception, we

[06:49] execute a compensating action by calling another activity, the refund payment activity, which it created as another class. Now, let me check if it also registered it in Program.cs. Yes, it did. So, this checks out. The inventory

[07:04] service now fails for some product. It seems to have added this for testing purposes. Then we've got some test cases here, and there's a test case for our compensating action. And it also added a bonus in the app host. So, there's a

[07:17] command on the Aspire dashboard that allows me to place an order that will be refunded. So, this will be a textbook implementation of this feature, and most definitely overkill for Cloud Fable. On the other side, Copilot is still

[07:29] It looks like it's going to be done pretty soon. So, it falls back to the same structure here, calling the refund payment activity. Now, this is the part that really won me over with GitHub Copilot, that you get to see the diff of

[07:44] it a prompt. It looks like it's going to converge to the same implementation, and it makes sense because they're using the same model. Now, Cloud Code is noticeably faster. It also used a dummy product ID in the inventory service for

[07:57] testing purposes. This is probably going to wrap up sooner or later, I don't confident that we're going to get the same solution that we did with Cloud Code, just slower. Now, the problem with Copilot is that this single prompt right

[08:13] now, which hasn't ran to its end, ate up 7% of my monthly allowance in Copilot think. Is this something that is sustainable and that you could keep using for an entire month? In my perspective, no. And you get a lot more

[08:28] value out of the baseline Cloud Code subscription. I'll leave it up to you to pushing one or the other. I'm currently using Cloud Code. I'm really happy with it. My main reasoning for this was that I was already using Anthropic models

[08:41] anyway, so why not just go to the main provider as well, as they will probably give you the best prices. Overall, we can definitely agree that the prices for these AI agents are heavily subsidized, and I'm expecting the prices to go up

[08:54] rather than go down, at least in the near future, unless compute and really looking like that's going to be the case. And I just have to say I'm a little disappointed that I can't continue working with Copilot because I

[09:08] really enjoyed the developer experience and being able to view the diffs I'm not sure if this is something that's possible with Cloud Code. As far as I know right now, it isn't, but if anybody knows a solution for this, please let me

[09:21] know. I would love to have the same experience inside of Cloud Code. So, finally, Copilot completed the work. It used up about 600 credits. Now, in Copilot's defense, I was using Claude 3 Opus, which is incredibly expensive. I

[09:34] was also using it at max effort, which is absolutely overkill for what I asked it to do, but just giving you an example of what this usage might cost you. And of what this usage might cost you. And on Pro Plus, this took 9% of my credits.

[09:47] In my opinion, this is definitely not worth it, and sadly Copilot isn't usable for me unless I could somehow have a large number of credits to spend. That being said, I still think it's an amazing product. I do love using it.

[09:59] Just the price aspect doesn't justify it for me. So, there is my June update for Copilot. Copilot out, Cloud Code in. I'm actually using a $200 a month plan as I'm working on a new project, which I hope to share more about in some

[10:14] upcoming video. Until then, thanks a lot for watching, and stay awesome.