SE Radio 732: Jason Gorman on The Effective Use of AI For Software Development

Welcome to Software Engineering Radio. I'm your host Giovanni Asproni, and today I will discuss the practices that help in making the use of AI for software development more effective with Jason Gorman. Jason has over 30 years of experience in software development, leadership and architecture. He's a long time advocate of foundational practices like the TDD, refactoring, continuous integration, and many others, and he has spent over 25 years teaching and mentoring teams. After three years of the research and data-driven experimentation, he describes himself as an AI realist. Jason, welcome to Software Engineering Radio. Is there anything that means that you'd like to add?

歡迎收聽 Software Engineering Radio。我是主持人 Giovanni Asproni。今天要和 Jason Gorman 討論哪些實務能讓 AI 更有效地用於軟體開發。Jason 擁有超過 30 年的開發、領導與架構經驗，長期倡議 TDD、重構、持續整合等基礎實務，也已教學與輔導團隊超過 25 年。經過三年以研究和資料為基礎的實驗，他稱自己為「AI 現實主義者」。Jason，歡迎來到節目；你還想補充什麼嗎？

Pretty much on the nose, yeah.

說得差不多完全準確。

Let's start with some context Now. As I said, you said that you spent three years researching and experimenting with AI to understand these capabilities for software development. Can you tell us more about that?

先補充一些背景。你說自己花了三年研究與實驗 AI，以了解它在軟體開發上的能力，可以多談談嗎？

Oh, sure. Well, OK, let's begin at the beginning, shall we? So I think it was late 2022 that ChatGPT was launched. So we're nearly four years ago now. And I kept one eye on it. And then the amount of press and hoopla about GPT 4 when that launched, I think it was in around March of the next year, 2023. And one of the things that I was hearing now that I hadn't been hearing before is how good it was at generating code other well, I need to look into this. So I started initially just doing little experiments, very off the cuff. I'll try this, I'll try that. And then getting more serious about it and doing closed loop experiments, particularly as we got into this in inverted commas, agentic era of AI code generation, where you could run the same experiment like 10 times and get a distribution and then change variables. Like what happens if I make the modules bigger or smaller? What happens if I include tests? What happens if I take tests out of the prompts, and so on and so forth.

當然。我們從頭說起。ChatGPT 大約在 2022 年底推出，距今將近四年。我起初只是持續關注，直到 2023 年 3 月左右 GPT-4 推出，媒體大量宣傳它很會產生程式碼，我才覺得必須深入了解。剛開始只是隨手做些小實驗，後來逐漸轉向嚴謹的閉環實驗。尤其進入所謂代理式程式碼生成時代後，我可以把同一實驗跑十次觀察結果分布，再改變變數，例如模組大小、提示中有沒有測試等。

But at the same time, following the research, the computer science research, but also a lot of physicists have become very interested in it because it encroaches on an area of physics called statistical mechanics. You're dealing with this enormous probabilistic, unpredictable systems. And having a physics background, I was able to read some of the research on that. And so over the years, without looking for theories, just trying stuff out and experimenting with stuff, some things have come into focus. And so I feel like I've got a better handle on it now as to what tends to work with this technology, what tends not to work with this technology, but also a better filter on what is pure hype and what actually has some basis in reality. And I'm really, I'm interested in what's real and what works. So when I talk about myself being an AI realist, what I mean is I'm just focused on what's real, what works, what's economically viable, what's environmentally viable at home, what's ethical, etcetera.

同時我也追蹤電腦科學研究。許多物理學家也開始關注這個領域，因為它觸及統計力學：我們面對的是龐大、機率性且不可預測的系統。我有物理背景，因此能閱讀部分相關研究。這些年我沒有先找理論套用，而是持續嘗試與實驗，漸漸看清哪些方法通常有效、哪些無效，也更能分辨純粹炒作和有現實基礎的主張。我關心的是真實、有效、具經濟與環境可行性，而且合乎倫理的做法；這就是我所說的 AI 現實主義。

You found some things, some practices or techniques that help in making these AI agents, AI tools a bit more reliable or let's say more effective when used in software development.

你是否找到一些實務或技巧，可以讓這些 AI 代理和工具在軟體開發中更可靠，或至少更有效？

I think maybe a bit less unreliable is the word that we're looking for here. So from a purely practical point of view, there are a number of effects that people have observed with large language models that tends to make the predictions they make more accurate and more reliable. And one of the big ones is the size of the context that they're working with. So the amount of information that's being fed in large language models are stateless. They're like web servers. You're having to either pass state back and forth, which is how we work with GPT 5 or Claude Opus or whatever. They have no memory of our conversation in the same way that a web server doesn't. That memory has to be managed and retrieved somehow. And the longer the conversation goes over, the more information we feed in, the worse they tend to perform in terms of reliability. So there are all kinds of effects that kick in. One big 1 is what they call a tension dilution. So the way that language model works is the context. They have to calculate what's called a tension between pretty much every token in the context, A statistical relationship between every token.

與其說更可靠，不如說「沒那麼不可靠」。實務上，人們觀察到幾個會影響大型語言模型預測準確性與可靠性的因素，其中一個關鍵是工作上下文的大小，也就是餵給模型的資訊量。大型語言模型本身沒有狀態，像網頁伺服器一樣；使用 GPT-5、Claude Opus 等模型時，對話記憶必須由外部管理與取回。對話越長、資訊越多，可靠性通常越差。其中一個主要現象叫注意力稀釋：模型必須計算上下文中幾乎每個 token 彼此之間的統計關係，也就是注意力。

Now that's 1000 tokens, then that's a million calculations and a million probabilities that come out the other end. If it's 100,000 tokens, then that's a massive calculation. And all these probabilities ultimately have to add up to one. That's the way the probability works. And so the more token attentions you're having to calculate, the smaller and smaller and smaller the probabilities get until they become insignificant, at which point nuance, detail, that kind of stuff just gets completely lost. So they don't perform very well with large, no matter what the vendors say is the maximum token window. The reality is there are some very good studies that back this up. They've found this. The reality is the effective maximum token limit is orders of magnitude smaller. The point at which accuracy and reliability starts dropping off a Cliff is in the order of hundreds to maybe the low thousands of tokens, maybe 100 to 1000 when they say, oh, we can go up to 400,000 tokens or even a million tokens, yeah, but you're not going to get a lot of out of sense out of the model at that kind of scale.

一千個 token 可能就意味著約一百萬次計算與一百萬個機率；十萬個 token 的計算量更龐大，而所有機率最後仍必須加總為一。需要計算的 token 關係越多，個別機率就越小，直到細節與語意差異變得無足輕重而流失。因此，不論供應商宣稱上下文視窗可達四十萬甚至一百萬 token，實際上模型在大上下文中表現並不好。研究顯示，有效上限往往低好幾個數量級；準確性與可靠性可能在數百到低數千 token，甚至約一百到一千 token 時就開始急遽下滑。

And so I guess one of the things we need to do is to use these models in such a way that we somehow minimise the context.

所以我們使用模型時，其中一項要務就是設法縮小上下文。

How do we do that? What kind of techniques do we use to do that?

那要怎麼做？可以採用哪些技巧？

The ways that we broke problems down in Agile software development, for example, they're not just compatible with AI assisted and I gentic software development, they're kind of essential. You solve one problem at a time. And when it comes to managing context or what they call context engineering, the context should only contain the information needed to solve that problem. We're doing test driven development, for example, and we've agreed a bunch of tests about an online shopping cart. So we've agreed some scenarios, some slices into the behaviour of the system, and the first slice you want to tackle is adding items to the cart. We might set up a context for writing a test that fails for adding one item to the cart. That would be the context. And then when we've seen that test fail, we might flush that context and say, OK, now we want the model to write the simplest code that will pass that test. That's a brand new task and a brand new context. Writing failing tests, writing code to pass tests are not the same thing.

敏捷開發用來拆解問題的方法不只與 AI 輔助或代理式開發相容，甚至是不可或缺的：一次只解決一個問題。所謂上下文工程，就是只放入當下問題所需的資訊。以購物車的 TDD 為例，團隊先約定若干行為情境，第一個切片是把商品加入購物車。第一個上下文只用來寫出「加入一件商品」的失敗測試；看到它失敗後就清空上下文，再要求模型寫出通過該測試的最簡單程式碼。寫失敗測試和寫通過測試的程式碼是兩個不同任務，也應使用兩個不同上下文。

So you're going through these very short steps where you're constructing the smallest, most specific context possible to execute the current task, the current step in the process in the workflow. And then when it's successfully completed that task, we go with a new context. We say, what do we need to know? What does the model need to know in order to write the code to pass the test? Well, it needs to see the code for the test, of course, and it needs to see some existing code that is relevant, if there's classes or functions that are related to this that are going to be involved. But it doesn't need to know any more than that. It needs to know the task. And very importantly, it needs to know the test, which is what are we aiming to achieve here. And I think people are discovering, and this is something I discovered quite a long time ago, large language models, for two reasons, I think perform much better when you give them tests. You say this is all we need to do and here's the test or the example that it needs to satisfy. And there are two good reasons for that. One is that's how they're trained now.

整個流程由非常短的步驟組成，每一步都建立最小、最具體的上下文。任務完成後，再為下一步建立新上下文。模型要寫出通過測試的程式碼，只需看測試、相關既有程式碼和明確任務，不需要更多資訊。尤其重要的是把目標測試交給模型。大型語言模型在有測試或使用範例時通常表現更好，一個原因是它的訓練方式本來就會把程式片段和用法或測試配對，讓模式匹配更準確。

They don't just give you a code snippet from wherever it's come from, GitHub or Stack Overflow. They will tend to pair them with examples or tests of how that code will be used so you can get a much better match when they're pattern matching. But also, I think more recently in the last year or so, they're actually using test execution to train the models. So here's the example. You complete the example, please, let's run the test. Let's see if the test passed. No, it fails, let's feed that back. So reinforcement learning has evolved. I think for that reason, it's not at all surprising that they work well with tests. But also if you're working with agentic coding where it might take multiple passes at it, having a test gives it something to converge on. Without the test, it just goes round and round in circles. But if you give it a very clear success criteria, very clear exit criteria, the completion rates that I found with closed loop experiments are significantly higher, a lot higher, never 100%, but they're pretty good. But the main thing about context is once it's achieved its goal, it's written the test. You've seen the test fail. OK, flush the context.

近年模型訓練也開始利用測試執行：請模型補完範例、執行測試、再把失敗結果回饋給它，這是強化學習演進的一部分。因此模型擅長配合測試並不意外。代理式編碼往往需要多次嘗試，有測試就有可收斂的目標；沒有測試，它很容易原地打轉。若提供清楚的成功與退出條件，我的閉環實驗顯示完成率會高出許多，雖然永遠不到百分之百。重點仍是：目標完成、測試寫好並確認失敗後，就清空上下文。

Now here's a new task that needs a new context that's only related to that task. And so I sort of break it down into these short steps, these micro steps that have every single one of which have some kind of explicit exit criteria. And once it exits, we flush the context and we construct a new context specifically to the next task.

接著是一個只與下一任務相關的新上下文。我把工作拆成這些短小的微步驟，每一步都有明確退出條件；達成後便清空上下文，再為下一個任務專門建構新上下文。

And I had some success for that, but I was finding that there are times when you're asking the Ralph loop to throw 13 and it just can't do it. I call them doom loops. It just goes round and round and round and it can't do it. The problem it's trying to solve is out of its training data distribution. And sometimes you can solve that by dropping a gear and saying, OK, let's go back into planning mode and break the problem down into if you can't throw 13, try throwing A6 and A7. They're more probable. But that doesn't always work. Sometimes you have to step in and go, OK, this problem is just out of the models distribution. It's not been trained on this. You can see this with diffusion models, the text to image models where you ask it things like can you generate an image, please of a wine glass full to the brim? And they just can't do it. And there are all kinds of examples because it's not in the training data distribution that language models just can't do, no matter how many passes you give them. So you have to be ready to intervene. You have to watch it like a walk and go, OK, it's thrashing here. It's in a doom loop, so I need to intervene.

這方法有時成功，但也會遇到無論如何都擲不出「13」的情況，我稱之為 doom loop：代理不斷打轉，因為問題超出訓練資料分布。有時可以降檔，回到規劃模式，把「擲出 13」拆成更可能成功的「6 和 7」；但並非總有效。這就像影像模型很難產生一只斟滿到杯緣的酒杯，因為訓練資料裡缺少相應模式。你必須隨時準備介入，看到模型陷入反覆掙扎時就接手。

The goal, I think of what we might call long horizon, fully autonomous agentic coding, that we give it a set of requirements and then we go to the beach, I think is complete fantasy. I've seen nobody succeeding. I've seen people succeeding at having agents run for hours. What I haven't seen yet is anyone succeed at having them do that and produce something that actually works at the end.

至於給它一組需求、自己去海灘，期待長時間、全自主的代理式編碼，我認為完全是幻想。我看過代理跑上好幾個小時，卻沒看過它最後真正產出可運作成果。

So small context ATDD will give help the model, create a test, then flash the context, create the code that solves the test. So basically trying to keep the context always small. Then we need to grow this thing. How do we work? Do we create 1 module at a time if you run our modular system?

也就是用小上下文和 TDD：先讓模型建立測試，清空上下文，再產生通過測試的程式碼。可是系統還要逐步長大；對模組化系統而言，是不是一次建立一個模組？

We do it the way that we've always done it, which is there are two sides of this. One is the map and one is the terrain. This is a message that I found really difficult to get across because test driven development and refactoring and continuous integration, though very much putting 1 foot in front of the other. And I think maybe the message we're not getting across is you do have to look where you're going and you have to have at a high level some kind of route plan that there are times when you're not sure, take a step back and just maybe do a little sketch or write it down on paper or whatever. This is the general road map, I think now, I mean, Ken Beck recently published A blog post about what he's calling Canon Test Driven developments. And in Canon Test Driven Development, there are such things as test lists, which I've talked about in his, I've got it right here just in case. In his book on TDD, he talks about test lists that you're not going into this cold. You've actually thought about the problem that you're facing and you make a little list, just a high level sketch remembering of course, the map is not the terrain. And that gives you a way to keep your place.

做法其實一如既往：一邊是地圖，一邊是地形。TDD、重構與持續整合看似只是一步接一步，但仍然必須抬頭看方向，在高層次上有大致路線。不確定時可以退一步，畫草圖或寫在紙上。Kent Beck 最近談到 canonical TDD，其中包括他早在 TDD 書中寫過的測試清單。你並不是毫無準備就開始，而是先思考問題、列出高層次小清單；同時記得地圖不等於地形，清單只是幫助你掌握目前與下一步。

OK, That's the step we're doing now. This is the step we're doing next. Remembering, of course, that as we go, we're going to learn and we're going to adapt and that map is going to change. And so after the fact, we've got practices like refactoring, continuous architecture and emerging architecture, that kind of deal of let's visualize what we've got, let's think about it, let's talk about it. And surprise, surprise, when we're doing it with AI, it's exactly the same. But we need to remember the AI that large language models, let's be specific, super autocomplete is not capable of that kind of reasoning. I mean, literally not capable of it. Language models, and this is a problem with deep neural networks, generally cannot be trained on long range patterns. They cannot be trained on things that are far apart on the bigger picture because those probabilities are so low, so insignificant. And large language models are kind of like averaging machines. We see this with model collapse. You take a model's output and trade another model on it and it just gets Dumber.

進行過程中我們會學習、調整，地圖也會改變；之後還可透過重構、持續架構與演進式架構，把現況視覺化、思考和討論。和 AI 合作也完全一樣。不過要記住，大型語言模型只是「超級自動完成」，並不具備真正的長程推理能力。深度神經網路很難學習相距遙遠的模式或整體圖像，因為相關機率太低。語言模型也像平均機器；用模型輸出去訓練另一個模型會產生模型崩潰，結果越來越笨。

And then you take that model's output and train another model on it and it gets even Dumber because they tend to produce the most probable response. So all the nuance and the detail gets lost.

再用第二個模型的輸出去訓練第三個，會更進一步退化，因為模型傾向產生最可能的答案，細節與語意差異便逐漸消失。

So basically what you are saying is that in terms of well vision, general direction on what the system shape should be is something that humans are good at. And here we can use the AI system to actually helping with the implementation of these things.

所以願景、整體方向與系統應有的形狀，是人類擅長的工作；AI 系統則可以協助實作這些構想。

Absolutely, yeah.

完全正確。

Maybe we can have the AI if we are exploring architectural ideas with the appropriate models trained on some architectural patterns or something. We can in general high level we can can use the system to help us, but we cannot really tell the system this is the architecture going and implemented.

若探索架構構想，也許可以使用在架構模式上受過適當訓練的模型，從高層次協助思考；但不能只告訴系統「這就是架構，請實作」。

There is an informational disconnect between. We could represent architecture at a higher level and they will be able to match patterns in it, but then the moment you start asking to generate code, you're a much deeper level of representation and I don't see how they can fix that disconnect. I guess. I mean, when we talk about AI, always talk about actual intelligence. You can't take the actual intelligence, the ability to see the big picture and the ability to actually reason, the ability to learn from very small number of examples, which they just can't do.

這裡有一個資訊斷層：我們能用高層次表示架構，模型也能在其中比對模式；但一旦要求產生程式碼，就進入更深層的表示，我看不出這個斷層要如何消除。談 AI 時，我總會區分真正的智慧：看見整體、真正推理，以及從極少量範例學習的能力；大型語言模型做不到這些。

And so basically whatever we do with the techniques that we use are aimed at keeping this context constrained. Well, there are other aspects as well, but keeping the context constrained is an important one.

因此，我們採用的技巧有許多目的，其中讓上下文保持受限是一項重要原則。

In scope, yeah, this is what I meant.

對，限制範圍就是我的意思。

And constrained in time as well. Yeah, to be as recent as possible.

時間也要受限，所用資訊應盡可能新近。

And you mentioned agile. So here we're talking about practices that we already use basically in agile development with this TDD, very short feedback loops, small steps. And then some things that you mention also in your writings is also continuous re factoring, continuous code review. I guess the continuous code review is important because AI is prone sometimes to generate some questionable quality in code. And so we need to make sure that things are proper. And this also means that we really need to go small because we cannot really review 1000 lines of code at the time easily.

你提到敏捷，這裡談的其實是既有敏捷實務：TDD、很短的回饋迴圈、小步前進，以及持續重構和持續程式碼審查。AI 有時會產生品質可疑的程式碼，因此審查尤其重要；這也代表步幅必須很小，因為一次審查一千行程式碼並不容易。

Or we can just not very well, not in a very kind of focused kind of way. This is the other side of it, I think is in terms of workflow. So nobody is questioning whether tools like Claude Code and Codex and others can produce a lot of code very fast. Nobody's questioning that's definitely true, much faster than humans. The downside of that is that was never the bottleneck in software development. So if you let them run, they all produce large chain sets, large batches of changes that then start hitting these downstream bottlenecks like code review and like testing and like merging as well. And So what we're seeing, and this is there's a big body of evidence being built up on this by Circle CI and by Faros who do the sort of software development telemetry people, by the Dora folks, the DevOps research and association research and assessment people. That shows a very clear picture, which is those initial gains from code being generated faster are being lost downstream to this chaos of things sitting in queues waiting to be reviewed and bugs being discovered, fixes being scheduled and all this stuff. And on top of which far more problems are leaking into production than before.

勉強可以審查，但很難專注。Claude Code、Codex 等工具確實能比人類更快地大量產生程式碼；問題是寫程式碼從來不是軟體開發的瓶頸。放任它們運作，便會形成龐大變更集和大批次工作，接著堵在下游的審查、測試與合併。CircleCI、Faros 和 DORA 等團隊累積的資料呈現出清楚圖像：最初的生成速度增益，後來被排隊審查、發現缺陷、安排修復等混亂抵消，而且流入正式環境的問題比以前更多。

So outages and production incidents are up, delays and delivery are up. This is the the Circle CI data. And if I say Pharos was seeing the same thing, which is I think the way Pharos put it in their report, starting is easy, finishing is hard, and finishing was always hard. And what generative AI has done is made starting even easier and even faster. It's a fire hose. You're drinking from a fire hose. And when you're drinking from a fire hose, the limit isn't the fire hose, The limit's you. How much can we drink at a time? So you let the fire hose go in these very short bursts. 1 mouthful the time. The same. It always was. And that was always the real speed limit in software development and continues to be how fast can we test code? How fast can we understand code comprehension debt? We're seeing that building up now with Circle CIS data there. They've been seeing over the last two or three years that the time taken to fix problems in production has been rising. And I think, I suspect a big factor in that is that nobody understands the code anymore. And there's this gap you have to leap to say, before we fix this, we need to understand it.

CircleCI 資料顯示，服務中斷、正式環境事故與交付延遲都在增加；Faros 也看見同樣現象，稱為「開始很容易，完成很困難」。完成工作原本就難，而生成式 AI 只是讓開始更快、更容易。它像消防水帶，限制不是水帶，而是人一次能喝多少，所以應只短暫開啟，一次喝一口。真正的速度上限仍是測試和理解程式碼的速度。CircleCI 還看到近幾年正式環境問題的修復時間上升，我懷疑一大原因是沒有人真正理解程式碼，修復前得先跨越理解落差。

That's an interesting one, because sometimes when you there are some people that say you know what is generated by AI, we don't really need to understand it as long as it does the right thing. On the other hand, the problem is that if we don't understand it, we don't even know if the code actually does what we want.

有些人說，只要 AI 產生的程式碼能正常工作，我們不必理解它；但若不理解，也就無法確定它是否真的做了我們要的事。

Probability is a big factor in all of this, which is it's all an interaction with. A large language model is essentially at the thrower of the dice, and if you calculate the probabilities, maybe 8 times out of 10 you get the number you want. That's good for one interaction. You may go, OK, so if we don't get it the first time, we throw it again and our probabilities go up. The problem is, and this is where I diverge from a lot of people who are working with this technology who are enthusiasts of it. The thing that makes me a realist is the chances of what they call sort of long horizon software development where I specify 20 features and go to the pub. The chances the model getting through all of those features with all the steps involved without falling over or without taking a wrong turn, and the moment it takes a wrong turn, you're now building on top of that. You're not going in the wrong direction. And I think the chances of that completing successfully are so astronomically remote that I dismissed them out of hand. I think long horizon, fully autonomous agentic coding is a fantasy and a myth. It's the part of gold at the end of the rainbow.

一切都受機率支配。每次與大型語言模型互動都像擲骰子：單一步驟也許十次有八次得到想要的結果，失敗還可再擲一次。但我不相信所謂長時間自主開發：若我指定二十項功能就去酒吧，模型要正確完成每個功能的每一步而不跌倒或轉錯方向，機率低得驚人；一旦走錯，後續成果便建立在錯誤之上。因此我把長時間、全自主的代理式編碼視為神話，就像追逐彩虹盡頭的黃金。

But the reality is there is no rainbow. It's a fool's errand.

實際上連那道彩虹都不存在；這只是在做徒勞無功的事。

I'm concerned with the reality, which is OK, let's look at the technology as it is now and what it can do now. If you're going to use this stuff, what is the smartest way to use it?

我關心的是現實：面對目前的技術和它當下能做到的事，若要使用，最聰明的方法是什麼？

As you know now there is a lot of talking about people managing agents, maybe 10s of them working at the same time and agents that work with each other to solve a problem and apparently people reviewing the code of the agents or agents reviewing each other code. So you think that this system actually is not that viable after all?

現在很多人談同時管理數十個代理，讓代理彼此協作、互相審查程式碼，再由人審查。你認為這套模式其實不可行嗎？

I've seen nothing that persuades me. Some of the larger scale ones done with cursor and done with Claude Code have indeed run for days and weeks with, as they put it, little human intervention. But they have produced broken software. The web browser that was done using I think they said like 100 or more cursor agents. Someone I couldn't be bothered, someone used the APR and GitHub to go back through the commit history. The build histories there are more than 10,000 commits and the thing broke it around commit 100. It was probably broken before that and never worked again until humans stepped in and fixed it so that it would at least compile. The problem with continuous integration and concurrency. You know this very well is that ultimately, no matter how many developers you have working in parallel or seemingly in parallel, ultimately they all have to go through the same garden gate, which is it all has to get onto the release branch. And there are two ways you can go about this. One is the discipline way, which is to have traffic lights and you wait your turn.

我還沒看到足以說服我的證據。用 Cursor 或 Claude Code 所做的大型實驗，確實能在很少人工介入下跑上幾天或幾週，卻產生了壞掉的軟體。有人用一百多個 Cursor 代理打造瀏覽器；其他人翻查其 GitHub 歷史，發現一萬多筆提交中，系統大約在第一百筆時就壞了，此後再也沒有正常運作，直到人類介入讓它至少能編譯。這牽涉持續整合與並行的根本問題：不管多少開發者平行工作，最後所有變更都得通過同一道花園門，進入發布分支。紀律做法像交通號誌，大家必須排隊等候。

So if a build's in progress, you don't push on top of that because you might be pushing on top of a broken build, or even it's not a broken build. You might be pushing changes that will conflict. So you have to wait and you certainly don't pull changes. So you can't synchronize either way until the bill succeeded. So the bill takes a few minutes. You're going to end up with every one of those agents lining up in single file, which I think is why I ended up when I experimented with Claude code with concurrent agents, I ended up quite quickly. It only took me a week to go. OK, that doesn't work to go back to. OK, I'll just have a single process and I ended up going back to Ralph Loops and then from Ralph Loops back to OK, I'll control the process myself.

建置進行時不能再推送變更，否則可能疊在壞掉的建置上，或產生衝突；建置成功前也不能拉取變更，因此雙向同步都得等。若每次建置花幾分鐘，所有代理最後會排成單列。這就是我測試 Claude Code 並行代理後，只花一週便放棄，先退回單一程序的 Ralph loop，最後又退回由自己控制流程的原因。

In that case, basically, even if the agents were working, let's say yeah, the continuous integration would act as the bottleneck.

也就是說，即使代理本身能工作，持續整合仍會成為瓶頸。

Yeah, it's like it doesn't matter how many lanes you add to the motorway that goes to the ferry port in Kent. There's only one boat and they're all going to go on the boat one at a time. So it doesn't matter how many lanes you add.

這就像通往肯特郡渡輪港的高速公路：增加再多車道也沒用，因為只有一艘船，所有車還是得一輛一輛上船。

And so basically, I guess then from what you're saying that even if we try using some technical practices like you mentioned before with TDD, continuous testing still is a problem because having say hundreds of agents, they will still have the problem of the bottleneck at the CI level.

因此即使用 TDD、持續測試等實務，數百個代理仍會在 CI 層遇到瓶頸。

Yes. And also the fact that they are individually in parallel changing the code with their picture of that code which is diverging from everybody else's picture. So you have that, as with all things concurrent, the real question to what extent do they depend on each other and therefore how often do they need to synchronize? And that's, again, it's nothing new. We still kill the old way, which is OK, then small teams, and maybe it's better if the team actually works in a single thread so that we all work together. This is, I guess, the value proposition of mob programming or teaming, which is maybe it's better if we bring different expertise to bear at the same time on a single thread of execution, rather than trying to have it work in parallel. And I think we're discovering the same limitation with agentic programming, which is the synchronisation problem. The mythical Man Month always gets us in the end.

是的，而且每個代理都在平行修改自己眼中的程式碼版本，逐漸與其他版本分歧。所有並行系統的核心問題都是：彼此依賴到什麼程度、需要多常同步？答案並不新鮮：使用小團隊，甚至讓團隊在單一工作執行緒上共同工作。這正是 mob programming 或 teaming 的價值——在同一條執行線上同時集結不同專長，而不是各自平行。代理式編程碰到的也是同步問題；《人月神話》最後總會追上我們。

Do you think that actually having agents working on separate modules or something, if the modularization is done properly would actually help?

如果模組化做得好，讓代理分別處理不同模組，會不會有幫助？

Yes, And this is something that I've tested and seen and I think a lot of other people have as well. Separation of concerns, modularity, it's always been the secret sauce, but but when you speed the whole thing up, it becomes so important, especially when it comes to context management because the blast radius of the changes you're making has a massive impact on how big the context is going to be. If you can change one source file and that change is contained to that source file, are you going to get a small context? If you don't have that kind of separation of concerns, if changing that file there is going to involve changing all these other files, you've got a big context and then you're going to run into all these problems with big contexts. So yeah, separation of concerns super duper important.

會，這是我和許多人都測試並觀察到的。關注點分離與模組化一向是關鍵，當整體速度加快後更是如此，尤其會直接影響上下文管理。變更的爆炸半徑決定上下文大小：若修改一個原始檔，影響能完全侷限在該檔案，上下文就很小；若缺乏關注點分離，一處修改牽連許多檔案，上下文就會膨脹並引發前述問題。因此關注點分離極為重要。

Several people are saying that AI is an amplifier. If you are good, we'll help you be maybe a bit better or a lot better. But if you are not so good, you'll actually experience even more trouble. So what do you think about that and also how much more productive these teams actually become?

不少人說 AI 是放大器：能力好的人會變得更好，基礎不好的人則會遭遇更多麻煩。你怎麼看？這些團隊實際上提高了多少生產力？

That's a very good question. So I could only go on the data here. So the Dora folks have found fairly consistently over the last three years that as you say, this technology is an amplifier, that there is a small percentage of teams that were already high performing who appear to experience a sort of a productivity boost. Lead times get shorter, throughput gets higher, and it's not at the expense of the stability of their releases. What we can't rule out is whether this is a result of the technology or result of what the technology is inflicting on their processes. That it's a kind of a load test for software development methodology for the way you do software development. That if they were, here's the thing, if there are bottlenecks in your process, attaching this code generating fire hose to your plumbing is going to show up where those bottlenecks are. And I can't rule that. And I've watched teams do this where they've said, well, we are taking smaller steps now and we're doing code review more often and so on and so forth.

我只能依資料回答。DORA 過去三年相當一致地發現，這項技術確實像放大器：少數原本就高效能的團隊，前置時間縮短、吞吐量提高，而且發布穩定性沒有犧牲。不過我們無法排除真正原因不是工具本身，而是工具對流程施加的壓力；它可能等同對開發方法做負載測試。若流程有瓶頸，把程式碼生成消防水帶接上管線，瓶頸立刻會顯現。我看過團隊因此改採更小步驟、更頻繁審查。

And I, I feel like maybe this technology is highlighted where they were experiencing those kind of bottlenecks to some small extent and that they've addressed them because they're measuring the outcomes. And I think that's the important thing. They are measuring things like lead times and they're saying the dials go in the wrong direction. They go right, OK, we need to review code more often. One thing that's delighted me of the last couple of years is how many teams now who just ignored me on things like mutation testing have suddenly become really interested in mutation testing and really interested in static analysis and speeding up these feedback loops. So I can't rule that out. And if that's the effect that it's having, if the practical effect is that it's a stress test for your process, as one person put it to me recently, then I don't think that's a bad thing. It's highlighting areas where you're a week where it is a bad thing is when they're not measuring those outcomes. So they're not measuring, to use that analogy of the fire hose, they're connecting the fire hose to their homes plumbing. They're not measuring the pressure of the water coming out of the shower.

我懷疑工具只是凸顯瓶頸，而團隊因為持續量測結果而修正了流程。當前置時間等指標往錯誤方向走，他們就更頻繁審查。令我欣慰的是，過去忽視突變測試、靜態分析和加快回饋迴圈的團隊，近兩年突然變得很有興趣。若 AI 的實際效果是替流程做壓力測試，這不是壞事；真正糟糕的是不量測成果，只看到消防水帶流量變大，卻不量家中淋浴水壓。

They're measuring the pressure of the water coming out of the hose and go, oh, there's lots more water coming out of the hose now. So that's greater productivity. So I see all these teams who are measuring more lines of code, more commits, more pull requests, more features, more throughput, more stuff going into the system, and they're not measuring what's coming out of the system and the value of what it's worth. And I think those teams are probably blissfully unaware that this technology is making them worse. It's an interesting irony if you think about it. One of the reasons that they're probably not high performing in that sort of Dora classification is because they don't measure those things, They don't pay attention to those things. And because they don't pay attention to those things, they're blissfully unaware that AI is making them worse.

許多團隊只量程式碼行數、提交、拉取請求、功能和輸入系統的工作量，卻不量系統輸出的價值，因此可能完全沒察覺 AI 正讓他們變差。諷刺的是，他們之所以不是 DORA 分類中的高效能團隊，往往正因為不量也不關心這些結果；也正因如此，才會對 AI 帶來的退步渾然不覺。

Yeah, because they're measuring what they can measure easily, at least. Yeah, I see some wonderful posts. I saw one recently where someone was saying thanks to Claude Code, we've gone from this is a team of developers. The team has gone from I think it was 50 commits a week to 500.

對，因為他們只量容易量的東西。我最近看到有人炫耀，團隊用了 Claude Code 後，每週提交從大約五十次增加到五百次。

It still doesn't tell you anything about.

這仍然沒有告訴你任何有用的事。

It tells you nothing. It just tells. They could be that the batch sizes are much smaller. I suspect that's not what they're not they're actually doing. But you could just say, well, we're just committing a few small changes at a time.

確實什麼也沒說。或許只是每批變更變小了——雖然我懷疑實際情況並非如此——也可能只是更頻繁提交少量修改。

So now a question about working in teams because when you look around a lot of these AI advice is seems to be at least aimed at individuals. Code is produced by teams and has to be done by teams. Because you can imagine in a company if a company has a single person that say manages 100 agents, great and then this person goes on only this what happens, Oh, there is a problem, what happens? So there has to be some redundancy there. So, as always, to be a team. Now, how can we integrate these AI tools in a team environment? Well, what is your take on that?

多數 AI 建議似乎針對個人，但程式碼由團隊共同產生與維護。公司若只靠一個人管理一百個代理，那個人休假或出問題時怎麼辦？團隊仍需要人員冗餘。那麼，該如何把 AI 工具整合到團隊環境？

I've been using them alone. This is something that I've noted many times that we saw this over this Christmas and New Year. A lot of chief technology officers and heads of engineering went away over over the Christmas break and that gave them a bit of time to well, I better take a look at this Claude Code thing that everyone's talking about and they get the proverbial car up to 200 miles an hour on a straight Rd. with no other traffic around and they come back. I saw this, they came back into the office in January and they went to their teams and they said you will use this. This is the way it's going to be. I was so productive working by yourself with no customer, no team, no other stakeholders, no regulations, no legacy code, no dependence is well done you, yes, you can get the car up to 200 mph. That's no one's denying that these things can generate code really fast. But most software development teams are driving through heavy traffic and living in London, I know for a fact that doesn't matter how fast your car is, you're going to be going out best 9 miles an hour, sort of average speed. A bicycle is faster than a Bugatti. Faster cars does not equal faster traffic.

我主要是獨自使用。聖誕與新年假期時，不少技術長和工程主管終於有時間試 Claude Code。他們就像在無車的直路上把車開到時速兩百英里，回公司後便命令所有團隊採用。獨自一人、沒有客戶、團隊、利害關係人、法規、舊程式碼和依賴，當然能很快產生程式碼；但真實團隊是在壅塞交通中行駛。在倫敦，再快的車平均也只能跑約九英里，腳踏車甚至比 Bugatti 更快。車更快，不等於交通更快。

I think that the mistake that so many people have made is that they've taken what is a local optimization, code generation, writing code. And it's kind of our fault as an industry as well because we've so often equated software development with coding, whereas in reality coding is just a small part. Coding is the part where you are driving forwards in the rush hour traffic, but most of your journey is waiting, waiting for input, waiting for feedback, etcetera, etcetera. And we've made a terrible mistake. But also remembering that there are other cars on the road, that a lot of what we do when we're driving through cities or towns is trying not to bump into each other, that we're dealing with dependencies and mergers and conflicts and competing concerns. And that's what takes up all the time. And coding was only ever such a small part of it. I remember when the software craftsmanship movement sort of really took off in 2000 and six, 2007. There was this sudden fascination with everyone typing faster. They're all on the Mavis Beacon Typing Tutor course and we need to write code faster.

許多人把程式碼生成這個局部最佳化，誤當成整個軟體開發。業界也有責任，因為我們常把開發等同寫程式；實際上寫程式只占一小部分，旅程的大部分是在等待輸入和回饋。路上還有其他車，大家要處理依賴、合併、衝突和相互競爭的需求，這才最耗時。2006、2007 年軟體工藝運動興起時，大家也一度迷上提高打字速度，彷彿寫得更快就是開發得更快。

And I remember saying, apart from it being ableist, because I know some fantastic software developers who really get stuff done, but because of certain circumstances can only type, for example, with two fingers at a time, but at the end of a session, they've still got just as much, if not more done in real terms. So I had a problem with it for that. But the main problem I had was typing code was never the bottleneck. It's never been the bottleneck, not since we were punching holes in cards. It's never been a real problem. Looking things up has never been the bottleneck. Finding out how to call this API or use that library has never really been the bottleneck. The bottleneck has always been the part that does the heavy lifting, the dependencies that the communication, the feedback loops and the big one, the really big lever that that no team wants to pull, which is batch size. Try as I might to persuade them, say just take a smaller step, *** but that's it feels so slow, Jason, it's slowing me down. I get weirdly, we've already delivered. It's already shipped.

這種想法也忽略身心障礙者：我認識極優秀的開發者因自身狀況一次只能用兩根手指打字，實際成果卻不比別人少。更根本的是，打字從來不是瓶頸；自打孔卡時代以來就不是。查 API 或函式庫用法也不是。真正做重工的瓶頸一直是依賴、溝通、回饋迴圈，以及團隊最不願碰的巨大槓桿——批次大小。大家總覺得小步驟很慢，但往往就在他們抱怨時，成果其實已交付上線。

No, we said that basically the teams that are getting the most out of these tools. So the things that are already work in a certain way, Yeah, so we say all the values, small batches you just mentioned, then we say maybe using TDD, the kind of small steps, so keeping the context small, short iterations, continuous integration loop. So basically all the things, the technical practices that using most agile processes, all the ones that are mostly associated to extreme programming, which for the listeners is one of the most well known agile processes. In this there were one that is probably the most technical in terms of practices, is probably the only one that is still surviving very well, but nobody really talks about it because of all the practices that we do nowadays come from it in a way or another.

能從 AI 獲得最多效益的團隊，本來就採取某些工作方式：小批次、TDD、小步驟、小上下文、短迭代和持續整合。這些多半源自敏捷流程，尤其是極限編程。對聽眾而言，XP 是最知名、技術實務最完整的敏捷方法之一；它其實活得很好，只是如今大家使用的實務大多源自 XP，反而不再特別提它。

It's like extreme programming was modern art, and beyond that now it's become sort of postmodern, but it's part of art, but we don't mention it anymore.

極限編程就像現代藝術；如今已進入後現代，仍是藝術的一部分，只是大家不再提它的名字。

And now there is a question here because, OK, the teams that are doing this well, they can get something out of AI. We said there is the Dora report as well. Maybe in the report we see correlation. We don't know exactly where the causation is from what you said. You know what is that is make this is of AI, why this is of AI make things better for those teams. But there is correlation. They use it and seem to be more productive somehow. Yes.

問題是：做得好的團隊確實似乎能從 AI 獲益，DORA 報告也呈現相關性；但如你所說，我們還不知道因果方向。究竟是 AI 讓這些團隊變好，還是原本更好的團隊較能善用 AI？目前只能說使用 AI 與某種生產力提升相關。

Yeah, for whatever reason.

對，不論背後原因是什麼。

For whatever reason, now the question is, we have a team, they say they've never used these technical practices or used them not very proficiently, and they decide to use AI to be better. Now, what would you suggest to this team?

如果一支團隊從未採用這些技術實務，或運用得不熟練，現在想用 AI 來改善，你會怎麼建議？

What I would probably do now is say, look, let's take a look at the evidence and let's see where you fit in the distribution of teams that the evidence is telling us is likely to produce this kind of result. Is it likely to shrink lead times or make them longer? Is it likely to make your releases more stable or less stable? So I would suggest for example, going to the Dora website, they have this sort of quick chat check, which is like a little survey you take, which is about things like lead times and release cycles and stuff and just try and establish where you are in that distribution. If you're not very comfortably in the blue towards the high performing and elite end, you need to try and get into the blue before you connect the fire hose to your plumbing. And that should be your first priority, which is OK. What kind of teams, for whatever reason, whatever the causal mechanism is, are seeing modest boosts in productivity and quality and etcetera. How do we to position ourselves there?

我會先請他們查看證據，判斷自己位於團隊分布的哪個位置：AI 可能縮短還是拉長前置時間？讓發布更穩定還是更不穩定？例如 DORA 網站有一份關於前置時間、發布週期等項目的快速檢查問卷。先確定自己的位置；若還沒有很穩定地落在高效能或菁英團隊的藍色區域，就應先進入藍區，再把消防水帶接上管線。第一優先是讓自己具備那些能同時取得適度生產力和品質提升的團隊條件。

The bad news for those teams and for those organizations is that's if you're starting way further down, if you're poorly performing or, or in a band that I've invented and I one day hope to persuade the Dora folks to include it, which I call catastrophically bad, but you're not actually delivering anything for months and months. If you're in that kind of end of the spectrum. The good news is it's actually quite easy to get into averagely performing from there. It's usually just a small change you can make. Like, OK, if you're going to change the code, right, I test for it first, something as simple as that. But the longer journey to get into the blue, you'll see if you go to the website, there's like a little color spectrum that goes to red all the way up to, you know, a nice lovely blue. If you're the sort of purple down towards red, you probably shouldn't maybe not touch this technology but quarantine it. Say, OK, we will use ChatGPT or we use Claude Code, but in a purely advisory role, we'll use the chat window. We'll take a look at the inline completion suggestions, but maybe just don't let this thing touch the code directly.

若目前表現很差，甚至屬於我自創的「災難性糟糕」等級——幾個月都交付不了任何東西——好消息是，從那裡提升到平均水準通常不難，也許只要做到修改程式碼前先寫測試。但要走到藍色高效能區仍是一段長路。若你還在接近紅色的紫色區，或許不用完全禁用 AI，但應把它隔離：只讓 ChatGPT 或 Claude Code 擔任顧問，使用聊天視窗或查看行內補全建議，先不要讓它直接碰程式碼。

What I'm not convinced about is that all the magic and all the gold is in understanding the tools. I actually genuinely believe that most of the value is in the processes and the practices that you use those tools in. I think that's where the big money is.

我不相信所有魔法與價值都藏在對工具的理解裡。真正的大部分價值，其實在使用工具時所採取的流程和實務。

Is almost the tools are are an excuse to use good practices.

這些工具幾乎只是促使大家採用良好實務的理由。

Absolutely. I mean, if I had to, don't make me. But if I had to, I'd be quite happy getting back into doing manual refactorings. I'd rather not. I'm quite fond of my automation. I'm quite fond of Intellij and Rider and Resharper and all of those. So I'd rather not do it. But if I had to, I would get back into that and I don't think it would have a massive impact on my overall productivity. And I think it's the same with these tools. I think it's the same as learning your IDEs don't think learning your IDE sort of shortcuts and inside outs makes you massively more productive overall as a software developer in terms of value delivered. But I do think it helps. It removes friction. And I think tools like Codex and Claude code and cursor and copilot can remove some friction, but in return they can also add in a lot of friction as well A.

完全正確。雖然我不想放棄 IntelliJ、Rider、ReSharper 和自動化重構，但真有必要時，我可以回到手動重構，整體生產力也不至於大幅下降。AI 工具也是如此。熟悉 IDE 的所有快捷鍵不會讓軟體價值產出暴增，只會減少摩擦。Codex、Claude Code、Cursor 和 Copilot 的確能消除一些摩擦，但同時也可能引入很多新的摩擦。

Lot of problem now.

以及很多問題。

Another question about skills. So there is a lot of talking about the skills we should be hiring for now. You say now we have AI and so we need to change the skills we look for when we hire people say, I think that if I remember correctly, I read also things around AI native developers or things like this. So the question is, do we really need to look for different skills, additional skills, new skills?

接著談技能。現在大家常說有了 AI，招募應尋找不同能力，甚至出現「AI 原生開發者」等說法。我們真的需要不同、額外或全新的技能嗎？

Yes and no. I think I can only speak from my personal experience, what I have found useful to understand. Obviously you need to understand how Claude Code or cursor that how they work. Compared to IDEs that we're using these days like VS Code Intellij, these tools are actually relatively straightforward. They don't take a lot of time to learn and compared to all the other stuff you need to know to be an effective software developer, it's a blink of an eye. They take no time to and so I will, as it is with programming languages, it's the same argument, which is OK. If you take AC sharp developer and you're interviewing them for a Java role and they're a really good C# developer and then you interview, you know, a pretty decent Java developer, I'll take the C# developer 1st and say it'll probably take you a few weeks to get up to speed on Java. Same with any of these tools really. There's there are doing the job.

可以說是，也可以說不是。當然要了解 Claude Code、Cursor 等工具怎麼運作，但與 VS Code、IntelliJ 等 IDE 相比，它們相當直接，不需要花很多時間學。和成為有效軟體開發者所需的其他知識相比，工具學習只是眨眼一瞬。這和程式語言一樣：招募 Java 職位時，我寧可選很優秀的 C# 開發者，而不是僅僅還不錯的 Java 開發者；前者幾週就能上手 Java。工具也是如此，真正重要的是完成工作的能力。

All the other stuff you need to know, including the domain knowledge of its banking or retail or whatever it is, is a mountain compared to the molehill of the tools and the technology usually, unless you're actually literally training neural networks and then that's a whole field. But Intellij and Visual Studio, these are not fields you do don't need to go away and spend years learning. You can get relatively productive pretty quickly with them. My experience with AI coding assistants and is that they're in the same probably even simpler. What can happen I think is people can become sort of deluded into thinking that there is more to these tools than they than there really is. And that's when you get into the whole area of prompt engineering and all those kind of, you know, like the when you go to the junction, the busy junction and you're pressing the button to Cross Road, you know, the buttons not doing anything. It has no actual effect. The whole thing's on, you know, control centrally or whatever.

領域知識——銀行、零售或其他業務——以及所有其他必備能力，像一座山；工具和技術通常只是一個小土丘。除非工作本身是訓練神經網路，否則 IntelliJ、Visual Studio 和 AI 編碼助手都不是需要多年研究的領域，很快就能具備生產力。問題在於，人們容易幻想工具比實際上更深奧，於是衍生出提示工程等現象；就像在繁忙路口不停按行人按鈕，實際上號誌由中央控制，那顆按鈕根本沒有作用。

And I think there is an element of that to a lot of this, which is the illusion of control, that there is more to this, that these things are really thinking, they're really planning that this prompt is going to work better than that prompt. Do you remember at the beginning of all this? You go online and people were selling prompts as slow as selling dice throws. It's like, what is the point of this? And that's not to say though, that there aren't principles behind this and there isn't stuff that is worth understanding. So I think a grounding in how large language models work at a high level, it's very useful.

其中有一種控制幻覺：以為模型真的在思考與規劃，或某個提示一定比另一個更有效。早期甚至有人販售提示詞，簡直像在賣擲骰子的結果。不過，這不代表背後沒有值得理解的原則；從高層次掌握大型語言模型如何運作，仍然很有用。

As you said, you know, having knowledge of the systems can be quite useful. For example, I find it useful. Sometimes technologies you don't know, you want to explore them, you want to try something out. Occasionally, you know, starting to do some project. It gives you, helps you solve the blank page problem when you're getting started.

了解系統也很有幫助。例如探索不熟悉的技術、嘗試新東西或開始小專案時，它能協助突破面對空白頁面的起步障礙。

Yeah. So there are quite a lot of things that can be useful for, but we need to be careful not to overestimate the capabilities.

對，它有許多實用之處，但我們必須小心別高估能力。

I think the psychology of it is fascinating. There was a study done that showed a very strong correlation. Let me see if I can find them. Put it in my bookmarks here somewhere. Super intelligence or superstition. Exploring psychological factors influencing belief in AI. Predictions about personal behaviour. And what they discovered is. This is mostly from the MIT lab, and what they discovered is that there was a quiet striking correlation between confidence in AI output and belief in the paranormal horoscopes and that all that kind of stuff. Psychology of it is fascinating, the way that we anthropomorphize these models. I've watched people, I've been in pairing sessions with developers where they, I've watched them intervening, editing code, changing things, saying no, that's wrong, do this instead and then get to the end of the session and then have them say to me and all with a totally straight face, see, Claude did it all. It's what I call the clever hands effect. You heard of clever hands? This was a horse at the beginning of the 20th century, a horse in Germany that apparently could do arithmetic and read maps and do all kinds of stuff.

其中的心理學很迷人。有一篇《超級智慧還是迷信？探索影響人們相信 AI 預測自身行為的心理因素》的研究，主要來自 MIT 實驗室；研究發現，相信 AI 輸出與相信超自然現象、星座等之間有顯著相關。人類很容易把模型擬人化。我曾在結對開發時看著開發者不斷介入、修改程式碼、糾正模型，最後卻一本正經地說：「你看，Claude 全都自己完成了。」我稱之為 Clever Hans 效應。Clever Hans 是二十世紀初德國一匹據說會算術、讀地圖的馬。

And so he told all these fairs, and the great and the good came to see Clever hands, and his trainer would say what 6 * 9 and Clever Hands would tap it out when it's off, and so on and so forth. People were amazed, but there were some skeptics. And the skeptics, they organized some experiments and tests, and in some of the tests, the horse couldn't see his trainer. He was blinkered. And when the horse couldn't see his training, he got all the answers wrong.

訓練師問六乘九等於多少，Clever Hans 就用蹄子踏出答案，令眾人驚嘆。後來懷疑者設計實驗，讓馬看不見訓練師；一旦視線被遮住，它便答錯所有問題。

So there must have been something happening in the.

所以訓練師身上一定發生了某種訊號傳遞。

The trainer, probably totally unconsciously, was somehow signalling visually to the horse what to do. And I've seen people do that with LLMS. I've seen I've watched them step in, fix things, correct things, and at the end go see it did it all.

很可能是訓練師在毫無自覺下，用視覺訊號告訴馬該怎麼做。我也看過人們對大型語言模型做一樣的事：一路介入、修補和糾正，最後卻說全是模型自己完成的。

Do you think about programmers painting and using an AI tool? How do you see that integrated in the in the process? Should just be something they use to test ideas, conversations, or get the code done. How? How do you see that working out?

你怎麼看程式設計師結對工作時使用 AI？它應該只用於測試想法、對話討論，還是直接產生程式碼？你認為應如何整合？

Depends really on how they're doing it. I have found. I continue to find some of the frontier models useful as sort of rubber ducks if I want to bounce an idea off something. And it's kind of like the wisdom of crowds. It is a statistical approximation of, in that particular space, what a majority of physicists or software developers or bakers or guitar players might respond with. And I have found it useful for that. I'm well aware of the fact that it's holding a mirror up to me, partly because it is the wisdom of crowds, but mostly because it has been trained through reinforcement learning to do that, to say, yes, you are very clever. That's a brilliant idea. So I think we're seeing now with sort of AI psychosis how dangerous that can be. I have noticed recently the ChatGPT has started disagreeing with me a little bit. So maybe open AI starting to respond to this in the training, but it is useful, I think, to hold your idea up to a mirror and have it reflected back through the kind of wisdom of crowds with that kind of filter. So I find it useful for that, but that's not the main role of a pair in true pair programming.

取決於使用方式。我仍覺得部分前沿模型很適合當橡皮鴨，讓我把想法說給它聽。它像「群眾智慧」的統計近似，反映在特定領域中多數物理學家、開發者、烘焙師或吉他手可能如何回答。我知道它只是在照鏡子：不僅反映群眾智慧，也經強化學習訓練，傾向說「你很聰明，這是好主意」。AI psychosis 顯示這可能很危險。最近 ChatGPT 開始偶爾反駁我，也許 OpenAI 正在調整訓練。把想法放到這面帶有群眾智慧濾鏡的鏡子前仍有價值，但這不是正統結對編程中搭檔的主要角色。

If it's the navigator, I don't trust it to see the bigger picture. If it's the driver, I don't necessarily trust it to write the code without me intervening.

如果讓它當 navigator，我不信任它能看見全局；若讓它當 driver，我也不信任它能在我不介入時正確寫出程式碼。

I'm asking because I've used that in some pairing and teaming sessions so far. We always use the agent as a tool, so it's like discussing with the other person on a problem how we'd approach that. And then we had the doubt to say, let's check with AI what we get.

我之所以問，是因為在一些結對與團隊編程中，我們一直把代理當工具。先和另一個人討論如何處理問題，有疑問時再問 AI 看看會得到什麼。

What did a billion people say on Stack Overflow?

就像問：Stack Overflow 上的十億人會怎麼說？

And then have a look at the response and see if it is something useful for us or not. Maybe change the prompts. So we don't really treat the AI tool as a, let's say a member of the team is just another tool that we try to use it to help us thinking things through, especially in parts of the code we don't know well. Or maybe there are some technologies that we need to use that are not very well known by us and so we try to get help. I guess that sometimes it's like Googling for something or looking at Stack Overflow but using an AI system. Basically using the system for exactly the same purpose.

我們看回覆是否有用，必要時改提示，但不把 AI 當團隊成員，只把它視為協助思考的工具。尤其遇到不熟悉的程式碼或技術時，可以向它求助；本質上就像 Google 搜尋或查 Stack Overflow，只是改用 AI 系統達成同一目的。

I think it is conveniently useful in that sense. Whether it's worth the massive cost is a separate, but from a purely practical point of view, I think it is useful in that respect to have in a teaming kind of situation, for example, to be able to go. It's like if you're familiar with Who Wants to be a Millionaire? You've got that lifeline. Ask the audience, it's ask the audience. It's, we asked a billion software developers and sometimes it comes up with a good answer because it's got a lot of data in its training and sometimes the data is very flat and very sparse. And you get it's like if you were to ask the audience what's to the capital city of France, Paris would stand out as a probability, yeah, for the next token. But if you were to ask it what microscopic mechanism gives rise to superconductivity in semiconductors, you would expect the audience to just guess and you get a very flat response, a very low confidence response now. And in that respects, they're quite useless because if, if you're talking to about your field stuff, you know a lot about the stuff that you don't understand is more likely to be outside of its training data distribution as well.

這樣使用確實方便；是否值得付出龐大成本是另一回事。以團隊協作而言，它像《百萬富翁》的「問現場觀眾」求救方式：等於問十億名開發者。有時訓練資料充足，答案很好；有時資料稀疏且平坦，信心便很低。問法國首都，Paris 的機率會突出；但問半導體超導性的微觀機制，觀眾只能猜。最麻煩的是，你需要幫助的往往正是自己不懂、也更可能超出模型訓練分布的領域。

And that's where I've kind of found whether it's with software development or with physics or whatever it is, you very easily lead it out of its own data distribution. It's, it's very good at stuff I don't understand. But then everyone says that, oh, it's really good. You can't do my job, obviously, but it could totally do yours. That I don't understand.

無論軟體、物理或其他領域，都很容易把模型帶出它的資料分布。它在我不懂的事情上看起來特別厲害；每個人都會說「它當然不能取代我的工作，但完全可以取代你的」，因為他不了解別人的工作。

And that has I'm pretty sure been accelerating in the last three or so years, especially the last year that things that I would expect code, I would expect a developer with, you know, 234 years of experience to understand. I'm now finding that they can't understand it. Yes, we definitely can't say definitely because it's a personal experience, but I'm definitely experiencing groups of people who are struggling more and more, not just to understand maybe code or whatever I put in front of the, but to understand requirements for an exercise that maybe 10 years ago there was never a problem. And now I'm having to explain, you know, to stop them and go, no, I actually want to what it means is this, and I haven't changed the text of the problem. It's the same exercise. Nothing has changed. But now I'm having to intervene more often and go, OK, let's let's break this down more. And also programmers who have been working in, you know, Python or C# or jobs or whatever for several years who are unable to do certain very basic things from scratch in the language, like Java. Developers who don't know how, what a constructor is or how to declare a constructor because they've never done it.

我相當確信，這種下降在近三年、尤其近一年加速了。以前預期有兩、三、四年經驗的開發者能理解的程式碼，如今常常看不懂。同樣一份十年前大家都能理解的練習需求，文字完全沒改，現在卻更常需要我停下來重新拆解解釋。甚至有使用 Python、C# 或 Java 數年的程式設計師，無法從零完成語言中的基本工作，例如 Java 開發者不知道建構子的概念或宣告方式，因為他們從沒親手做過。

But this I think started really with Stack Overflow and AI has accelerated it that more and more people have been copying and pasting it, which is why I try to urge, although I appreciate it is harder work, particularly when you're starting out in programming and software development. Please don't by all means read code and copy it, but it's got to go in the eyes, through the brain and out of the fingers, I think to engage those centers of your brain that will actually that makes it sink in. It's like, it's like everyone's on a speed dial and no one can remember anybody's number. And the moment you take that that phone away, they can't remember anybody's number. And I'm seeing this more and more now, and it is from personal experience, it does really appear to be accelerating. But there's also research that is suggesting that it really is accelerating. And I really worry about that because that's going to hit another problem, which is that regardless of their ability to comprehend any code, is the fact that more and more code is being produced that nobody's even looked at.

這個趨勢從 Stack Overflow 時代便開始，AI 又加速了複製貼上。雖然初學程式設計時特別辛苦，我仍會勸大家：當然可以閱讀和參考別人的程式碼，但它最好要「從眼睛進去，經過大腦，再從手指出來」，讓相關腦區真正參與，知識才會沉澱。這就像所有人都用快速撥號，拿走手機便記不得任何電話號碼。個人經驗和研究都顯示情況正在加速，而更令人擔心的是，未來會有越來越多程式碼根本沒有人看過。

So they understand code slower, and there's going to be lots more code that needs to be understood, but it inevitably breaks and the language model can't fix it.

於是人理解程式碼的速度變慢，同時需要理解的程式碼卻大量增加；當它不可避免地出錯，而語言模型又修不好時，問題就會爆發。

Now another question may be related to this, you know to the cognitive ability and the skills. Now the most recent development for using AI is the spec driven development trend where people are supposed to write specifications in natural language and the the system will will create something based on that. Now, if this works out, maybe we don't need to know coding that much after all. What do you think?

這也關係到認知與技能。近期流行規格驅動開發：人用自然語言寫規格，系統依規格產生成果。如果這條路可行，我們是否就不必那麼會寫程式？

So there are different interpretations of what people mean by spec driven development. And I think I think probably the mainstream one that I see a lot, I say mainstream. It's mainstream on LinkedIn, which means it's probably not mainstream. But the one I see most often on LinkedIn is along those lines. You write natural language specifications and then you feed it to Claude Code or cursor or whatever or lovable and off it goes and chugs away at it and produces you a working system. It kind of sort of works, provided you you're not particularly attached to any particular end result it will produce. This is the problem with natural languages. They are inherently ambiguous. That's an unfixable problem. So it will produce an interpretation. As long as you're happy with the interpretation it produces, then that's OK. Under the hood, the code is likely to generate. Is going to be problematic in terms of. So large language models are really good at generating code that they're really bad at modifying later.

規格驅動開發有不同定義。LinkedIn 上最常見的版本，是寫自然語言規格，交給 Claude Code、Cursor 或 Lovable，讓它自行產生一套可運作系統。這種方法勉強可行，前提是你不執著於特定結果，因為自然語言天生具有歧義，無法徹底消除。模型會產生其中一種解讀；只要你接受那個解讀就行。不過底層程式碼仍可能有問題：大型語言模型很會產生新程式碼，卻很不擅長日後修改。

That is actually an interesting take because actually since that also again some research around that, it seems that the better the quality of the code, the better the agents are able to help with the modifying and refactoring that later.

這點很有意思，因為有研究顯示，程式碼品質越好，代理日後協助修改與重構的效果也越好。

I see people saying, for example, they claim, oh, code doesn't need to be readable by humans anymore because of AI, But then you get these. I look at my bookmarks here. I've got another great study here called How does naming affect language models on code analysis tasks? So this is a study that's relatively recent and I've done this experiment. It was this study is only a couple of years old, maybe a year and a half old. I've done this experiment myself where you take, you take code that is pretty easy to understand or is intuitive to me, you know, fairly easy to comprehend. And then you take it through a series of iterations where you obfuscate the code more and more and measure just how you know obfuscated it is. So you're changing function names and class names and everything until they make no sense whatsoever. And then you give the model tasks to conduct and then you measure, measure completion. How many of those tasks are complete? And it does indeed go off a Cliff. The less intuitive the code is to a human being, the more I think the clue really is language model. That should be our first clue. They are language models.

有人聲稱 AI 時代的程式碼不再需要讓人看懂，但研究《命名如何影響語言模型的程式碼分析任務》正好相反。我也做過類似實驗：拿容易理解的程式碼，逐輪混淆函式、類別等名稱，直到毫無意義，再讓模型執行任務並測量完成率。結果完成率確實急遽下降。程式碼越不符合人類直覺，模型越難處理；「語言模型」這個名字本身就已經給了提示。

They were originally created, invented to translate human languages. And when you start using language in the code that doesn't match the language of the problem, the language you're trying to quote, all kinds of things go wrong. So obviously that affects model performance. Complexity has a massive impact. Coupling and cohesion in modular design has a huge impact as we we discussed in terms of the potential blast radius of changes, how much code needs to be brought into the context, context that has to be need to be changed. Duplication has a similar effect. So all the things that we've been saying for years make code harder to change or harder to understand and harder to change for humans, including fast running regression tests, are exactly the same factors that research and environment and experience is telling us make code harder to change for large language models.

語言模型原本就是為翻譯人類語言而發明。若程式碼中的語言不符合問題領域的語言，各種事情都會出錯。複雜度也有巨大影響；模組設計中的耦合與內聚會決定變更爆炸半徑，以及必須放進上下文、必須修改的程式碼量。重複也有類似效果。多年來我們知道會讓人難以理解和修改程式碼的因素，包括回歸測試速度，如今研究與經驗都告訴我們，它們同樣會讓大型語言模型難以修改程式碼。

Do you think maybe we can come up with some different AI technology potentially?

那麼未來有可能出現不同的 AI 技術嗎？

That's not my feel, but I look forward to the day. I don't believe so. I've I'm, you know, 55 and I have been programming for a large part of that. And I have heard multiple times that either artificial general intelligence is just around the corner. It's always 30 years away and has been since I was a kid or certainly during those 40 plus years that I've been programming. Likely 45 years that we're not going to need programmers soon. I saw adverts for it in the 80s, I saw people claiming it in the 90s with computer aided software engineering. I've seen it with no code and low code, and we're seeing it again now. I think maybe coding, maybe me typing the code will one day no longer be a thing. But I think that wider activity of specifying, designing, delivering software in feedback loops, learning with customers and other stakeholders, I think that will always be the bulk of the job. I think we'll always ultimately about be about that bigger picture. I think I'm kind of insulted by people who go. I can see it replacing all software engineers soon.

我不覺得會，但期待有一天被證明錯誤。我今年五十五歲，人生大部分時間都在寫程式。這四十多年來，我多次聽說通用人工智慧即將到來——它從我小時候起就永遠距離三十年——也反覆聽說很快不再需要程式設計師。八〇年代有這類廣告，九〇年代有 CASE，之後是 no-code、low-code，如今又再來一次。也許有一天我不必親手輸入程式碼，但定義、設計、交付軟體，在回饋迴圈中與客戶及利害關係人共同學習，始終會是工作的主體。有人說 AI 很快會取代所有軟體工程師，我甚至覺得受辱。

Because what that implies is that what we do, what I do, what you do, and what 50 million other people do for a living in the world does not require general intelligence that it can be, it can be replaced by, you know, super autocomplete. And I find that quite insulting. And I just don't think it's, I'm pretty confident it's not true. Waiting for the the rest of the market to catch up with that reality is another thing entirely. But I'm really, I'm reasonably satisfied that it's not true. It's probably not going to happen in my lifetime. I think the future of software development is software developers.

那種說法暗示我、你和全球五千萬名從業者的工作不需要一般智慧，只靠「超級自動完成」便能取代。我認為這不但侮辱人，也相當確定不是事實。市場何時才會接受現實是另一回事，但這大概不會在我的有生之年發生。軟體開發的未來，仍然是軟體開發者。

What skills do you think there will be important for professional developers in the future?

你認為未來的專業開發者最需要哪些技能？

OK, so skill number one right at the top of the tree is communication and comprehension as well. The second most important skill is the ability to change software without breaking it.

第一名是溝通與理解；第二重要的是在不破壞軟體的前提下修改它。

So can I say that is pretty much the same skills that are necessary today if to write good software?

也就是說，和今天要寫出好軟體所需的技能幾乎相同？

Yeah, absolutely, Absolutely. I haven't had to memorize libraries and APIs. It's been so many years since I've had to memorize things. And so yes, I think language models could take some of that load off or they could, if they were more reliable, can take some of that load off us in exactly the same way that, you know, Intellisense and auto completion or code sense and all these other things have taken load off us, removed friction from the process of creating source code that does what we need. But the wider job has always been the bigger job and the most important part. But because the tools are not 100% reliable, you're always going to need someone who can understand and can edit and can fix and can debug. And I think at the core of it, I don't think those school skills are going to become less important. I think that's, it's that kernel that call that not that important. But it does have to happen and it does have to work. People do need to understand it. I don't think that's going away. And I see it going away. I see it sort of wandering off into the distance, into the fog, as people rely on this technology more and more.

完全相同。我早已不必背誦函式庫和 API；語言模型若更可靠，確實能像 IntelliSense、自動完成等工具一樣減輕記憶負擔，消除產生符合需求之原始碼時的摩擦。但更廣泛的工作始終更龐大、更重要。由於工具不可能百分之百可靠，永遠需要能理解、編輯、修復和除錯的人。這些核心技能不會變得不重要；然而，隨著人們越來越依賴這項技術，我反而看見它們正逐漸走進霧中。

OK. I think we're quite an interesting conversation, Jason. Thank you very much.

Jason，這是一場很有意思的對談，十分感謝。

Well, thank you.

也謝謝你。

Thank you, Jason, for being with us. This is Giovanni Asproni for Software Engineering Radio. Goodbye.

謝謝 Jason 來到節目。我是 Software Engineering Radio 的 Giovanni Asproni，再見。