You need reliable AI context for your site reliability

Hello. Hey, Ryan. Thanks for having me.

Ryan 你好，謝謝邀請。

So I wanted actually to be an accountant. That's what I started off with. Yeah, I like numbers, I like money. I thought that was a good combination. And this is where I wanted to be. But then like I was looking for a job and like a QA opportunity came in my way and I started doing that and I just fell in love with software and just felt right and felt the, the natural thing for me. So I went to school and studied like computer science and it made me even more fall in love with, with the topic. And I just during my, my studies, I started working as a software engineer part time in a few small startups. And then I also understood the startups are also the right fit for me. So that was my beginnings.

我原本想當會計師，因為喜歡數字也喜歡錢，覺得很適合。後來找工作時遇到 QA 的機會，一做就愛上軟體，感覺非常自然。我去學校讀 computer science，求學期間也在幾家小型 startup 兼職做 software engineer，並發現 startup 同樣很適合我。這就是我的起點。

So let's start with what's necessary actually is right Today in many organizations, SRE is a very pivotal position. SRE stands for site reliability engineer personally the guys who are responsible for keeping like the production environment stable, healthy and to make everyone happy, right? Both the the company providing the service and the users who are actually consuming the service. So SREs have that job and it's a lot of responsibility to do the job. And their part includes so much context to it. Even much before the AI era. It includes so much context to it to understand the business side of things, where those things fit. What's the relationship between services 1 to another. And over the years that we added a lot of capabilities of micro services and a lot of networking layer and cloud infrastructure. So we've just kept adding a lot and a lot on the shoulders of the SREs. We just have to make sure that production is actually reliable. So we put a lot of effort and a lot of responsibilities over the shoulder and just keeping adding those. It's just a bit problematic. So AI is a really good like to helping the SREs and making them be much better.

在許多組織裡，SRE 是關鍵職位。Site reliability engineers 要維持 production environment 穩定健康，讓服務供應方和使用者都滿意，責任非常重大。即使在 AI 以前，工作就需要大量 context：理解商業面、各項功能的位置、服務間關係。多年來又加入 microservices、networking layers 和 cloud infrastructure，不斷增加 SRE 肩上的負擔。他們仍必須確保 production 可靠，持續加責任並不可行；AI 可以幫助 SRE 做得更好。

So we started off in a very like naive way of just, you know, throwing a lot of stuff they are and start to understand like what the agent is capable and is not capable of doing and really fast. We got to a point where it was just too problematic. And the data we had to inject into the AI agent, we had to think about it really hard and really long on like what's the proper way of injecting the data? What kind of information is more valuable than others? So we had to do a lot of stuff. So we focused on 2 main areas. One is actually the data and how to collect and to inject it. And the second is the evaluation layer that no matter what change do we think is important or seems the right way to go. We have an evaluation layer which each change that we are conducting, we want to make sure that the change is in fact putting us in the direction that we want to head. So these two must live together. 1 without the other is just not enough. And it's not enough for every AI agent, especially AI agents to that claim to be the ones that are actually resolving production issues and putting us in like the most sensitive place in the organization.

一開始我們很天真，把大量資料全丟進去，藉此了解 agent 能做與不能做什麼，很快就發現問題太多。我們必須仔細思考資料該如何注入、哪些資訊價值更高。因此聚焦兩件事：第一是資料的收集與注入；第二是 evaluation layer。每次修改都要透過評估確認，變更真的把系統帶向預期方向。兩者必須共存，少一個都不夠，尤其是聲稱能解決 production issues、被放進組織最敏感位置的 AI agent。

So we have to balance between the two and work on 2 on a parallel way.

所以必須平衡資料與評估，並行改善兩者。

There's a lot of context to it to understand, like what are we aiming to do to resolve issues in a production scale? Large organizations, if an SMB's, their scale is very large, could contain a lot of services and one incident can go through maybe dozens of different services to understand how it goes. And when an SRE actually by themselves are investigating the issues, they are looking at so many logs, so many metrics, traces, logs, timelines of stuff, the gate, what has changed, what the new version includes, so many different stuff. Even before, they are so too much context to handle. And a lot of the time there's so much context in a realm that they are unaware of maybe the code themselves, the business plan that connected to it. So they had to involve other people in the organization as well, the product manager, the software engineer, the devils that was responsible at the same day. So getting all this context was very problematic. So we saw one, collecting all this information is a problem. Second is to understand what's critical and what's not, so how to do it and what's not. And we found that the best method is giving the the liberty to do a lot of stuff.

在大型組織，甚至規模較大的 SMB，一次 incident 可能橫跨數十個服務。SRE 調查時要查看大量 logs、metrics、traces、timelines、Git 變更和新版內容，context 本來就多到難以處理；其中還包括他不熟悉的程式碼或相關商業計畫，必須請 product manager、software engineer、當班 DevOps 等人協助。第一個難題是收集全部資訊，第二是判斷哪些關鍵、哪些不重要。我們發現最好的方法是讓 agent 在許多事情上有自由。

But in certain areas, we don't give it the liberty. We say these logs are important. I'm not asking you for two reasons. One is I don't want to have like the logs that happened during this time of the incident. I feel that they are crucial to understanding what the issue really was. That's one. So I don't want that maybe in 1% of the cases it won't be using it. That's first and second, also to save time and tokens, because if I'm giving the AI the liberty to do like the calls that it wishes, so it will do the first round of LLM iteration. Understand that it needs to go through the logs, then call the logs, get it back while injecting it in the first place before the LLM starts. Saves a lot of both time and tokens.

但有些領域不能完全自由，例如我們會明確指定 incident 發生期間的 logs 必須使用，因為它們是理解問題的關鍵，我不想承擔那 1% 被忽略的情況。預先注入也能省時間與 token；若完全讓 AI 自己決定，它得先跑一輪 LLM 才發現需要 logs，再呼叫工具取回。若一開始就放進去，兩者都省。

So actually it took us quite some time to get to the separate agents because first of all, the foundation models are are really good, right? We at Komodor are not building our own models. We're using off the shelf models, especially Claude and they're doing a very good job. The context window is getting bigger and bigger. But even if the models are capable of doing a lot of stuff, we really want it to be efficient and fast and we don't want to rely on it. We don't want to rely that the model is good enough with handling of hundreds of thousands of tokens, even if it does have this capability, because it will succeed when providing them a few, let's say 10 or 20 thousands of tokens, it will be right 99.9% of the time and when we provided the 200,000, it will be right 98% of the time. 98 is just not good enough. You can say I'm wrong in 49 out of 50 production incidents and it will be right in 50 out of 50. So we must aspire to that 100%. That's why it's very important for us, even when it's under the context window, we still have like a lot of like improvements and optimizations that we are doing.

我們花了一段時間才走到多 agent。現成 foundation models 已經很好，Komodor 不訓練自己的模型，而是使用 Claude 等模型，context window 也越來越大。但能力足夠不代表效率與可靠性足夠。我們不想依賴模型能正確處理數十萬 token：一兩萬 token 可能有 99.9% 準確度，二十萬 token 也許掉到 98%；在 production incidents 裡，98% 遠遠不夠。你不能說五十次有四十九次答錯、一次答對也算可以，我們必須追求接近 100%，所以即使沒超出 context window，仍要持續最佳化。

Yeah, yeah. So it depends on the task. If it's a task that we feel that doesn't require a lot of heavy lifting or something like that, we try it out with different models. We do it a lot like trying out different models I talked earlier about like our evaluation layer. So every change that we are making getting into some kind of testing period in which it's being test in production, real life incidents that our users are currently working with our production agents and sub agents. But under the hood we are also running the staging or the non production agents. And this way we can compare the results. We have like a judges that are very good at comparing which one was better and after a few thousands or 10s of thousands depending how much confident we want to gain, which is like OK, the actually the testing sub agent was actually behaving better or behave this with the same accuracy, but we spend much less tokens on it. So we have a lot of things to compare with. Accuracy is our only great of course. And then we select which one was better to us. And like, this is how we keep improving.

視任務而定。若不需要重度推理，我們會試不同模型，並利用 evaluation layer 比較。每個變更都要進測試期：使用者照常跑 production agent，底層同時讓 staging／non-production agent 處理真實 incident，再由 judges 比較結果。累積數千到數萬次後，才能判斷測試版是否更準，或準確度相同但 token 更少。Accuracy 當然是主要標準，再選出更合適的版本，持續改善。

I don't have like a magic hint here. Something that would work, I think with working with LLM, this is not the only the most important part of your work. I think it's having good judges. Is the base for everything in the pyramid of velocity like the judge or the baseline? If you have a judge that you trust on that, you say yes. I would pick the same thing. I would judge it the same way in 99% of the cases this way, like it improves velocity a lot. So we started off actually with manual evaluations. We built our own judges and then in a separate manner we started like going over them manually, which did a really good job. We felt like the judges are very close but not enough. So we optimize them, optimize them, optimize them and then we just re evaluated. And the easier trick that judges can have is that the input is static. The input, we say this was the issue, this was the way of investigation. Was it good? Did it find the correct root cause? Yes or no? So their input is static, unlike production incidents where the input is dynamic.

沒有魔法提示。建立可信任的 judges 是 LLM 工作最重要的基礎，也是速度金字塔的底層。若 judge 在 99% 情況都會做出和你相同的判斷，開發速度就會大幅提高。我們先人工評估，自行建立 judges，再逐一人工核對，反覆最佳化與重新評估。Judge 的優勢是 input 固定：問題是什麼、調查如何進行、是否找對 root cause，都能保存後重複比較；production incident 的 input 則會隨 logs、metrics 和 services 持續改變。

So we want to make sure that we are good not only for today but also for tomorrow.

我們不只要今天表現好，也要確保明天仍然可靠。

I can tell you what we found out up until now. First, we do see a lot of appetite to it like we do see that a lot of our customers are actively asking about the we do have this capability of doing it like autonomous way. But having said of that, we don't see a lot of like like production environments that are fully automated with like AI doing the investigation, offering the remediation and actually performing the remediation offered all in this together. What we do provide here at Komodor is we do all of that, but with human in the loop, but with someone that Claudia, our AI agent, that's her name, she's doing the investigation later on she's doing the remediation plan. The remediation plan ends with this is the command needs to be executed. These are the reasons why I think this is the command needs to be executed with all related evidences, etcetera. And then the users can get in the way and say like whether they approve or reject the suggested fix. And if they approve, Claudia does that ourselves. We do see some time that users are not proving or rejecting, but copy pasting it and doing it themselves. We also consider that as the win.

客戶對 autonomous remediation 很有興趣，但真正完全自動化的 production environment 還不多：讓 AI 完成調查、提出 remediation 並直接執行，大家仍很謹慎。Komodor 的 Claudia agent 可以完成整個流程，但保留 human in the loop。她先調查，再提出 remediation plan，列出要執行的 command、理由和證據，使用者可以核准或拒絕；核准後才由 Claudia 執行。有些人不按核准，而是複製 command 自己操作，我們也視為成功，因為他相信修復方向正確。

Yeah, if they thought that our remediation was the correct 1. So we consider it as a win. So we do see the world where more appetite to it. We do feel there's like foot in the door that is there. We do see like a lot of testing playgrounds, etc that are using like the autonomous remediation just to gain more trust in it. We do see like in certain areas where we are being asked and we do have the capability of allowing autonomous remediation, but only for let's say a rollback only. Let's say for certain actions that I'm confident that I do not like create a new version or something like that or apply the code fix, let me do it myself. And stuff that are more like soft. They have like less issues to adopt. And we do see like the trend and the foot in the door getting there. But as I, as I started off, we do ask ourselves the same question of how can we gain the trust more quickly? What do we need to do? And we feel like the road is clear.

市場對自動化的接受度正在增加。許多人先在 testing playgrounds 使用 autonomous remediation 建立信任，也有人只允許特定低風險動作自動執行，例如 rollback；涉及新版本或 code fix 則保留人工處理。這些較柔性的操作更容易採用，算是先踏進門內。我們一直問如何更快取得信任，而前進路線已逐漸清楚。

Yeah, definitely. Both from the SRE perspective, both of the system perspective, right? Time is money, and if I can be down not for 15 minutes but for 30 seconds, that means a lot.

當然，無論對 SRE 或系統都是如此。Time is money；服務中斷 30 秒，而不是 15 分鐘，差別非常大。

I do think that the next line of stuff that we started off with detection, we moved off to remediation and saw it to investigation. Then we, we are now working like on remediation and gaining trust and what we talked about. And I don't think like the next cycle of things would be much more preventive. And in the world of preventive, you need to come up with like what's the main issues that are currently taking place, what's the most common stuff, what's the problematic areas in my infrastructure, etcetera. So we do have some agents that are capable of doing a lot of things. And Claudia has a sub agent that is in the preventive side of things that you can talk to Claudia, she will then activate a preventive agent. It will go through your infrastructure and imply also we have the possibility of investigating and already resolved the issue. And why is that? Because we see a lot of people coming back to resolve the issue to look what it was and they talk with AI, how did it happen? How can I improve in the future? So Claudia is very good in in helping preventing stuff from the long term.

我們的發展先從 detection 到 investigation，現在處理 remediation 與信任，下一循環會更偏 preventive。預防工作要找出常見問題和 infrastructure 的薄弱區。Claudia 已有 preventive sub-agent，可檢查 infrastructure，也能回頭調查已解決的 incident。許多人修完後仍會問 AI：事情怎麼發生、未來如何改善。Claudia 很擅長協助長期預防。

So once we feel that there is more like specific pinpointed speciality that needs to be applied, invoke our sub agents to it and they're like they're doing a pretty decent job.

只要確認某個領域需要高度專門能力，就叫用相應 sub-agent；目前效果相當不錯。

Yeah. So this is something we do have in like 2 separate ways. 1 users are capable of like uploading and connecting us to their like knowledge databases or some kind we can connect to it, read their past postmortems and improve our investigations going forward. Second, we have like a memory layer which like investigations that are taking place that we can learn from future investigations. So we definitely do this. And also when you like trigger Claudia, once an issue has been resolved, you can talk to her, ask her anything she would like. She has all the context. Of course, she's familiar with the issue that just got resolved or got resolved yesterday and she understand why it happened and how did it got there, etcetera, etcetera. It brings us to the point where she's very good at like not only doing the investigation, but also like closing the loop and providing a suggested postmortem that you can like work on and refine on it and make it better. We see the world that people are like having products such as Claudia off the shelf, like AI SRE, but we see a lot of people starting like building their own orchestrator agent, like using Claudia not as the interface but as the brain.

有兩種方式。第一，使用者可連接自己的 knowledge databases，讓我們閱讀過去 postmortems，改善之後的調查。第二，我們有 memory layer，從歷次 investigations 中學習。Incident 解決後，使用者仍可詢問 Claudia；她保留完整 context，了解問題如何發生，能協助閉合迴圈、產生 postmortem 草稿，再由人調整。我們也看到企業建立自己的 orchestrator agent，把 Claudia 當作 brain 而非 interface，讓它和其他 agents 協作。

So they connect, they build their own like orchestrating agent which is capable of calling other agents such as ours. And I think this is a very interesting world and I think this is where the world is gonna be. We say like 6 months ago, 100% of the people would just go to our UI and this is what 100% of the usage of Claudia was through our UI. And like in the past three months, four months, something like that, I can say it's changes because it's still like 80% of the usage. But we do see like this higher usage of being from our MCP, from our API, from Slack, like people not even going into using their own interfaces, agents, whatever. In some cases, they don't even know that Claudia was being triggered behind the scenes because it happened in like somewhere agentic flow somewhere.

六個月前，Claudia 幾乎 100% 從我們的 UI 使用；近三四個月已降到約 80%。越來越多呼叫來自 MCP、API、Slack 或客戶自己的介面與 agents。有時使用者甚至不知道 Claudia 在後台被觸發，因為她只是更大 agentic flow 的一環。這很可能就是未來。

I think like the era of open source I think is coming back in. Some people say I can use an open source for it or I'll build something myself. I don't need this like off the shelf something. And we do see like it's getting really good result and a really good demo. It creates a really good demo when you're tackling production incident with a lot of noise and like real life happens. You want to get the professional tool and not the tool that I just built yesterday in my garage.

Open source 的時代似乎又回來了。有人會說，拿開源方案或自己做就好，不需要 off-the-shelf 產品。這確實很容易做出漂亮 demo；但面對充滿 noise 的真實 production incident，還是會想要專業工具，而不是昨天才在車庫做出來的東西。

My name is Asaf Savich. I'm AI engineering group manager at Komodor. You can learn more about Komodor and it's AI agent Claudia by going to Komodor dot IO. And if you want to link me, feel free to add me in LinkedIn by searching me Asaf Savich ASAFSAVICH.

我是 Asaf Savich，Komodor 的 AI Engineering Group Manager。想了解 Komodor 與 AI agent Claudia，可前往 Komodor.io；也歡迎在 LinkedIn 搜尋 Asaf Savich 與我聯絡。