An anonymous poll, a debate about multimodal testing, and some uncomfortable questions about who benefits from the AI rush.
Recently, I published ...
✕
Wait! Don't Go Yet 🚀
Get our FREE eBook "10 Programming Tips That Changed Everything" when you subscribe!
No spam. Unsubscribe anytime.
This really resonates with what I’ve been experiencing too. 👀
AI has absolutely increased my productivity, and I think its basic coding ability is already very strong.
But higher output also creates new kinds of work.
I now spend a significant amount of time reviewing AI-generated changes, and then building workflows to automate parts of that review process itself. 😹
There’s also more time spent on requirements and design, because once implementation becomes faster, I can suddenly handle many more tasks at the same time.
And that creates another problem: context switching.
When we’re juggling more tasks at once, it becomes harder to focus deeply on each one, and I think that can quietly reduce the overall quality of the work.
So for me, the challenge isn’t really
It obviously can.
The harder question is
If we blindly generate code, tests, documents, and features without understanding why we need them, we’re not necessarily becoming more productive. We may just be creating a new kind of debt: a growing pile of outputs that someone still has to review, understand, maintain, and eventually clean up. 😹
At the same time, I’m still very optimistic about AI.
I think the important thing is learning how to work with it well, rather than simply trying to generate as much as possible.
AI can help us build better things much faster, but we still need humans to decide what is worth building, what is actually correct, and what should not be built at all.
For me, that balance is probably the most interesting challenge of the AI era. 😸
Haha this is so spot‑on and mirrors exactly what I wrote in the article 😂
From my QA perspective, my automated test agent happily runs through the night and spits out dozens and dozens of defect reports like there’s no tomorrow. It makes it look as if productivity has exploded, but the trick is that it hasn’t erased the work. It just shifts it right into my lap.
I end up spending a lot of my time sorting out what’s real, what’s just stale PRD noise, and what needs to be tossed out entirely. Even when I build extra workflows to auto filter obvious junk, there’s still that pile waiting for me every morning. And yeah, with more items to review and validate, I keep jumping between threads, and it gets harder to hold onto that deep focus we all love.
AI can absolutely crank out code, test cases and reports faster than we ever could. It just does not know what is worth caring about. If we let it run wild, we are not really building faster. We sign ourselves up for a never ending cleanup shift, and pile up a shiny new mountain of AI generated debt.
The fun part is this is not about holding AI back. It is about learning to be the captain instead of the full time janitor. The machine does the exploring, and we decide what actually matters💪
Haha yes, exactly! 😹
I really like your “captain instead of the full-time janitor” analogy. 🏴☠️
That’s probably the balance we need to find.
Let AI explore, generate, test, and surface possibilities at a speed we never could before, but keep humans responsible for deciding what actually matters.
Otherwise, “more productivity” can quietly become “more things waiting for someone to clean up.” 😹
And the context-switching part is especially real. Even if each individual task gets faster, handling many more of them at once doesn’t necessarily make the whole system better. 😿
AI is incredibly powerful, but I think the real skill now is learning how to control the flow of that output instead of drowning in it. 😸💪
Haha I’m happy that analogy landed well 🏴☠️ It really is that balance. We enjoy the speed AI brings, but we have to keep from getting drowned in all its output. Controlling the flow instead of just cleaning it up is the real trick 😂
Duplicates and invalid findings are the same bug seen from two ends: the finding carries no evidence of its own, so "again" and "new" both need a human to reread it. Give every item three fields before it is allowed into the list - the failing assertion, the environment fingerprint (commit + fixture), and the exact input that triggered it. A second report with the same assertion on the same fingerprint is a +1 on the existing row, not a new row; in my own queue that path prints accepted=false (already queued) and costs one log line instead of a triage slot. Two different assertions on one fingerprint is the case worth escalating, because that is not a duplicate, that is a fix that did not take, and it should reopen the item instead of closing twice.
That's a solid approach to making findings evidence-driven. But I think there's another layer upstream that deserves attention: Is the test oracle itself still valid?
If the PRD isn't updated when requirements change, an AI agent may keep executing perfectly valid test cases against outdated expectations.
For example, a field that used to be mandatory becomes optional. The implementation is updated, but the PRD and test cases aren't. The agent then reports every instance of an empty field as a defect.
The finding may have a valid assertion, environment fingerprint, and exact input. It may even be perfectly reproducible. But it's still not a real bug.
So I'd add another question before triage: Are we validating the software against the current requirements, or just validating it against whatever documentation happens to exist?
Evidence quality matters, but requirement freshness matters just as much.
Stamp every test with the PRD revision it was written against and gate on it: if
git diff --quiet "$oracle_rev"..HEAD -- prd/reports a diff, emitstale_oracleand route it to the requirement owner instead of filing a defect. One comparison turns your example into a non-event at zero triage cost — the field that became optional never reaches the queue, the spec diff does.That's actually where the problem gets harder in my experience.The PRD is already outdated, and nobody updates it. In that case,
git diffreports no change, so the gate treats an old oracle as if it were still valid.It can detect changes to the document, but it can't tell us whether the document still reflects the actual business requirements.For example, a field was originally mandatory but later became optional. The development team changed the implementation, but nobody updated the PRD or the test cases. The AI keeps reporting empty fields as defects, and technically, all its evidence is correct.
I think we need another layer here: someone has to own the validity of the requirements, not just their version history.Otherwise, we're simply automating the execution of outdated assumptions.
And that's a much harder problem to solve with code alone.
PRD-vs-PRD diff goes blind exactly when the doc gets abandoned, so take the change signal from the code that did record it:
git log "$oracle_rev"..HEAD -G'(required|optional)' -- src/validators/names the commit where that field's rule flipped, and a failing assertion whose field has a validator commit but no PRD commit is spec_lag — the hash and the author are the requirement owner you're asking for. Validity itself still can't be computed, so make it a signed artifact instead of a code path: one line in oracle_overrides.yaml (field, optional_since, owner, date) is what closes the item, and until that line exists the finding parks in needs_owner and never reaches triage. Nobody has to rewrite the PRD, but somebody has to sign one line, and the depth of needs_owner is the only honest spec-debt number in the pipeline.Love this idea. It uses code history and a simple override file instead of chasing perfect PRD updates, and the pending‑owner queue becomes a true metric for spec debt. But here’s the catch: much of the code is now written and committed by AI, so git history may not reliably show who actually made the requirement decision.
Great post, mate! Laughed while sitting here, unemployed as a fresher who didn't get a PPO because of manager politics😭🙌🏻. Keep posting!
Hahaha thanks so much for your support! There’s an old saying in China: when the gods fight, it’s the mortals who get hurt.🤣 Hope you land that great role soon.
China and it's sayings😭 watched so many cdramas lol, this is one common thing in them all😂
Hahaha we have tons of old sayings like this in China. Each one is like a neat little compressed file, so fun 😂
Yeah and hella dramatic flashbacks😭 especially the noble family dramas😂
Hahaha wow, you’re watching the same short dramas my parents love 😂 Though I’d suggest not binging them too often. Too many short videos can wear down your patience. It gets harder to sit through a full movie or a long article. I barely install any short drama apps on my phone. I prefer novels, especially time‑travel stories. They’re so fun.🤣
"From that perspective, AI isn't necessarily the original cause of the employment pressure. It may be the trigger that makes an existing imbalance more visible" - yep, there have been years of ferocious hiring - companies were scrambling to pick up anyone who could program their way out of a paper bag - there HAD to be a correction to that, and that correction would probably have come without AI, but AI is probably accelerating it ...
"The harder question is what all that extra productivity is actually going to achieve, and who gets to benefit from it" - that's very philosophical! What is the point and the purpose of it all - is 'work' only a means, or is it also a 'goal'? You'd say it's "only a means", but that's probably a simplification - work gives many people a sense of purpose ...
And regarding "who gets to benefit from it" - all of us (well, most of us) hope it's not just the oligarchs and the "Big Tech bros" who will benefit from it - but that's also an issue for the people who govern us to consider ...
The way I see it, this is just the fourth (fifth?) Industrial Revolution, and it will cause big changes in society and in the economy - but, like previous Industrial Revolutions, it does bring productivity gains, and hopefully the benefits will "trickle down" to all of us, not just accumulate at the top ...
Your Industrial Revolution comparison also made me think about Marx's Das Kapital, and some ideas from Mao's writings, particularly On Contradiction and On the Ten Major Relationships.
Marx raised an important question about who owns the means of production and who benefits from the productivity created by them. I think that question is becoming relevant again with AI. The models, computing infrastructure, data, and platforms are becoming important means of production. But the people using them don't necessarily own or control them.
Mao's writings offer another angle worth considering: technological and economic changes don't happen in isolation. They create new contradictions with existing social and organizational structures. If productivity changes much faster than the way work, employment, and wealth are organized, those contradictions become harder to ignore.
There's also a practical question here. We shouldn't judge progress only by how much more a company can produce with fewer people. We should also look at what actually changes in people's lives. Do they get more time, better opportunities, and a greater share of the value they help create?
I suppose this brings us back to your point about work and purpose. If AI can reduce the amount of human labor needed, that could be an opportunity to give people more freedom. But whether that opportunity becomes a reality depends on how society chooses to organize and distribute the gains.
Technology can expand what is possible. It doesn't decide who benefits.
And here's a slightly crazy thought to end with: maybe a few years from now, each of us will have a little AI robot companion following us around, helping with everyday tasks. Things could get a lot more sci-fi than we imagine. Hahaha! 🤣
Yeah I like your take, I "gemmed" this one, lol ...
"If AI can reduce the amount of human labor needed, that could be an opportunity to give people more freedom" - if the predicted productivity gains materialize then I could just see us moving from our current 5 day work week to a 3 or 4 day work week ...
I mean, that would make all the sense in the world, wouldn't it? Our (great-)grandparents worked 6 days in a week (Saturdays as well), so it's a natural evolution ...
Making this actually happen will need "new" (or stronger) social contracts, because certain people (looking at you, oligarchs and billionaires) might want to push back against this - just employing fewer people, or the same people but working fewer hours BUT for a lower salary, would probably sound attractive to them (more $$$ in their pockets and higher profit margins for their shareholders) ...
Your (or actually Marx's, lol) comment about "who owns the means of production and who benefits from the productivity created by them" is also spot on - I think at this moment a huge unbalance is being created between the oligarchs/billionaires/tech bros, versus the rest of us ...
Maybe we should start running more LLMs ourselves at home, lol? :P
Haha, exactly! 🤣 Technological progress is supposed to make life easier. Take the agent I trained: once I clock out, it keeps running and carrying on my work at the office. In a way, it’s like having a second me.
Theoretically, I could work roughly three days and still deliver about what I used to get done in five. But in reality, I just end up with even more test projects piled on my plate. We used to hold things back by saying we didn’t have enough hands — well, that excuse doesn’t really work anymore 😂.
And speaking of running LLMs at home… Jensen Huang must be sitting pretty. GPUs stay perpetually short in supply and just keep climbing in price.😂
Jensen Huang made a "whoopsie" ;-) and will have some questions to answer, but I have the feeling he'll be okay :-)
Probably just something to shift everyone’s attention away 😄
Your 36-defect list is where my queue rule earns its keep: an item only leaves the list when its closing is recorded — done, dropped, or deferred, with a date. Quietly vanishing items were how I lost whole threads of work without noticing. The queue is boring now, and boring is the point.
That's a good point. Finding a bug is one thing, making sure it doesn't get lost is another.🤣
With AI generating more findings, I can see why your queue rule matters. Every item needs an outcome, even if the outcome is simply deciding not to fix it.
And I like your last sentence. A boring queue is a good queue. 😄
competitor?😂
NICE WRITEUP bro really..
🤣🤣🤣
🫡🫡
💪🤣💪
Top 7 😁
🙌🤣🤣Luckly,哈哈哈
哈哈 you deserve it! Congrats 😍
Haha thank you! 😆 It’s nice to see this piece resonating with people.
But your badge???
🤣我也不知道，不知不觉就这样了
The bit about automation exposing all the messy handoffs that people used to quietly handle is the most honest thing I've read on this. The productivity number always counts the code that got written, never the person who used to notice the PRD was out of date. Hope your former colleague lands something soon.
Totally agree. Productivity dashboards love to celebrate how much we build, but they never see all the soft work that keeps the whole system glued together. Catching a stale PRD, smoothing messy handoffs, knowing when something just does not line up none of that gets counted. I hope my former colleague finds a great role soon, they brought so much of that quiet, valuable work to the team.
Glue work is the right name for it. It's invisible right up until the person doing it leaves, and then everyone wonders why things suddenly feel slower. Fingers crossed for them.
So true. Glue work keeps everything running quietly. I’m hoping he finds a great job soon.💪
Great post again, Lingfeng! 😍😍😍
Hahaha, thanks! This was pretty spontaneous. That 500-person QA group had been almost silent for months, then suddenly the floodgates opened and everyone started talking about AI. Guess people have been holding it in for a while. 😂
The 36 vs 11 result makes review cost visible: confirmed findings matter more than raw volume. I also liked the point that stale requirements can turn a technically correct test into a misleading report. Recording the requirement version alongside each finding could help separate product defects from spec drift.
Love this idea. Linking findings to requirement versions is a clean way to spot spec drift.
Agreed. Keeping the requirement version attached to a finding helps separate implementation drift from a changed acceptance criterion. Preserving the source wording or revision alongside a normalized identifier also makes later reviews much easier.
The detail about people collecting free credits every morning made me laugh and then wince a little. From the security side, that's where the budget problem turns into a data problem: screenshots of a staging build with a test account visible in the corner, or the PRD for an unreleased feature, going to whichever platform had credits that day under someone's personal login. When management isn't paying for the tokens, it usually isn't reviewing those vendors either. A company key with a per-team spend cap fixes both at once, and it costs far less than finding your roadmap sitting in someone's free tier. Did anyone in the group say whether their company has an approved tool list at all?
No one in the group brought that up. I also grab those free credits every morning, kind of like an insurance policy. If our company’s AI gateway goes down someday, I’ve got something to fall back on, hahaha 😂
Ha, insurance is a fair way to put it, and the timing is funny: Microsoft 365 was down for most of last night here, so having a fallback isn't a silly idea. I'd just keep the free tier for things you'd be fine seeing on a public forum and leave work files out of it. Thanks for the fun thread!
Couldn’t agree more. It’s smart to keep work data away from free tiers. I enjoyed this conversation a lot 😆
The 36-defect example is the part that stayed with me. AI found 36 potential defects against 11 from manual testing, but a good share of the AI finds were not bugs at all — just agreements between product and development that lived only in conversation, so the AI had no way of knowing about them. That feels like the real story here: the tool did exactly what it was asked, and the gap was in the shared context, not the model. The token-cost point lands the same way. If a company wants testers to use AI, paying for the tokens has to be part of the budget, not something people quietly cover themselves. And thank you for being careful with the poll numbers — 16 out of 76 in one group is a glimpse, not an industry survey, and it is stronger for being presented that way.
This is why I’ve stopped looking at the false positives as the AI messing up. It’s almost like it acts as an impartial outsider. It only reads what is written down, and catches every place where our team’s real understanding has drifted away from the document.
What strikes me most is how many companies treat AI experimentation as something employees should dip into on their own dime. Token costs are not a little hobby expense. If the business wants this workflow to work, it has to fund it properly. And the poll was really just a quick peek inside our own group, a glimpse through a narrow tube, so to speak. I never intended it to stand in for the whole industry.
The 36-vs-11 example is the clearest part — the AI did exactly what it was asked, but it was working from a stale document while the real agreement lived in conversation. That triage work is invisible on a productivity dashboard, and it's why the useful setup is AI surfacing findings for a person to check before anything is filed or acted on.
Exactly. Productivity dashboards usually only count how many findings AI generates, not all the hidden triage work. The human check is critical — especially when specs drift and AI is just faithfully following outdated information.
The 36 against 11 comparison is the part I keep thinking about, mostly because of what the other 15 turned out to be. They were not really false positives. They were places where the PRD said one thing and the team had agreed another in chat, and the agent could only test what was written down. That makes the tool a very honest reader of your documentation, which is useful and uncomfortable at the same time.
It also explains why the productivity numbers feel wrong from inside QA. The agent runs overnight and produces findings, but someone still has to triage each one, and that work never shows up on the dashboard that reports the gain.
At Ascendra Ventures we have started treating the written acceptance criteria as part of the deliverable for exactly this reason. If the spec is stale, every automated check downstream inherits the error, so keeping it current is scoped work rather than something people do when they have time.
After that review, did your team change who owns updating the PRD when an agreement is made in chat?
Totally agree. The triage work is invisible on metrics. We haven’t assigned a dedicated owner for PRD updates. We rely on weekly syncs to keep specs up to date, and it’s still a long game.
The distinction between generating more output and creating more value is important here. AI can increase test coverage and accelerate defect discovery, but it can also amplify outdated requirements and undocumented decisions. The knowledge-base example illustrates that well: the AI wasn't necessarily testing incorrectly; it was working from a different version of the truth than the team.
At IT Path Solutions, we see this as an important part of making AI-assisted development and testing work in practice. The real productivity gain comes when teams improve requirements traceability, review generated findings, and keep product, development, and QA aligned — not simply when they automate more steps. Otherwise, teams end up spending the time saved on resolving noise created by automation.
Perfectly put. AI scales whatever truth you feed it. Bad requirements turn into piles of noisy findings.
The 36-defect list is where a cost nobody budgets shows up: triage. Our overnight agent once filed 40+ defects in a single run, and 12 of them were the same bug rediscovered through different paths. The human cost isn't reading them, it's deciding them — every entry needs a verdict, and a runaway list borrows that time from actual testing.
What fixed it for us was treating report generation like an API with a rate limit, not a log. A per-run budget caps how many defects get enqueued, overflow lands in a backlog and earns budget again next run, and a dedupe pass collapses repeat findings before a human ever sees the list. Unlimited reporting made the agent look productive while making us less productive — the budget converted it back.
What I do is have our overnight agent check against the existing defect log before submitting anything, to avoid duplicates. Every found issue also gets re‑verified with fresh data to confirm whether it can be reproduced. Most of our invalid findings come from outdated requirements — the product and dev teams reach agreements offline, and those never make it into the documentation in time.
The 36 findings read to me as a hint that detection and triage want separate measurement. Detection can scale, but the judgment step needs its own numbers, and your four questions (real issues caught, false positives, run cost, maintenance when the UI changes) apply to each side on its own. Recording why each rejected finding was rejected would also pay off the next time the document drifts, because it separates a stale requirement from a detector error. How do you track the false-positive rate today, per release or per report?
We calculate it based on the final status of every submitted defect report.
The 36-defect story is the part that stayed with me — AI found 21 real bugs, and the rest weren't bugs at all, just a PRD nobody updated while the real agreement lived in chat. That's the bit that never makes the sales pitch: the AI doesn't just test the software, it audits how disconnected the team already is. And the token line — 'if management isn't paying, they can go to hell' — is the most practical sentence in the whole piece.
Exactly. Those extra flagged items are not just noise. They show how disconnected stale requirements are from what the team actually agreed. AI audits the team as much as the product. When it comes to tokens, I firmly believe I should never pay for work myself, hahaha 😂
Section 3 is the one I keep thinking about. When the same system writes the code and the tests, the tests mostly confirm the code's own assumptions. They pass, and they'd pass just as happily if the requirement had been misread at step one. The check has to come from something independent of the thing being checked.
We hit a human version of this on a recent backend rebuild: two teams on separate repos, and a rule that every "done" had to be verified against the running system, not against anyone's status notes. Several of our own notes turned out to be wrong. Nobody was careless; a written status is just a claim nobody re-checked. AI-generated tests feel like the same problem at a much bigger scale.
The "boss builds a demo, now build it" pressure rings true too. A demo proves something can exist; it says nothing about the edge cases that make it shippable. Has anyone in your group found a way to make that gap visible to management before the deadline gets set?
That's a good question. I don't think there's a magic formula, but one thing that helps is changing what we ask management to look at.
Instead of presenting a demo as a sign of delivery progress, we should make the gap visible: What has actually been demonstrated? What has been implemented? What has been verified against the running system? And what are we still assuming?
I'd also make the unknowns explicit before anyone commits to a deadline. For example, a feature might work perfectly along the demo path, while permissions, failure recovery, data consistency, or unusual inputs haven't been tested at all.
The important part is to show these gaps early, with evidence, rather than waiting until QA finds them near the deadline.
A demo should come with a gap list, not automatically with a delivery date.
I think this becomes even more important with AI. When building a demo gets cheaper, it's easy to mistake a faster proof of concept for a shorter path to a reliable product.
Great perspective
From a systems architecture perspective, AI isn’t eliminating complexity; it’s shifting where that complexity lives.
Generating code and tests faster means little if validation, security, and maintenance costs grow even faster.
The real productivity metric shouldn’t be how much AI produces, but how much reliable, maintainable software actually reaches production.
More output doesn’t automatically mean more engineering value. 🙂
This is such a sharp way to frame it. A lot of teams chase AI throughput and forget complexity just migrates into later stages. It’s not about how fast we generate things; it’s whether we can keep carrying that cost all the way to production. By the way, where were you on my last article, hahaha 😂
AI finding 36 defects against 11 manual sounds like a rout, but only 21 of the AI finds survived review, so more than a third were noise the team still had to triage.
Exactly. That’s the catch. Those extra flagged items look impressive at first glance, but a big chunk is just noise. Triage itself becomes a new, ongoing cost we can’t ignore. It shows raw numbers do not tell the whole story.
Another great post, Lingfeng! Loved reading this one too 😃
Hahaha thanks a lot! My next post will be Stratagem 33. I’m still polishing it. This one is just a filler to tide everyone over for now, hahaha 😂