But it was being too nice — after I pushed back on the two key “findings” that were entirely intentional behavior, we ended up with 8 new issues — a couple of functionality issues and coding best practices — but the most valuable changes were around Accessibility and some wonky behavior around not being able to reach certain comments without “Expand All”.Opus actually came back with substantially different feedback, and a fair amount more detail, particularly finding several UX and UI issues that DeepSeek missed. It called out the built-in Matrix polling ECA model as a security issue (it’s entirely by design, not a security issue).
What I built, and why
But… testing it out on a post with over 200 comments, it proved to be… not really usable. And the whole tale illustrates how different the job of developers is today, compared to just a year ago.Opus, like DeepSeek, also failed to evaluate against Drupal caching layers. After having it write up its first round of conclusions, I prompted it to evaluate if there’s a good way to leverage Drupal’s cache layers that we are missing — and hit another 5-hour rate limit.Given all the hype around Astra and AI coding in general, I’ve found the state-of-the-art Frontier models are still far from perfect. And the Chinese open-model ecosystem is hilariously more cost effective to use, and able to find substantial flaws left behind by Astra. It’s also really interesting to see how different models are good at evaluating different things — Opus seems far better at finding UI issues than DeepSeek – whereas DeepSeek did a much better job isolating specific fixable performance issues — and did it at a tiny fraction of the cost.Now, I haven’t actually been using any IDE all that much recently — most of my recent work has been entirely using OpenCode, and if I need to edit anything, I just open it in NeoVIM. I’m finding I really like Zed as an editor — it’s fast, easy to find code, and it’s great for clicking on a file mentioned in the agent conversation to see exactly what’s there.One thing I didn’t like: when using an agent for development, I generally have it spawn sub-agents for much of the work — and these sub-agents are not visible in Zed at all. So for long spans of time, I could see the little spinner icon churning away, with no update as to what’s happening. When I use Opencode, I can easily switch to the sub agents and go through to see each one — but in Zed I just had to wait. Occasionally I would interrupt it to get a status — and sometimes it turned out a sub agent had failed or stopped for one reason or another, and so it hadn’t done anything at all! While other times it was just churning for 12 hours straight before coming back with a huge amount of work done.
Building with Astra, and Zed Editor
I had it change all the functional hooks to object-oriented hooks, and implement the new Route Attributes (which is why it has a minimum 11.4 version).Which brings me to today. Today I thought, let’s have a couple different LLMs review Astra’s code, and see what’s happening, what we might do to fix it. And let’s make sure there aren’t some other issues as well.
I would like to do a review of advanced comment threads, to cover several different angles: UI look and feel, user experience, performance, security. The module is in @advanced_comment_threads/ , and it’s deployed in @freelock – there’s a post there with over 200 comments: http://freelock.ddev.site:9177/blog/john-locke/2011-10/top-6-reasons-drupal-really-sucks-developer-edition — this is a local site copy so it can be used for testing Before publishing alpha1 of the module, I used e0ipso’s fantastic Self-Review application to go through the code. This actually took several hours — there was a lot of it! I was reviewing it to be comfortable understanding what it did, how it worked, and how well it adhered to Drupal best practices — not just current, but upcoming.
Token usage and rate limits
Another issue I had was Drupal Canvas – I’ve already found on several different sites that comments were not supported at all in the new Canvas page builder.Of even more interest was the cost. My Opencode Zen subscription was at zero, so I had to put in some more balance – but once I did, it completed the full review, helped me author some issues for Drupal.org, and evaluated the caching issues, for a grand total of 22 cents.But… there were a bunch of problems with the module that took another two days, and one of my banked resets, to work through. I basically hit 100% of my weekly usage 3 times to get the module to the alpha2 release.
Human Review
I had 22 questions in all, which I gave back to Astra, leading to a bunch more changes and improvements. I tested on a demo site freshly spun up, and on a local dev copy of our main internal Drupal-based PM system.
So I took the time to write this blog post.I’ve already built a simple automatic comment reload in Matrix Recipes – the Matrix Poll Endpoint. That creates an ECA model that automatically adds HTMX tags to comment fields, and basically gets them to reload the comment thread every 10 seconds. (That solution is pure config, no code necessary! Which illustrates the power of ECA, along with the magic of HTMX — but that’s for another post…)Then I published alpha1, and installed on a dev copy of this site, where I found a bunch of UX quirks and issues, particularly around the moderation workflow, and making it work better with our Dripyard theme — and then published an alpha2 with those fixes.As I’ve written before, I am deeply uncomfortable with some of the practices of the large AI companies, and don’t care to support them with large monthly subscriptions. Even so, getting on the /month plans from several providers gave me plenty of usage for much of this year — until last month. Starting in August, with the new generation of frontier models, it seems like their goal is to get you to scale up your plan. Astra certainly falls into that plan — I found it could go for an hour or two before hitting my 5-hour limit. OpenAI had given me two “usage resets” but I held off using them for a while – instead I just did other things and checked in now and then to restart its development (it was the weekend after all!)I just heard that DeepSeek v4.1 Flash is getting results comparable to Fable and Astra — but it looks like that’s not available on any of my plans yet, so decided to just try v4. I wanted to use Fable, but it looks like I don’t currently have access to that on any plan I have right now without buying more credits – so I settled for Opus 5. I gave both this prompt:
Performance issues
Last week I built a really great improvement to Drupal comments, the new Advanced Comment Threads module.By Monday it had made impressive progress, and I decided that my usage resets would get me a lot more tokens if I went ahead and upgraded my plan. Going to the 0/month plan gave me 5x the tokens – and made my usage resets much more valuable when I chose to use them. By Tuesday I had the module basically working – and I think OpenAI might’ve reset my usage once or twice more without telling me, because at one point it seems like it had all reset (and I see I have my full weekly usage available now, even though it wasn’t supposed to reset for another 3 days).[5 hours later…]
I also decided to try it out in Zed Editor, which I’ve been playing around a bit with as an improvement over VS Code.
DeepSeek results
DeepSeek gave some really great feedback, particularly around the thread loading, identifying a bunch of performance issues with clear paths to resolve. It gave the security and user interface reviews high marks, and user experience basically suffered due to the performance issues it identified, which made the large threads basically unusable.
I also had it review the thousands of test assertions created along the way — with test-driven development, a lot of these aren’t all that necessary for ongoing development — and it turns out there was a lot of duplication, and some big improvements to the test system it could do.On my /month “Pro” plan, I kicked off the same prompt with Opus, wanting to see what it would identify. I had my full 5-hour limit available when I started — and the review stalled out 3/4 of the way having used it all.
Opus results
But that solution won’t scale very far, especially if a post starts getting active comments, and having the entire comment tree refresh could become extremely annoying — so I started thinking about how I could make this better.
It failed to look at the caching strategy without prompting — one of Drupal’s great strengths is its multiple different caching strategies. But after I asked, it identified that Astra had actually put in code that entirely, unnecessarily, disabled the page cache, bypassed some of the object caching, and completely failed to cache the thread structures which might lead to some easy wins here.All in all, the review document Opus wrote was 1547 lines long, with 18 recommendations. Two of these were wrong – undoing or misunderstanding explicit design decisions. The performance recommendations were a bit hand-wavy compared to DeepSeek — DeepSeek had only 5 recommendations, and 238 lines of review — but it had clearly identified several specific server-side and browser-side bugs. Opus, on the other hand, gave a bunch of really good UI feedback, noting issues with dark themes, accessibility, and just plain broken JavaScript functionality.I’m involved in an initiative to modernize the old Drupal Groups site – which for a while was at the heart of the Drupal community, but is now basically dead. My part of it is building an integration between Matrix and Drupal, making it so people can converse over a chat interface, or right on the main Drupal.org site. And as I’ve built out the integration, I’ve come to find that Drupal core comments have had really no user experience improvements whatsoever in at least 15, maybe 20 years? And I couldn’t find any improvements in the contributed modules space, either.Drupal.org’s validation jobs had 5 out of 6 fail, so I’ll also have it do the code styling/linting clean-up to get those to pass, and then I’ll publish another alpha with those fixes for further testing in production environments, see if these improvements make the module more usable.The whole reason for making this an alpha was to collect some real-world usage to see how well it works. I added 3 different comment styles out of the box, 3 different loading strategies, and 3 different defaults for how much of each comment to show — mainly because I didn’t have any idea which of these options would work best. When I gave a demo to the team, I pulled up our page with the most comments on it — and when I clicked “Expand all”, nothing happened — it basically froze my UI.
Next steps
The current module does work through polling — but the planned next step is to implement a submodule to provide a long-running push connection to browsers, using a pluggable side channel that might be implemented in Node.js, Mercure, or FrankenPHP — when we go live on Drupal.org, we don’t want to have potentially tens of thousands of open browsers polling the main PHP server to find out if there’s any new comments, so the remaining work is to create a websockets,”SSE (Server-Sent Events)”, or another push mechanism to push comment notifications out to browsers, only when there’s actually a new one, so the central server doesn’t get crushed.Over a few days at the beginning of the month, I brainstormed some ideas of how a better comment system might work, particularly for inserting new comments in a page without a reload, without jumpy rerenders — and how to get it to work with Drupal Canvas.I had it flesh out a Design Tokens yaml file, having seen this get committed to Drupal 12 (and now I see the change record, and commit for 11.5 – need to go back and make sure we did this right!).After getting Opus’s writeup, I gave that to DeepSeek to evaluate and compare with its own findings. “I’ve read Opus’s full review (1,528 lines). Honest verdict: it’s substantially deeper and more accurate than mine, and it caught two things that invalidate conclusions I stated. Here’s the real comparison.”I also need to validate Drupal 12 compatibility — HTMX version 4 has already been added there, and the change record notes some explicit backwards-incompatible changes that I don’t think the module handles properly yet.
Summary
By the time I had a plan forward, OpenAI had just announced their newest Frontier model, GPT-6 Astra. And by the time I was ready to start development (last Saturday), it was available in my OpenAI Plus account (the /month level), so after hearing all the hype about how much of a generational improvement it was at everything, as well as how efficient it was at getting results without high token usage (compared to GPT 5.6 Sol or any Anthropic model), I decided to try it out.
