Rendered at 17:32:08 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
Jdstanhope 30 minutes ago [-]
I have been using Claude to build a clone of Bash in Rust (https://github.com/jdstanhope/huck). It has been goin going on now for 81 days, 2600 commits and I have covered a good chunk of the features such that I can source by .bashrc with a number of extensions. It has been doing pretty good but it will sometimes spend hours to do few line fix and other times pushback on making big changes despite have over 3000 tests.
That being said, I am still guiding it on a regular basis and only recently let it run open ended trying fixing small to medium size bugs. The next stage will be try let it run in a loop trying to fix the 100 issues it has identified as well as getting most of the bash test suite to pass.
rco8786 14 minutes ago [-]
> it will sometimes spend hours to do few line fix and other times pushback on making big changes despite have over 3000 tests.
you ever worked with human engineers before? :D
Jdstanhope 11 minutes ago [-]
I was going to say that it's pretty much exactly like working with an experienced software engineer ... except it never seems to take a step back and look at the bigger picture. It will happily fix the same class of bugs over and over without realizing a bigger change is required.
justincormack 12 minutes ago [-]
I find most of the time it would be fine on its own, but every now and again you avoid a really bad degradation through guidance. Some kinds of big change definitely problematic.
gipp 19 minutes ago [-]
"Reproduce an existing piece of software" doesn't seem like a problem that generalizes well to "develop a new piece of software," for a fair number of reasons.
thisisauserid 26 minutes ago [-]
I believe they can make ANY software project the largest it can be!
This solution should look for more problems.
marginalia_nu 24 minutes ago [-]
Yeah came here to say the same thing. The Hard Problem of software engineering isn't making a complex monstrosity with millions of lines of code and endless feature creep, but not doing that. This was as true 20 years as it is today.
AlotOfReading 3 minutes ago [-]
I apologize for such a big program — I didn't have enough tokens to write a small one.
- my coworkers
muglug 9 minutes ago [-]
> MirrorCode only covers programs with plain-text output. MirrorCode does not test AI abilities in software that is
inherently visual
It’s an interesting eval but this is a distinct limitation — most applications that consumers use directly are not purely text-based.
jph 32 minutes ago [-]
I'm experimenting with large-ish software projects using Claude Fable. The results are surprising to me.
- Project portfolio management software. Good results so far. This is unguided experimentation: Fable researches feature ideas and plans them, then Opus 5 implements them, without me in the loop. The features include things such as an auth server, full-text search, audit logging, resource leveling, HA/DR, IaC, and more.
- FHIR & SNOMED medical software experimentation in demo-only environments. This has mixed results so far. This is guided experimentation: the FHIR specifications and SNOMED specifications are thorough and I'm in the loop. Fable plans OK, but Opus 5 is thrashing a lot, going down many rabbit holes.
stared 18 minutes ago [-]
I am curious how Claude Opus 5 fares - similar, better, or (my guess) worse than Fable 5.
logicchains 12 minutes ago [-]
For really hard problems (maths) it's far far worse than Fable 5, not even as good as Sol 4.6. Current benchmarks are inadequate as they fail to capture how much worse Opus 5 is than Fable 5.
I'm a bit suspicious of this as there are open source excel clones already.
It's impressive AI can put together it's own version, but I can't help but feel it's probably just relying on existing excel clones as it's a well trodden space.
Novel applications are what impress me. Not a clone.
swframe2 2 minutes ago [-]
Agreed. Novel would be very impressive. AI tech is struggling with that.
It is important not to wait for today's AI tech to get to AGI. It is better to exploit the useful things it can do today while waiting for it to get better tomorrow.
Cloning existing software might be useful today.
It allows the model to
1) learn from its mistakes
2) mix known solutions (embed product A inside product B)
3) mine value (turn product A into a set of reusable libraries)
etc.
looksjjhg 45 minutes ago [-]
Why just the commercial ones !??
jdw64 54 minutes ago [-]
I'm curious about that too.
I'm building a language with AI, and when I leave the AI running on its own, it keeps generating code in strange directions. I wonder what kind of structure AI-only code would have.
onlyrealcuzzo 24 minutes ago [-]
Also building a language with AI.
It's decent at design and implementation, but strangely terrible at language consistency - and expectedly architecture.
Steering it on language consistency is easy. Steering it on architecture has required me to learn a lot more about compilers than I wanted to do for this project (since that's a lot more than I wanted to take on for something that was supposed to be a "fun" "side" project).
jdw64 15 minutes ago [-]
I have a similar problem.
I'm not sure about architecture. When I first implemented the slot system in my language, I designed it around that implementation, and in the end, the architecture suffered. There were quite a few cases where consistency was simply unavoidable.
It's not easy to create a completely new architecture with something that doesn't exist in existing languages. Consistency doesn't come together as easily as you'd think.
In that sense, I really admire the people who first introduced something new.
When I started implementing it, I focused on one killer feature and just went for it—with the feeling that if this succeeds, my language will succeed. But once I started building around it, it turned out to be very different from other compiler architectures.
With CRUD or other well-established domains, it's easier to maintain consistency. But achieving internal consistency in a program seems to be a different kind of challenge.
ivan888 48 minutes ago [-]
Is it like self driving cars, where 100% non-human code would be arguably safer?
But then it always comes down to having some human who is accountable for verifying the behavior. I think ultimately we just need a better set of languages and frameworks to minimally encode the types of applications we want to build today, so that human comprehension and verification of the source code becomes easier.
LetsGetTechnicl 45 minutes ago [-]
How would 100% non-human code possibly be safer?
micromacrofoot 37 minutes ago [-]
well if driving is supposedly safer when automated, why not code
dd8601fn 16 minutes ago [-]
The qualities of computation that make autopilots safer than human drivers aren’t the same qualities that produce better designed software products.
LetsGetTechnicl 24 minutes ago [-]
Is it actually safer? If you can't understand the code you can't verify it's safety, and AI code often has bugs in it. You can't trust implicitly.
ModernMech 31 minutes ago [-]
It’s supposedly safer than the average driver, not safer than professional drivers.
jdw64 40 minutes ago [-]
I've noticed that when only AI writes the code, there are cases where it ends up being unusable by humans. For small code snippets or function-level units, AI is often much better than me. But when I look at the whole picture, AI tends to perform poorly.
That being said, I am still guiding it on a regular basis and only recently let it run open ended trying fixing small to medium size bugs. The next stage will be try let it run in a loop trying to fix the 100 issues it has identified as well as getting most of the bash test suite to pass.
you ever worked with human engineers before? :D
This solution should look for more problems.
- my coworkers
It’s an interesting eval but this is a distinct limitation — most applications that consumers use directly are not purely text-based.
- Project portfolio management software. Good results so far. This is unguided experimentation: Fable researches feature ideas and plans them, then Opus 5 implements them, without me in the loop. The features include things such as an auth server, full-text search, audit logging, resource leveling, HA/DR, IaC, and more.
- FHIR & SNOMED medical software experimentation in demo-only environments. This has mixed results so far. This is guided experimentation: the FHIR specifications and SNOMED specifications are thorough and I'm in the loop. Fable plans OK, but Opus 5 is thrashing a lot, going down many rabbit holes.
It's impressive AI can put together it's own version, but I can't help but feel it's probably just relying on existing excel clones as it's a well trodden space.
Novel applications are what impress me. Not a clone.
It is important not to wait for today's AI tech to get to AGI. It is better to exploit the useful things it can do today while waiting for it to get better tomorrow.
Cloning existing software might be useful today. It allows the model to
1) learn from its mistakes
2) mix known solutions (embed product A inside product B)
3) mine value (turn product A into a set of reusable libraries)
etc.
I'm building a language with AI, and when I leave the AI running on its own, it keeps generating code in strange directions. I wonder what kind of structure AI-only code would have.
It's decent at design and implementation, but strangely terrible at language consistency - and expectedly architecture.
Steering it on language consistency is easy. Steering it on architecture has required me to learn a lot more about compilers than I wanted to do for this project (since that's a lot more than I wanted to take on for something that was supposed to be a "fun" "side" project).
I'm not sure about architecture. When I first implemented the slot system in my language, I designed it around that implementation, and in the end, the architecture suffered. There were quite a few cases where consistency was simply unavoidable.
It's not easy to create a completely new architecture with something that doesn't exist in existing languages. Consistency doesn't come together as easily as you'd think.
In that sense, I really admire the people who first introduced something new.
When I started implementing it, I focused on one killer feature and just went for it—with the feeling that if this succeeds, my language will succeed. But once I started building around it, it turned out to be very different from other compiler architectures.
With CRUD or other well-established domains, it's easier to maintain consistency. But achieving internal consistency in a program seems to be a different kind of challenge.
But then it always comes down to having some human who is accountable for verifying the behavior. I think ultimately we just need a better set of languages and frameworks to minimally encode the types of applications we want to build today, so that human comprehension and verification of the source code becomes easier.