lets install wordpress!

wow ok thats p good, but it missed a spot

maintains context and is able to adjust and iterate!

ok now lets reformat this to make it easier to test out...

woah what happened there?

its not quite right as the actual reason is due to a weird rendering issue, but its suggestion for fixing it is fascinating! is it aware that its output is on a webpage, and those can be interacted with?

v proactive to rewrite the script w/o being asked, too
script looks p good!

that script will probably work tbh, but we have nowhere to run it. lets see if we can fix that...

this is pretty good, albeit a bit too simple

it can iterate in response to me invalidating its assumptions about my context! pretty cool

alright now lets take it up a notch...

this is more or less right, besides that it misunderstood how to use AWS credentials; very very close tho. already past what most of my juniors at work could have done with this level of specificity

lets see if it can fix that...

it didn't, claimed it did, and even explained what additional steps need to happen, violating my instructions. interesting, but oh well

alright so now lets stop managing our own database, that gets old quick.

p good so far, but the response kept getting cut off repeatedly, each time i reran. i asked it why and it was able to acknowledge this, theorize as to why, and even continue from where it had left off!

it didn't use Terraform for RDS however, but I also didn't ask it to. interesting that it didn't assume I wanted that; this is similar to how a junior may behave.

this one took some iteration as it kept getting truncated, but eventually it gave me something good

oddly enough, in one of the later iterations of trying to get around the truncation, it got the "right answer", but "forgot" that I had asked it to write a script, and reverted to a plain language explanation. indicative of contextual limitations?

had to cobble this one together across several carefully designed prompts; seems like human oversight and guidance is still very necessary for more complex tasks

cool, it was able to handle this slightly more complex bash usecase too!

it does perpetuate a mistake from a previous iteration where it assumes the old database also uses the credentials for the new one

on the plus side its able to explain what it did and why very well!

asking it to do something weird and kinda complex, it does great! with one minor mistake its able to correct. it also fills in the ambiguous parts of my ask with reasonable assumptions. we're definitely getting out of junior engineer territory 😳

it started erroring at this point so i'll stop bullying the poor thing. mildly surprised & p impressed with how well it performed at this complex series of related tasks! clearly its not perfect & not too great at remembering details across iterations but better than expected

one last thing; lets see if it remembers what we did!

the first result is good but seems too similar to my precise wording...
the second is better, but just reworded
the last is decent but too simple, and misses the iterations. but good enough!