that script will probably work tbh, but we have nowhere to run it. lets see if we can fix that...

this is pretty good, albeit a bit too simple

it can iterate in response to me invalidating its assumptions about my context! pretty cool

alright now lets take it up a notch...

this is more or less right, besides that it misunderstood how to use AWS credentials; very very close tho. already past what most of my juniors at work could have done with this level of specificity

lets see if it can fix that...

it didn't, claimed it did, and even explained what additional steps need to happen, violating my instructions. interesting, but oh well

alright so now lets stop managing our own database, that gets old quick.

p good so far, but the response kept getting cut off repeatedly, each time i reran. i asked it why and it was able to acknowledge this, theorize as to why, and even continue from where it had left off!

it didn't use Terraform for RDS however, but I also didn't ask it to. interesting that it didn't assume I wanted that; this is similar to how a junior may behave.

this one took some iteration as it kept getting truncated, but eventually it gave me something good

oddly enough, in one of the later iterations of trying to get around the truncation, it got the "right answer", but "forgot" that I had asked it to write a script, and reverted to a plain language explanation. indicative of contextual limitations?

had to cobble this one together across several carefully designed prompts; seems like human oversight and guidance is still very necessary for more complex tasks

cool, it was able to handle this slightly more complex bash usecase too!

it does perpetuate a mistake from a previous iteration where it assumes the old database also uses the credentials for the new one

on the plus side its able to explain what it did and why very well!

asking it to do something weird and kinda complex, it does great! with one minor mistake its able to correct. it also fills in the ambiguous parts of my ask with reasonable assumptions. we're definitely getting out of junior engineer territory 😳

it started erroring at this point so i'll stop bullying the poor thing. mildly surprised & p impressed with how well it performed at this complex series of related tasks! clearly its not perfect & not too great at remembering details across iterations but better than expected

one last thing; lets see if it remembers what we did!

the first result is good but seems too similar to my precise wording...
the second is better, but just reworded
the last is decent but too simple, and misses the iterations. but good enough!

luv 2 get feedback :^)

even the robot thinks i'm weird 😮‍💨

where does this notion of a typical prompt come from? has it been retrained to include its own conversations?

so, its even able to do meta-analysis and contrast to characteristics of previous experiences; where is this coming from? I find this particularly surprising tbh