Multi-threaded programming is hard. I would say AI 🤖 models are bad at it based on how long this took and I had to figure out how to solve some of the core blockers myself, but honestly the model built the base parallel processor to deploy things in multiple accounts at once way faster than I would. It also helped me understand the ins and outs of how it worked - but it could not solve the problem with access to its own code and read permissions to the other projects. I had a heck of a time getting the error messaging right. I finally figured it out. I haven’t done a huge amount of parallel processing code but enough to understand that it is hard and locking can cause all kinds of problems. Data integrity is crucial. The need for atomic reads and writes. I understand the logic of it all but not a ton of the patterns people use and don’t ask me to recite all the specific code snippets off the top of my head - like someone did in an interview at AWS once. Recite how to implement a singleton pattern over the phone. Yeah no. So anyway you have this processor dealing with the queuing of things and I wanted it to be generic and reusable for different things. And it starts all these processes that do stuff. And those actions have dependencies so if the dependencies have not been met it needs to wait. And once the dependencies are met a specific type of tracking needs to be completed. And if there’s an error I want that to be added to the tracker diagram. At some point the wrong messages were showing up for the wrong items. Classic concurrency problems. I was asking agents in four different projects how to solve it. No joy. At one point one of the agents wanted to use flock to lock down file writes and it locked up the whole diagram. Got that fixed but the locking turned the parallel processor into a single thread app. So thanks. Ripped that out deal with collisions with waits and figured out how to make the three projects coordinate finally to consistently and accurately write the errors - across the stack - to the offending node, without clobbering the data. I moved running actual AWS commands to a separate project where I can validate things. So agents can’t touch that error logging and turn it off to solve a problem. That error messaging across all layers is the key to getting AI agents all they need to solve problems - without ever giving them my AWS credentials. And now the rest of my org should get done quickly. One thing left is renaming an account caused those accounts to fail - because the model uses a shaky method to look up the account id associated with a configuration node. As a result a rename to a non-standard pattern breaks everything without the deployment account id. It was a poor implementation in the first place and I had to stop the agent from changing code all over the place and break things. It can’t pinpoint problems in this complex application at times and guesses before doing a full code trace. But I think that’s fixed and now I just need to let it run. I didn’t want to leave it run with admin creds while sleeping even though it is all deterministic code. No AI. That’s why in my repo right now there’s mostly green at the top and then it starts showing a lot of red and yellow nodes. The other thing that tripped agents is handling the things that need to deploy to every account or every account in an environment. The agent producing the failed items list to retry from the data model was missing or adding things incorrectly. The diagram was too in its logic to flip colors of fits based on status. I had to basically hand hold the agents to walk data structures across projects. It was tricky. I would say for complex apps it is nearly impossible to use AI without: - proper error handling which AI (and a lot of people) are really bad at doing consistently and correctly. - verification methods agents can’t bypass. - visuals that help humans quickly see problems. - segregation - sandboxing https://github.com/2ndsightlab/ai-tracker