Decide the Review Budget Before You Start

Recently I found myself in sort of a pickle... I had put myself in it, and was unaware in the moment of what was happening. In a world where AI harnesses arguably have become the standard for code development, there is a lot of confusion on what good or bad practices are. This is one piece of an operating discipline I actually wrote down for myself while doing my last project, and I plan on sharing more of them in the future.

The one I'd like to share now is reviewing code. The simple approach that makes sense is you iterate/plan your chunk of work, then the model writes the code, and then you use a built in code review tool or tell the model to review its work. This will work, but in my experience, it's not the best result. This is analogous to having any developer review their own work with no peer review. So it's pretty common for people to have another model (often a whole different platform) review that code. That, I would agree, is the right direction. Whoever/whatever did the work shouldn't grade it.

Now that implementer also should not get to tell its reviewer its justification for doing things. If anything needed explaining that's a hole in documentation and is something that should be corrected. To accomplish this I start a new blind session and usually use the built in code review tool. I've never had it come back with nothing, and whatever it does come back with, I copy and paste over into the implementing session. Then I reason it out with the implementer, and execute the changes that need to be made. Once those changes are made I actually review the code myself line by line. If I see anything, resolve it then. Next step is have the implementer essentially respond to the reviewer for a scoped review of the fixes. Then same process as before. Now this is where I got myself in a pickle...

One blind review, one scoped review followup, then stop. After the second review, sort out with your implementer on what the change is, and after that make sure tests are written to frame and leave receipts for those decisions. Anything you decide may be relevant but is out of scope, low priority, makes bad assumptions etc... that you file in a markdown document with what, why, and when. You never know when you may want to revisit it, and by then you will have forgotten, and the model will have no idea why it was decided last month. How did I end up in a pickle jar... in the middle of a really intricate and complex section I found myself on maybe my 5th or 6th review of the same section bouncing back and forth. I didn't even realize I was doing it. Every review seemed to have relevant points about claims vs implementation, alternate ways to do things, etc. Fortunately I caught myself, and here is what I've learned. The reviewer will find something forever. That's its job, and it can pick apart and question everything, pretty much every time; in this workflow you are the one that has to scope it and enforce that scope.

This isn't a tenet or a law or a best practice, it's what has worked for me. My advice, put very simply is: Decide the review budget before you start. For me, that's one blind pass, one scoped followup, and then tests and documentation. YMMV.