you can get 100x gains by optimizing your code rather than throwing endless compute at your model

woah, that sounds dangerously close to saying this is all just developing computer software. Don't you know we're trying to build God???