Solving the Pain that is CI/CD - Dan Manges

The biggest pain point in the article stems from the inefficient development process when working on CI pipelines. Having to commit and push and wait for jobs to be picked up on remote infrastructure creates a painfully slow feedback loop. The presumed solution to this problem is being able to run pipelines locally. ... … The problem with the feedback loop isn't that you can't run the entire CI/CD stack locally. It's simply that the time from making a change to knowing whether that change worked is far too slow. It also just feels bad to commit and push code which ends up being broken, including for things as simple as syntax errors. Solving this problem is not hard, but it does require a few fundamental changes. … The CI/CD feedback loop is also too slow due to setup steps being repeated in each run. As you begin to make changes further downstream in a workflow, the feedback loop lengthens. Often, you're repeating the steps to install system packages, clone a git repository, install a language runtime, run a package manager, setup a database, etc. before you get to the point where your changes execute. … Another common complaint for CI workflows is YAML configuration. The presumed solution to this problem is avoiding YAML. However, this problem has been misdiagnosed. Effectively, the specification for CI pipelines is going to be *something* that wraps shell scripts. ... Ironically, some of the attempts from tools to solve this problem by moving away from YAML make the underlying problem worse. Writing pipeline definitions in specialized SDKs encourages more complexity outside of the shell scripts.