cicd: improve parallelisation of jobs using DAGs
If I am correct, all jobs from the analyse stage do not produce any artifacts. This means jobs from the test stage can run in parallel.
Unfortunately, GitLab's DAG have a serious limitation:
If needs: is set to point to a job that is not instantiated because of only/except rules or otherwise does not exist, the pipeline will be created with YAML error.
Others exist too: https://docs.gitlab.com/ee/ci/yaml/README.html#requirements-and-limitations.
This means we cannot needs on the cicd image jobs, as they typically do not exist during regular development work. It is also means that using DISABLE_* variables to disable certain stages will lead to problems.
-
Document using partial DISABLE_*can lead to errors because of this limitation, if such partialDISABLE_*is required the user can simply remove theneeds:causing fallback to basic stage-based pipeline.
What we can concretely do:
-
Jobs in the teststageneedthe quickest-runninganalysejob, as depending oncicdjobs is not possible.-
I believe the quickest analysejob isclang-format, test locally and posts results in this issue first to confirm.
-
- Jobs in the
packagestage should probably stay as is, you only want packages built when tests are passing. -
The current deploy job only needthe coverage report and the html docs, so it can run before sanitise and package jobs are done.However, do we want to deploy the review docs when sanitise and/or package are failing? I think yes, because it's just a review deployment. Opinions? (Comment.)-
Deploy review docs even when sanitise and/or package jobs are failing.
-
Perhaps I am missing some, start a discussion if you have ideas.
See also https://docs.gitlab.com/ee/ci/directed_acyclic_graph/index.html.
As of GitLab 12.8, jobs in DAG pipelines can be set up with no predecessors by specifying an empty needs like this needs: [] as presented on https://about.gitlab.com/releases/2020/02/22/gitlab-12-8-released/#jobs-in-directed-acyclic-graph-dag-pipelines-can-be-set-up-with-no-predecessors