cicd: improve parallelisation of jobs using DAGs
If I am correct, all jobs from the analyse
stage do not produce any artifacts. This means jobs from the test
stage can run in parallel.
Unfortunately, GitLab's DAG have a serious limitation:
If needs: is set to point to a job that is not instantiated because of only/except rules or otherwise does not exist, the pipeline will be created with YAML error.
Others exist too: https://docs.gitlab.com/ee/ci/yaml/README.html#requirements-and-limitations.
This means we cannot needs
on the cicd image jobs, as they typically do not exist during regular development work. It is also means that using DISABLE_*
variables to disable certain stages will lead to problems.
-
Document using partial DISABLE_*
can lead to errors because of this limitation, if such partialDISABLE_*
is required the user can simply remove theneeds:
causing fallback to basic stage-based pipeline.
What we can concretely do:
-
Jobs in the test
stageneed
the quickest-runninganalyse
job, as depending oncicd
jobs is not possible.-
I believe the quickest analyse
job isclang-format
, test locally and posts results in this issue first to confirm.
-
- Jobs in the
package
stage should probably stay as is, you only want packages built when tests are passing. -
The current deploy job only need
the coverage report and the html docs, so it can run before sanitise and package jobs are done.However, do we want to deploy the review docs when sanitise and/or package are failing? I think yes, because it's just a review deployment. Opinions? (Comment.)-
Deploy review docs even when sanitise and/or package jobs are failing.
-
Perhaps I am missing some, start a discussion if you have ideas.
See also https://docs.gitlab.com/ee/ci/directed_acyclic_graph/index.html.
As of GitLab 12.8, jobs in DAG pipelines can be set up with no predecessors by specifying an empty needs
like this needs: []
as presented on https://about.gitlab.com/releases/2020/02/22/gitlab-12-8-released/#jobs-in-directed-acyclic-graph-dag-pipelines-can-be-set-up-with-no-predecessors