Toolkit: provide mechanism to auto-install build deps when building on azl

[dmcilvaney]

Merge Checklist

All boxes should be checked before merging the PR (just tick any boxes which don't apply to this PR)

• The toolchain has been rebuilt successfully (or no changes were made to it)
• The toolchain/worker package manifests are up-to-date
• Any updated packages successfully build (or no packages were changed)
• Packages depending on static components modified in this PR (Golang, *-static subpackages, etc.) have had their Release tag incremented.
• Package tests (%check section) have been verified with RUN_CHECK=y for existing SPEC files, or added to new SPEC files
• All package sources are available
• cgmanifest files are up-to-date and sorted (./cgmanifest.json, ./toolkit/scripts/toolchain/cgmanifest.json, .github/workflows/cgmanifest.json)
• LICENSE-MAP files are up-to-date (./LICENSES-AND-NOTICES/SPECS/data/licenses.json, ./LICENSES-AND-NOTICES/SPECS/LICENSES-MAP.md, ./LICENSES-AND-NOTICES/SPECS/LICENSE-EXCEPTIONS.PHOTON)
• All source files have up-to-date hashes in the *.signatures.json files
• sudo make go-tidy-all and sudo make go-test-coverage pass
• Documentation has been updated to match any changes to the build system
• Ready to merge

---

Summary

As part of a new test pipeline, it is necessary to install build dependencies for the toolkit. Instead of relying on a hard-coded list of packages that will get out-of-date, provide a mechanism to automatically install build dependencies when building on azl (ubuntu is possible, but would require more than just running tdnf -y install...).

To this end, add a canonical source of truth for requirements (prerequisites-src.json). This is validated against changes in our documentation with a github action. The requirements change so rarely I don't believe its worth the complexity of automating this processes, detection and alerting should be sufficient.

A user may then call sudo make install-azurelinux-prereqs to automatically install all the required *.rpms.

I believe we should not use this in our general documentation, better to leave the users to decide how they want to manage their own systems, but it is very useful for pipelines.

We should consider this for back-port to 2.0 so that our suggested pipeline workflows that partner teams may use can take advantage of it, but this is not critical.

Change Log

• Add prerequisites-src.json
• Add prerequisites.sh which can both print out prereqs, and validate the *.md files which are user facing.
• Fix up existing .md files to have correct ordering.
• Add Prerequisites Check GitHub action to detect inconsistencies in the prerequisite readme.
• Add install-azurelinux-prereqs build target which leverages prerequisites.sh to install the required packages

Does this affect the toolchain?

NO

Associated issues

• Tangentially related to (used in the validation pipeline for the fix):
  • https://microsoft.visualstudio.com/OS/_workitems/edit/54578711
  • https://microsoft.visualstudio.com/OS/_workitems/edit/55008881/

Test Methodology

• Using new mechanism: https://dev.azure.com/mariner-org/mariner/_build/results?buildId=675755&view=results
  • PR: https://dev.azure.com/mariner-org/mariner/_git/CBL-Mariner-Pipelines/pullrequest/21071

[cwize1]

This script seems overly complicated.

Would it make sense to have install-prerequisites-ubuntu.sh and install-prerequisites-mariner.sh scripts and then just have the Markdown reference those 2 scripts? That way, we still have a single source of truth but with significantly fewer lines of code.

[dmcilvaney]

The "real" install has some extra stuff I really don't want to deal with... this is for use with pipelines first and foremost. I'd be open to just removing the makefile target completly and having the pipeline call the script directly.

[nicogbg]

approved with minor comments

[dmcilvaney]

PR for the pipeline: https://dev.azure.com/mariner-org/mariner/_git/CBL-Mariner-Pipelines/pullrequest/21071