Modules and package management proposal v2

[myitcv]

Proposal #851 presented an initial proposal on how CUE can fully support package and dependency management. Thank you to everyone who provided feedback on that proposal in the issue and subsequent discussions.

In this discussion, we present a revised package management and modules proposal that entirely replaces #851.

Please ask questions and provide feedback in this discussion. If there is sufficient demand/interest, we might look to arrange office hours to discuss the proposal and feedback in more detail, live.

Please subscribe to #2032 for future updates on package management and modules.

Abstract

There is a big demand from the CUE community to allow sharing CUE code within a broader ecosystem, much like the package managers that are available for programming languages or schema stores.

To address this demand, we propose the creation of a CUE registry that will serve as a central repository for CUE modules. A module is a collection of packages that are released, versioned, and distributed together. The registry will allow users to share and discover CUE modules and will provide versioning and dependency management features.

https://github.com/cue-lang/proposal/blob/main/designs/2330-modules-v2.md

Sub proposals

• Proposal: CUE modules storage model · cue-lang/cue · Discussion #2449
• Proposal: Module-publishing GitHub app · cue-lang/cue · Discussion #2448
• Proposal: CUE modules Supply Chain Security · cue-lang/cue · Discussion #2450
• Proposal: Module backwards compatibility · cue-lang/cue · Discussion #2451

[mxey]

The historical context that led to the choice of the direct, VCS-based design for Go does not apply to CUE. It seems likely that Go would have opted for a registry design if the historical context would have permitted it.

What is the historical context that no longer applies? While I understand the presented advantages of a registry, I still feel that the Go-style approach would be better for users:

• there would be no publish step:
  • everything in the VCS can be used. You can easily use a commit instead of a released version if you need to. Other tools like RubyGems which use a registry also usually support a separate VCS mode to let you use unreleased versions, so the ability to use VCS is present anyway.
  • there can be no difference between what's published on the registry. I think that's an advantage in simplicity and security (nothing is hidden). It's always clear that what you see on GitHub is what you actually use.
• private modules are easy to do because you just piggy-back the authentication on Git. Even with the Go Modules Proxy, you can just set GOPRIVATE and it still works.
  • Would a private registry be used selectively by CUE like Docker does? If so, how does CUE know which modules are on the private registry and which are on the public one, since github.com is presumably not the domain of the public registry?
  • If not, would a private registry have to proxy everything from the public registry? What if I need modules from multiple private registries? These scenarios can be quite annoying, with Maven for example, and Go completely avoided this.
  • We don't need to run a private registry for Go, Git just works. The CUE approach is more heavyweight and needs a CUE-only additional software.
• the public registry, as opposed to a simple proxy like with Go, needs to have its own authentication and authorization. Would it only work with Github anyway? If it supported other domains, how would I prove (and maybe continue to prove?) that I'm at example.com/foo?
• Using domains as the identifier and then the modules are not actually necessarily at that location can IMHO be quite confusing. At that point I'd rather you use cuehub.com in the module name instead of github.com, so it's not misleading.

Given the usage of CUE for configuration, I think private modules are actually a lot more important than they are with Go, and should be given more weight in any proposal.

[DavidGamba]

Lock files don't protect you from left-pad like problems (deleted modules). You would need to fork all your dependencies.

[verdverm]

You don't need a proxy for tag changes, a sum DB like gosumdb or cosign to verify integrity is sufficient. My understanding is that the Go proxy was mainly for performance reasons. Given that NPM presents a generous lower bound, I don't see performance as something that will be an issue no matter the solution. Back to tags, the cue.sum file is explicitly for verifying what you download is what you expect and will catch tag changes. The remote sumdb adds additional peace of mind. cosign does even better in this regard.

Central repositories have a host of their own security issues, as we can see from the numerous examples that have made the news. Many orgs are moving to a mirroring strategy to remove the middleman in the supply chain, both for security and reliability. I have had to implement retry functionality around every one of them we use. The arguments for supporting centralization made in the proposal are not strong in my opinion. If the proposal talks about advantages for the registry, it should also talk about the disadvantages as well as the adv/dis for alternatives. As this part of the proposal stands, it presents a very limited and biased perspective. I'm working on a long-form comment, which will go into more detail, but will take several days to consider & think through.

There are trade offs for both and no one feature is a trump card over the alternatives. Centralization is a very contentious and impactful decision and perhaps one that deserves a separate proposal. It at least deserves more debate.

FWIW, Go modules get out of the developers way, more so than any other dependency management system. I have not heard Go end users complain about dependency management. Docker's storied history is also worth considering. Both seem like something to put attention towards and use to weigh on this decision of centralization.

[mxey]

JFTR, the new subproposal for module storage resolves my concerns.

[verdverm]

Are you sure @mxey?

private modules are easy

It seems to be that if you have one private module, you must now mirror all public dependencies you use there. See my top-level comment, trying to confirm this deviation from other dep mgmt systems

In other words, a cue mod invocation will only talk to one registry at a time and requires CUE_REGISTRY to also be set

[mxey]

@verdverm I don't have any plans to use third-party modules, I just want a better way to depend on my own (private) modules.

Using an OCI registry means I won't need to set up another CUE-specific service.

[cedricgc]

I am not opposed to eschewing version control (VCS) support. I can understand the pain points integrating multiple VCS and overhead in cloning operations to effectively get some files. That said, we should support lightweight methods to pull CUE code without going through a registry.

Why would this be advantageous?

1. There are familiar decentralization arguments to be made here, and they look achievable even with a fallback mechanism like HTTP fetching of modules on the actual domain.
2. There are strong advantages for a data based language to have minimal friction for pulling structured data on the internet. If we take CUE's existing features and add a scheme for hosting/fetching that data from anywhere, I think we can actually have a lightweight but competitive protocol for developers to share canonical data and schemas. This can be potentially valuable in itself and open CUE for much more utility.
3. CUE is not a general purpose programming language. Much of the setup Go requires is because Go source must be gathered at once and compiled into an eventual executable unit. CUE can depend on other CUE code but we can fetch dependencies much more dynamically and incrementally. We can consider a case where we want a simple way to publish and share CUE cmd scripts that can be used in multiple projects, but is not tied as a dependency like schema code is. Possibly with HTTP module fetching we can use cue run with a URL to fetch the 'script' on demand?

Even if registries are self-hosted that friction can preclude some unique opportunities that CUE can do.

I think the main blockers of the current proposal is that the @v1 version would not conform to an actual URL that could be fetched and that hosting on domains like GitHub. I think we can support the registry while keeping a direct pull option open by making the version tag optional and defining a distribution format for CUE modules (like a zipped directory or a manifest) that can be hosted on any URL.

With the scheme above, the registry would be the fast automatic cache with fallback to the actual defined domain to fetch CUE code from anywhere if needed. This preserves the registry for caching and fallback while granting that same benefits to other domains at the module, not registry unit.

[verdverm]

A few points or opinions...

• Fetching dependencies from any HTTP endpoint or host, and then potentially cue running that code, is a huge security concern. This can be done at an application level, where appropriate.
• Fetching dependencies in production is an anti-pattern and should not happen automatically. It should be explicit for those who wish to do so and go against best practices.
• You can still have cue run domain.com/repo/proj@v1 like semantics which can fetch and dependencies then run the code. It's a two-phase approach (1) fetch mod & deps (2) do thing user asked. In fact, we actually use this for hof create. This example shows without versions, but also works for exact versions as well. Notably, docker run works this way. Also, Go build or run will only fetch the dependencies needed for the task at hand.

Generally, I think it best to build on existing and growing standards for packaging like OCI, over implementing new protocols and semantics. If CUE is going to be widely used in many contexts, security should be at the forefront of architectural decisions, especially as it applies to the software supply chain.

[khepin]

Under such a design, a module author publishes a version of a module to a registry. Module consumers or users then discover, resolve, and download dependencies via the registry.

This is likely to incur a non negligible cost in bandwidth. Taking away resources focus and attention from where it might (hopefully) be better allocated to cue itself.
A registry that hosts metadata on how to download the packages and modules and enables verifying them (checksum) would be a much cheaper alternative. It also allows multiple options for downloading the package if available.
Similar (I believe) to linux package managers. The registry hosts metadata, but the download can come from a variety of sources (mirrors).

Maybe something to consider around the potential monetary impact of the decisions here.

[AntPAllen]

I too am wary of another custom registry implementation. Hosting such a private internal registry in an organization is often more hassle than it is worth. Package management is the number 1 feature that I am waiting for in Cue, however, I think having to host a package manager that is accessible to the rest of our organization securely would be a major blocker in being able to drive Cue adoption within our org. Having either VCS or OCI support would, in my opinion, provide a frictionless approach to distributing modules within an organization.

[kajogo777]

I built 2 PoCs based on the new module proposal with a centralized repo and have been using it for the past couple of months:

The first PoC I tried used inline imports to compile packages into a single file that can be published and shared, but this did not work well for schemas, and some constraints as references were removed so effectively the schema changes (created an issue here)

The second PoC creates an overlay of all the files and their paths as is, then copy them over to an abstract storage (This can be an OCI overlay, no-SQL document, a bucket whatever) with semantic versioning, but here are the issues I encountered while implementing this:

1. How should we resolve conflicts in nested dependency versions? -> At the moment this implementation just refuses to install
   conflicting nested dependencies
2. There is currently no way to add multiple versions of nested packages at the same time inside mod.cue/pkg
3. Validating packages cannot be done on the fly in memory, CUE load still requires a file system -> I used temp dirs
4. There is no canonical CUE format, this prevented me from generating cryptographically secure digests of CUE values, this will be super important for configuration supply chain security (I don't believe generating file digests is enough, slack thread here)

All the code is publicly accessible in this project, it's still not very well documented but I would be happy to walk anyone interested through it.

[verdverm]

1: MVS is how Go, Hof, and CUE (plans to) resolve dependency versions

2: Multiple versions of the same dependency is very complicated and confusing to users. My opinion is to not do this.

4: git, OCI, and Hash1 are all crypto secure, so the code can be validated throughout the supply chain

Hof's dependency management for CUE is implemented in these two dirs:

• https://github.com/hofstadter-io/hof/tree/_dev/lib/mod
• https://github.com/hofstadter-io/hof/tree/_dev/lib/repos

[kajogo777]

1: How will MVS be used to resolve dependency versions in the context of this proposal?
2: In my use case there are multiple packages that depend on different versions of an underlying standard library of CUE modules, using the same underlying standard library is guaranteed to break one of the packages
4: But these hashes are sensitive to white spaces and how the author writes CUE values, while the defined schema should be the same
for example

a: b: 123
# should have the same digest as
a: {
  b: 123
}

this will prevent different formatters and configuration processing tools from accidently changing the digest

[verdverm]

How will MVS be used to resolve dependency versions in the context of this proposal?

https://github.com/cue-lang/proposal/blob/main/designs/2330-modules-v2.md#minimum-version-selection

It's a choice they made for a version selection algorithm, there are a number of resources for understanding how it works.

2: In my use case there are multiple packages that depend on different versions of an underlying standard library of CUE modules

You will have to resolve this, the CUE standard library should be stable so that this is not the case, we are pre-v1, so the compatibility guarantees have not started yet. If you are referring to your own set of modules that form a basis for your users, I think the same logic applies.

But these hashes are sensitive to white spaces and how the author writes CUE values

Modules are the unit of distribution, versioning, and hashing. There should not be any differences, white space or ordering, when it is fetched and used as a dependency. If it is modified, it should be invalidated.

---

I would also be aware that the current proposal is not finalized and there are changes coming fwiu. There were conversations at KubeconEU that have influenced the proposal but have not made their way to the document. I wasn't there and have not had time to sync up with the CUE team since then.

[verdverm]

From: https://github.com/cue-lang/proposal/blob/main/designs/modules/2449-modules-storage-model.md#detailed-design | https://github.com/cue-lang/proposal/blob/main/designs/modules/2449-modules-storage-model.md#working-with-private-registries

The location of modules will be determined by a single setting, configured on the command-line by setting $CUE_REGISTRY, that’s used to prefix all module paths in order to find the contents of a module.

One of the advantages of using the OCI registry model is that it’s possible to use a private or cloud-specific registry to serve content. We see two main requirements there: (1) Copying modules from the central registry into a local registry. (2) Uploading private modules to a local registry only.

This seems to imply, if I am fetching dependencies, they all must come from the same registry. By proxy, this means if I want to have one private module, I must then mirror all other dependencies.

Is this the case and intention of the design?

Why is fetching from multiple locations in one command, such as public modules from the public registry, and private modules from a private registry, not part of this design?

Every major package manager supports this, why is CUE deviating from the standard? No rationale is provided for this decision

• https://doc.rust-lang.org/cargo/reference/registries.html#using-an-alternate-registry
• https://docs.npmjs.com/cli/v9/configuring-npm/package-json#git-urls-as-dependencies
• https://cloud.google.com/artifact-registry/docs/nodejs#scopes (for how we can scope sets of NPM packages to a registry, GAR in this case)

Most seem to also support multiple registries side-by-side with going directly to source as will, usually limited to git. There is a lot of value in being able to just tag a git commit and have a module available, with no publishing step required.

Given CUE cites these two ecosystems as models, why is the deviation to force a single registry part of the proposal?

[nyarly]

But these hashes are sensitive to white spaces and how the author writes CUE values

Modules are the unit of distribution, versioning, and hashing. There should not be any differences, white space or ordering, when it is fetched and used as a dependency. If it is modified, it should be invalidated.

I think the concern here should be that an adversary might use insignificant white-space in order to produce a collision with a module they're trying to subvert. If the threat model is "modules might be trashed in transit" and you want a digest for error correction, by all means use SHA-1 on whatever comes over the wire. If you want to prevent a MITM from e.g. changing DNS: good.org to DNS: evil.com, then a strong hash on a canonicalized form is called for.

[myitcv]

This proposal was superseded by #2939. As we updated in #2939 (comment), that proposal (and its sub-proposals) have now been accepted, hence we are closing this discussion to represent the decision on the v3 proposal.