Rewrite it in python, rust, or go #789
Comments
|
With nixos-anywhere we need to be able to build disko from source in a memory constraint environment. Currently our kexec image targets 1GB of RAM memory usage. We cannot depend on a binary cache just now because disko is not part of nixpkgs and the disko configuration is derived from the module system. We do have a dependency of python in the installer. We almost got rid of perl because of the perlless profile except for nixos-generate-config. As see just now is that both the go and rust compiler might be almost as large as our kexec image (also this is compressed). |
|
One nice property of bash besides the really bad error handling is that its execution is very easy to introspect with |
|
also a problem of the rewrite is, that we can only rewrite the creation and mounting, the nixos configuration has still to be generated in the module system, so we would have two parallel implementations of the disko code. |
|
I don't follow this logic. We already have _create and _mount in bash so there is already a second imperative implementation next to the NixOS configuration. |
|
yes, but instead of templating bash we would be templating rust, python or go code? |
|
@Mic92 that's very useful information, thank you! That makes it seem like python is the only viable option.
Which installer? I booted a NixOS ISO and didn't see python in there, but I assume you're talking about something else?
This is not a problem, it's just a different way of modelling the problem space. Of course, we have to avoid duplicate implementations. If a value is required by both the nixos configuration and the mount/format code, its value must be computed in nix code. But there's no need for all the logic that actually creates partitions and mounts them to be implemented as templated bash, we can choose to implement them however we want.
No, not at all. My goal is is remove all It's true that we would have to change the structure a little, so next to
Very true! Well wouldn't you know it, python allows you to do exactly that with the trace module. |
Afaik it's used for systemd-boot installer, but it's not in $PATH - which might be a problem for disko actually. However we have it in our kexec image for sure for restoring network configuration.
I wouldn't use templating at all. I would parse json instead and generate a plan based on that. |
|
However overall it might be quite ambiguous to re-write a 3k LOC project written in a terse language in one go. I would suggest if you want to go this route is to replace one smaller module with some prototype so we can see if this goes in the direction we are all happy with. |
|
nushell is an interesting option for this use case. it has much better error handling and debugging, and is meant to easily interop with existing shell tools that are heavily used in disko. |
|
That's an interesting idea. It's been a few years since I had a look at nushell. I'll read up on it. I also considered xonsh, it has a similar advantage of allowing easy interop with shell tools while working with resulting values in the python world. How well is nushell suited to building CLIs in your opinion? |
|
How would nushell compare to pythonMinimal in terms of file size? The binary release sizes at https://github.com/nushell/nushell/releases seem to suggest they might be in the same ballpark? . |
|
+1 to nushell. Much better handling of structured data, easy interoperability and few if any footguns compared to bash |
|
I'd throw elvish into the mix too. I haven't played around with nushell but I've started to convert all my scripts to elvish mainly because of the much better/enforced error handling. Its like what everyone expects from bash's |
|
We already depend on pythonMinimal for some filesystem utilities. Not just the networking script, that I added in nixos-images btw. Adding nushell would increase our image by 50MB (currently at 300MB), which seems like a lot. |
|
Yes but at the same time you could drop dependency on most of coreutils, jq and the like as all data processing can be done within Nushell directly. |
Rewriting in python also allows dropping jq. Most other dependencies are always part of a nixos installer, so we likely don't save much there. I also want to point out that many of the tools we use can't be dropped. For example, you might think that sys disks can be used to replace lsblk, but it doesn't give us enough information: So the savings here might not be as big as you assume.
Which kind of disk image are you talking about here, and how are you determining the 300MB?
Just for fun, I thought I try to find out using nushell itself: ~> nix path-info -r --json 'nixpkgs#nushell' | from json | select path narSize | update narSize { into filesize } | append { path: Sum, narSize: ($in.NarSize | math sum | into filesize)}
╭───┬────────────────────────────────────────────────────────────────┬───────────╮
│ # │ path │ narSize │
├───┼────────────────────────────────────────────────────────────────┼───────────┤
│ 0 │ /nix/store/1w90l4fm5lzhlybipfilyjij2das6w98-openssl-3.0.14 │ 6.3 MiB │
│ 1 │ /nix/store/22nxhmsfcv2q2rpkmfvzwg2w5z1l231z-gcc-13.3.0-lib │ 8.7 MiB │
│ 2 │ /nix/store/3dyw8dzj9ab4m8hv5dpyx7zii8d0w6fi-glibc-2.39-52 │ 28.8 MiB │
│ 3 │ /nix/store/8hq38sx1hjgganlj6dn0fvvba0g7gcpi-libidn2-2.3.7 │ 352.6 KiB │
│ 4 │ /nix/store/b4jwx295fqxg036jxb5bavvd4knikiix-nushell-0.98.0 │ 37.7 MiB │
│ 5 │ /nix/store/lw2macnzp3av47m41qv6vcnq2daibgl1-libunistring-1.2 │ 1.8 MiB │
│ 6 │ /nix/store/q0iz2x35ki1aaqpjagjfi8s7q053cmc5-gcc-13.3.0-libgcc │ 155.9 KiB │
│ 7 │ /nix/store/x45xiwvk2v97bw1yp1vb6rnmvhagpgyz-xgcc-13.3.0-libgcc │ 155.9 KiB │
│ 8 │ Sum │ 83.8 MiB │
╰───┴────────────────────────────────────────────────────────────────┴───────────╯
~> nix path-info -r --json 'nixpkgs#python3Minimal' | from json | select path narSize | update narSize { into filesize } | append { path: Sum, narSize: ($in.NarSize | math sum | into filesize)}
╭────┬────────────────────────────────────────────────────────────────────┬───────────╮
│ # │ path │ narSize │
├────┼────────────────────────────────────────────────────────────────────┼───────────┤
│ 0 │ /nix/store/22nxhmsfcv2q2rpkmfvzwg2w5z1l231z-gcc-13.3.0-lib │ 8.7 MiB │
│ 1 │ /nix/store/3dyw8dzj9ab4m8hv5dpyx7zii8d0w6fi-glibc-2.39-52 │ 28.8 MiB │
│ 2 │ /nix/store/3hxxbbjc8r66nravvjind6ixhz7cpij1-mpdecimal-4.0.0 │ 217.0 KiB │
│ 3 │ /nix/store/4w3c2c11j2kmwxrznji2i7is53z5aldi-xz-5.6.2 │ 807.4 KiB │
│ 4 │ /nix/store/8hq38sx1hjgganlj6dn0fvvba0g7gcpi-libidn2-2.3.7 │ 352.6 KiB │
│ 5 │ /nix/store/f0dmadrh535bpayz2r4q5lslg0m17lnw-expat-2.6.2 │ 271.0 KiB │
│ 6 │ /nix/store/izpf49b74i15pcr9708s3xdwyqs4jxwl-bash-5.2p32 │ 1.6 MiB │
│ 7 │ /nix/store/lhfgwjwwdacygiq702yvbahvpqmxivpx-libxcrypt-4.4.36 │ 128.0 KiB │
│ 8 │ /nix/store/lw2macnzp3av47m41qv6vcnq2daibgl1-libunistring-1.2 │ 1.8 MiB │
│ 9 │ /nix/store/p6mm7p3i4325i2df4l993plgbmscdhlr-libffi-3.4.6 │ 71.9 KiB │
│ 10 │ /nix/store/q0iz2x35ki1aaqpjagjfi8s7q053cmc5-gcc-13.3.0-libgcc │ 155.9 KiB │
│ 11 │ /nix/store/qhf8n3srrg0vdd44k7q71fb0p5az15b5-bzip2-1.0.8 │ 78.9 KiB │
│ 12 │ /nix/store/rqs1zrcncqz3966khjndg1183cpdnqxs-zlib-1.3.1 │ 125.2 KiB │
│ 13 │ /nix/store/saa0d82cwvsy8y8qx3z8lp64gh98a689-python3-minimal-3.12.5 │ 26.3 MiB │
│ 14 │ /nix/store/x45xiwvk2v97bw1yp1vb6rnmvhagpgyz-xgcc-13.3.0-libgcc │ 155.9 KiB │
│ 15 │ Sum │ 69.4 MiB │
╰────┴────────────────────────────────────────────────────────────────────┴───────────╯So yeah, the difference is pretty small, but as @Mic92 said we already depend on python for some utilities, and I'm unsure if all of them could/should be rewritten as well.
Something like It does, however, have very nice built-in logging levels.
Then maybe you can answer a question: What is the error handling story of nushell? What happens when a command fails? I found create your own errors, which is cool, but I'd like to know how to handle a command that can potentially fail (like invocations of sgdisk). Note for implementation: parted has a --json flag that may allow for easier parsing of its output than the currently used gptfdisk. |
There are two approaches here. You can wrap the command in Also, the store-info size can be shortened to:
|
|
Ahh nice,
Not true, you need to remove |
|
It seems we're getting slightly different output of As for lsblk if all that's missing are uuids they can be retrieved like this: Other missing fields can similarly be retrieved by |
|
That seems to be the case. Could be because I'm running Lix instead of NixCpp. Indeed, though I feel like that's not a good use of developer time. Using |
|
Maybe not but it's one less utility to depend on |
|
I feel like this is at risk of becoming a bit of an academic debate until someone goes and writes a demo with a port of at least some Disko modules to either python, nushell or whatever seems fashionable and does not inflate disk size requirements too much :) Should be doable and advice is available if someone decides to invest the Time. Only then can we really decide on its merits. An idea I would like to tryout sometime is to just serialize the resulting Disko options to JSON instead of bash script snippets and parse that JSON in whatever language, be it jq, python or nushell. With a schema sprinkled on top this could help us becoming more flexible in both directions |
|
@phaer I started hacking on it |
|
After a lot of experimentation, I've now decided definitively that Python is the way to go. If you're interested in checking the progress, you can see my (now abandoned) attempt at rewriting in nushell at disko-rewrite-nushell and the current attempt at rewriting in python at disko-rewrite-python (which has feature parity with the nushell version at the time of writing). First off, I want to note what I liked about nushell for this task
However, there is still a lot of stuff missing from nushell for it to be a sensible choice for this usecase:
ConclusionI will be using Python going forward. I'm not opposed to anyone advocating for other languages still, but before you do, please read what I wrote above and think about whether your suggestion would solve the issues I outlined. |
|
This issue has been mentioned on NixOS Discourse. There might be relevant details there: https://discourse.nixos.org/t/what-is-nixpkgs-preferred-programming-language/53848/40 |
|
As I see disko is being rewritten in Python? |
|
Partially. We just want to replace most of if not all the bash with python, not the nix code. You can check out |
|
I read the thread and understood there's a problem with dependencies. I think would be a good idea rewrite Disko in C/C++ because glibc ships with NixOS by default, lang is easy to read and expand and blazing fast. |
|
@Sk7Str1pe Sure, feel free to give it a go. You can use my branch(es) for reference. I think you'll find quickly that neither C nor C++ is a good language for this task. I will be using Python. |
Problem
Disko is currently somewhat tricky to maintain. Part of this comes from the current architecture that puts nix itself at the center of the entire operation, and uses it as a crude templating engine for bash scripts.
This makes implementation of some features difficult. A few examples (I'll add more when I come across them):
by-partlabel#551Proposed solution
Switch to a different architecture where the nix files only define configuration values, and a bigger program written in a proper language (not bash or perl) runs
nix eval --jsonand works with the JSON output to plan and execute all the required actions.Considerations
Compatibility
The interface for users should not change at all.
disko.devicesshould have the exact same type and result in the same configuration values as before.Some programs (most notable nixos-anywhere, AFAIK) use disko's
diskoScriptoutput and call this script directly. Because this is a derivation, it should be possible to just swap them out no matter the language, as long as the resulting path has a binary with the same name. However, a simple shim with a single-line script that just calls the CLI with the required flags might be sufficient for this.Language to use
The language should make it easy for others to contribute, have decent error-handling capabilities and be well-suited to write a CLI in.
I am therefore immediately discarding bash (or any other shell language), perl, JS, TS and Haskell.
I see three main contenders: python, rust, and go. They are all well-known and have good general support. Rust is additionally loved in the nix community for its ethos of correctness, python is super easy to write, and go is kind-of a middle-ground between the two.
One thing to consider with this choice might be closure size. As disko is often run in installers, we can expect most dependencies to not be present yet (this is the singular advantage of bash; it is guaranteed to already be present). The closure sizes of the main executable vary quite a bit:
It seems somewhat unreasonable to me to require the user to download 1.2 gigabytes just to be able to run disko. A binary cache could alleviate this, though this has additional usability implications.
In the case of rust and go, there would also be compilation times added, though it's hard to say if this will matter much given the size of this project.
In general, I don't yet have a strong preference for which language to use.
The text was updated successfully, but these errors were encountered: