pascal jungblut posts |

Mounting a Toy Nix Store with Snix

This text assumes minimal familiarity with Nix. If you ever wrote or adjusted a Nix configuration, that’s probably more than enough to get it.

TL;DR; We will rebuild the derivation {...} -> .drv -> $out CLI workflow with Snix in Rust to understand its implementation and to have a base for further experiments.

Nix can feel like magic the first time it runs — you write a bit of functional code, execute a command, and suddenly a reproducible package appears in /nix/store. This post pries open the hood.

The very core of that process is described in this “Nix Pill” (Our First Derivation). In this post I want to briefly go over a simplified version of that Nix Pill and then re-do it with the Snix project.

On a high level, the functional Nix programming language has only one side effect: it instantiates derivations to the Nix store. Say we wanted to write a file to the Nix store containing the string “Hello world”. In Nix that could look something like the following:

let
  message = "Hello world";
in
derivation {
  name = "my-message";
  system =  builtins.currentSystem;
  builder = "/bin/sh";
  args = [ "-c" "echo '${message}' > $out"];
}

If we write this to a file hello-world.nix and perform the instantiation with nix-instantiate hello-world.nix, it will output the path with the resulting store derivation. A store derivation is nothing more than a serialized form of instructions of how to build something. In our case the instructions would specify how “Hello world” should end up in a file in the Nix store. It consists of a “builder”, a program that is called to generate the output file(s), and its arguments args that are passed to the builder.

$ nix-instantiate hello-world.nix
/nix/store/zyyyxas4l14pagim19iwl478i455rr8q-my-message.drv

Aha! The derivation was actually written to the store, let’s inspect it with nix derivation /nix/store/zyyyxas4l14pagim19iwl478i455rr8q-my-message.drv. We only need to use this command because Nix uses a slightly obscure serialization format for these files, but in the end there’s nothing special about it:

{
  "/nix/store/zyyyxas4l14pagim19iwl478i455rr8q-my-message.drv": {
    "args": [
      "-c",
      "echo 'Hello world' > $out"
    ],
    "builder": "/bin/sh",
    "env": {
      "builder": "/bin/sh",
      "name": "my-message",
      "out": "/nix/store/8xyaxfx92n684ijx4iqnc5ang05d139n-my-message",
      "system": "x86_64-linux"
    },
    "inputDrvs": {},
    "inputSrcs": [],
    "name": "my-message",
    "outputs": {
      "out": {
        "path": "/nix/store/8xyaxfx92n684ijx4iqnc5ang05d139n-my-message"
      }
    },
    "system": "x86_64-linux"
  }
}

So the derivation already encodes its intended out path. How do we create the file (or directory) at the out path? A second and completely separate step is to take the derivations (build instructions) and to actually perform the build. After this step, we have the file with its “Hello world” content in the Nix store at out (/nix/store/8xyaxfx92n684ijx4iqnc5ang05d139n-my-message in this case), additional to the store derivation. Let’s do that:

$ nix-build  /nix/store/zyyyxas4l14pagim19iwl478i455rr8q-my-message.drv
this derivation will be built:
  /nix/store/zyyyxas4l14pagim19iwl478i455rr8q-my-message.drv
building '/nix/store/zyyyxas4l14pagim19iwl478i455rr8q-my-message.drv'...
/nix/store/8xyaxfx92n684ijx4iqnc5ang05d139n-my-message
$ cat /nix/store/8xyaxfx92n684ijx4iqnc5ang05d139n-my-message
Hello world

Cool! Now, we have successfully built our slightly simplified version of the Nix Pill and it is available at the store. The way this works is also shown in this ASCII-art representation of Nix’ architecture. These few steps illustrate fairly well how Nix works from the outside, but if you want to dig a bit deeper into the details, it can be a bit confusing (and to some degree it gets more and more, the deeper you look).

What if we could use a clean interface to execute each step that would allow us to inspect the state and process up close?

Re-implementing with Snix

Enter Snix. Snix is a relatively new rewrite of “Cppnix” (the original Nix) in Rust. To understand Nix and maybe even extend and experiment more with an implementation, Snix is a good candidate. It is modular, still relatively small and flexible. Every component can also be controlled via gRPC, but for this post I’d like to use it the project as a library. Let’s build everything we’ve seen so far (and more) into one program that uses Snix’ Rust API.

The granularity of these steps is not much finer than what we’ve seen with the command line interface, but using the API makes it straight forward to dig deeper to understand what goes on in each small step.

Let’s go over the architecture of Snix, which is different than Cppnix’. Snix’ documentation has a component overview that shows each of its parts, what it contains and how it relates to others. The main parts are:

  • snix-eval, a bytecode interpreter for the Nix language
  • snix-store, an implementation of the Nix store
  • snix-castore, a flexible underlying content-addressable binary storage (much like git’s underlying storage)
  • snix-build, an interface which allows us to implement to builders for our derivations that produce their outputs
  • snix-glue, literally glue code for the above

Re-implement the above workflow using Snix’ existing components is pretty straight forward. More so, it makes it easy to dig into the machinery and examine what exactly is happening in each step. Since the project is still so young and nicely split up into modules, each step is cleanly separated.

Before we can perform our first step of parsing, we need to create the underlying store and its components. Throughout the entire example, I’ll skip all error handling for brevity.

// Get a tokio runtime so we can easily interact with the async parts
let tokio_runtime = tokio::runtime::Runtime::new().expect("failed to setup tokio runtime");

// Let Snix create a set of default "services" for us
let (blob_service, directory_service, path_info_service, nar_calculation_service) =
    tokio_runtime
        .block_on(snix_store::utils::construct_services(
            snix_store::utils::ServiceUrlsMemory::parse_from(std::iter::empty::<&str>()),
        ))
        .unwrap();

// Create a build service that stores its files in /tmp
let build_service = tokio_runtime
    .block_on(snix_build::buildservice::from_addr(
        "oci:///tmp",
        blob_service.clone(),
        directory_service.clone(),
    ))
    .expect("Failed to create buildservice from address");

// Finally, bundle it all up into a SnixStoreIO wrapper
let snix_store_io = Rc::new(SnixStoreIO::new(
    blob_service.clone(),
    directory_service.clone(),
    path_info_service.clone(),
    nar_calculation_service.into(),
    Arc::from(build_service),
    tokio_runtime.handle().clone(),
    Vec::new(),
));

That seems like a bit of boilerplate to parse some code, but it also contains some setup for the later steps. The blob_service and directory_service are part of the snix_castore crate, so they deal with binary data. In this example, we don’t need to interact directly with them and it’s all handled by the SnixStoreIO wrapper instead. As the name suggests, blob_service deals with binary data and stores it in configurable backends, for example in-memory, files or S3. The directory_service gives structure to the data, because it can describe the directory and file structure of the data stored by the blob_service. The path_info_service is already part of the snix_store crate (rather than castore), so it is aware about some Nix specific details. For example, the PathInfo it produces hold Nix Archive (NAR) information and a StorePath, which represents a specific path in the Nix store, like /nix/store/8xyaxfx92n684ijx4iqnc5ang05d139n-my-message from above. All the services created above use the default memory-based backend, so they are not persisted. Obviously, this is only sufficient for our small example but they also require no setup (like paths) at all. It would also be possible to run the store in a separate process entirely.

With the boilerplate out of the way, we can parse the Nix code from above and let it be evaluated.

// The Nix code from above
let code = "let message = \"Hello world\";\
in (derivation {   name = \"my-message\";\
    system = \"x86_64-linux\";\
    builder = \"/bin/sh\";\
    args = [\"-c\" \"echo '${message}' > $out\"]; }\
).outPath";

// The evaluation builder is useful to extend the evaluation engine from its
// "empty" starting state
let mut evaluation_builder = snix_eval::Evaluation::builder(Box::new(SnixIO::new(
    snix_store_io.clone() as Rc<dyn EvalIO>,
)) as Box<dyn EvalIO>)
.enable_import()
.mode(EvalMode::Strict);

// Add the "derivation" builtin
evaluation_builder =
    snix_glue::builtins::add_derivation_builtins(evaluation_builder, Rc::clone(&snix_store_io));

// Perform the actual evaluation
let value = evaluation_builder.build().evaluate(code, None).value

Snix uses the builder pattern here to create an EvaluationBuilder. Using the builder, we can easily extend the vanilla builder. Here, we add the derivation builtin that will have the side-effect of adding the drv file to the store. Other builtins, for example “fetchers” or import are available as well. And you can, of course, easily add your own experimental ones. Next, we force the evaluation of the code. Since Nix is evaluated lazily (meaning the evaluator does not compute values it does not “need”), we have to force the computation of the outPath attribute by, well, evaluating it.

The result value is of type snix_eval::Value, which is an enum representing all possible Nix types (strings, attribute sets, …) and some internal types like unevaluated hunks. Since we know that value must be of type string, we can unwrap it and proceed further.

Remember the boilerplate from before? The snix_store_io that is passed to evaluation_builder allows Snix to perform the equivalent of nix-instantiate during the evaluation of the derivation builtin. Querying the directory_service can return the drv file from the derivation call. Additionally, the SnixStoreIO keeps track of the derivations it evaluates and more importantly it keeps a mapping from the drv file to their outPath. This makes the next step, building, trivial from a caller’s perspective.

One way to get snix_store_io to build the file is to perform some form of IO on it, for example by importing it or calling readFile on it. In fact, just checking for the existence of outFile will trigger the build in the background (and then return true).

if let Some(snix_eval::Value::String(nix_string)) = value {
    let out_path =
        std::path::Path::new(nix_string.as_str().expect("Nix string is not convertible"));
    let out_store_path =
        nix_compat::store_path::StorePath::from_absolute_path(nix_string.as_bytes())
            .expect("Could not build an absolute path from the out path");
    snix_store_io.path_exists(out_path).unwrap_or(false) // returns true
}

Indeed, this builds the output file and we will verify that in a second. Remember the build_service we created earlier? It’s part of the snix_store_io wrapper as well and that is what performed the build, because snix_store_io sent it a BuildRequest. The build service created an OCI image and called runc to execute it. The OCI image specification can be found at the path we specified for the OCI build service. Taking a quick glance at the config.json confirms that it’s indeed what we expect.

$ jq .process.args config.json
[
  "/bin/sh",
  "-c",
  "echo 'Hello world' > $out"
]

You can run runc manually on the OCI container and convince yourself that it produces the expected output (a file with “Hello world”). We can observe the result in multiple other ways, for example we could query the directory_service and reconstruct the file from its data in the blob_service. But there is a more direct way: the Nix store is typically available on the filesystem. Snix has us covered there as well. It contains a FUSE integration that allows us to mount the Nix store (described by the SnixStoreIO from above), ready for inspection.

let fuse_daemon = tokio_runtime
    .block_on(async move {
        let filesystem = snix_store::pathinfoservice::make_fs(
            blob_service,
            directory_service,
            path_info_service,
            true,
            true,
        );

        println!("Starting fuse, CTRL-C to umount");
        snix_castore::fs::fuse::FuseDaemon::new(
            filesystem,
            std::path::PathBuf::from("/mnt/"),
            2,
            true,
        )
    })
    .expect("Fuse daemon failed to start");

Again, omitting some logic for cleanly unmounting. This will use the services from our SnixStoreIO wrapper and mount them as a filesystem so that it can be accessed using ordinary file-based tools. You know, tree and cat:

$ tree /mnt/
/mnt/
└── 8xyaxfx92n684ijx4iqnc5ang05d139n-my-message

1 directory, 1 file
$ cat /mnt/8xyaxfx92n684ijx4iqnc5ang05d139n-my-message
Hello world

Just what we wanted! Most notably, it produced the exact same store path as calling the nix-build CLI call did - wonderful! You might, reasonably, have expected the .drv file to exist as well in the Nix store we just mounted. That is in fact a difference in the handling of .drv files: Snix does not persist these files, while Cppnix (the CLI from above) does - we even explicitly told it to create the files.

One detail that I omitted here is where this /bin/sh actually comes from. After all, isn’t the build supposed to be isolated and pure? The mounts of the OCI give a good hint:

$ jq .mounts[-1] config.json
{
  "destination": "/bin/sh",
  "type": "none",
  "source": "/home/pascal/tmp/busybox/bin/sh",
  "options": [
    "rbind",
    "ro"
  ]
}

It is mounted from the outside. What is mounted in – in my case – is a copy of the statically built busybox sh shell. This option is a compile-time setting when building snix-build. If you want to get your head around how to avoid this, have a look at bootstrapable builds (but conceptually it’s a hard problem). So this is a trade-off to reduce the scope of this example.

Want to try this yourself? I invite you to clone the repository with the example - just follow the instructions. I’m certainly hooked and will explore some more complex applications.