TOC

NIX package manager and NixOS

I stumbled upon Nix more or less by chance when I happened to take a look at what brew.sh had installed on my Mac. I was shocked to see that over 500 packages had been installed. I didn't even know what most of them were for (obviously, with that many).

Upon closer inspection, it turned out that many of these packages were dependencies or tools that I had tried once and then forgotten about 😉

I've been keeping my dot files in a Git repository for quite some time now. I also added an installation script that tries to set up the environment again on a new Mac. It works more or less. But not always without problems. It doesn't work at all on my Linux servers - of course, there's no brew.sh there. So I had to create an IF OS==Linux construct.

To be honest, that got annoying. I tried to use xxh (a tool that temporarily transfers the local environment to a target machine before opening a shell via SSH). But that was only of limited use.

But Nix is much more than just a package manager. The great thing about nix is the reproducibility of installations. Also, the encapsulation of environments. But let's take it step by step...

Warning

A word of warning: nix and nixOS have been in use for several years or even decades, but the documentation is really a problem. I had to rely on help from some forums, especially Reddit was helpful. But unfortunately, it's not as self-explanatory as one would wish.

So what I've put together here is certainly not 100% correct, but reflects my learning process. I figured out some things myself, some were kindly explained to me on Reddit, and some can also be found in the documentation (but honestly, not as much as you would think). That's also one of the reasons why I'm writing this - maybe someone still needs a little help using nix.

In addition, nix files must be written in a functional, domain-specific language. Which doesn't always make it easy to understand what's going on. And the error messages are anything but intuitive / understandable. For example, if you try to install a package that doesn't exist, this is the error message you get:

Getting from "undefined variable" to "the package does not exist" already requires a bit of brain power. However, you can also test this language on the fly, so to speak.

In the example above, I imported the variables that represent the individual software packages (that's why there was an "undefined variable" message above - packages/derivatives/flakes are only variables in this context). Then I built the Hello package :b. However, nothing happened because the sources didn't change, so only the path was displayed. With this REPL, you can also perform the installation in the current profile/environment.

This would now install the Gnu-Hello package in the current profile (similar to nix-env).

It gets complicated because nix is a functional, domain-specific language. And the domain here is the description of installations and software packages, as well as their dependencies. You can quickly notice this, for example, when calculating, or rather, as one is accustomed to from other languages.

You don't need to calculate anything, but only rarely, so it's not a big problem, just explain clearly where you might get "confused".

Furthermore, you quickly come across useful functions that you're not supposed to use yet because they're still pre-release. (e.g. nix command). This quickly leads to frustration because you keep hitting a wall (especially at the beginning).

I looked at it because it's a cool project with cool features. But it's definitely not for everyone.

The Nix Package Manager

First of all, the nix package manager is nothing more than a package manager - I want to install something, and it does that. In principle, nix can be used on two "levels": system-wide, i.e. it also manages system tools and utilities, or locally. The easiest way to see the system variant in action is in NixOS. Even there, all builds are reproducible, down to the kernel.

The local variant can be installed on any Linux, MacOS, or even in the "Unix shell" of Windows. This is a local package management similar to brew.sh.

But it goes much further than you might think. Because where is the advantage of simply running software locally - no problem, you might think. But it goes even further:

Nix not only installs the software locally, but also its dependencies and keeps a checksum of the binary dependencies! At first, this may not sound very spectacular, but it has some pretty cool implications:

For example, the ls command on OSX has the following dependencies:

If one of the linked libraries changes, for example through a system update, then ls might no longer work. This is not a problem for software that is part of the OS, of course. But it might be a problem for installed tools. That's where nix comes into play. Let's take a look at the dependencies of the tool exa that I installed via nix.

Especially the dynamic dependency on a libiconv is of interest - this was also installed by nix and is also managed by it. The path to this dynamic library contains the checksum of the binary file! This allows for multiple libiconv to be installed and linked at the same time. No overwriting of libs with unwanted side effects[^admittedly, this rarely happens in OSX. It is more common in Linux and especially Windows].

Let's take a look at this again under Linux:

Here the separation is even clearer, even the central libc is a version not managed by Nix!

Reproducibility

And this achieves reproducibility: the dependencies for an installed software are checked for identity binarily. This allows me to ensure that we have the identical binaries as dependencies. So, if I want to install exa, the LibC with the checksum ibp4camsx1mlllwzh32yyqcq2r2xsy1a will be installed if it is not already present. And this allows me to be certain that the build will succeed and the binary will work (because exa also has such a checksum!).

This can be taken further and the encapsulation of the software can be taken to the extreme. The installed packages are self-contained, Nix manages the dependencies and the binary versions.

nix-shell

All of this can then be used to temporarily install any software that is only available for a short period of time. The tool for this is nix-shell: it starts a new shell in which one or more newly installed programs are available.

This way you are also able to change to a specific version of a certain software:

The software mentioned is installed temporarily and is not accessible in the login shell. This applies to all dependencies required for the software as well. These dependencies are available only for the duration of the shell and are subsequently inaccessible.

Functionality

In essence, the nix software efficiently manages environment variables (PATH, LD_DYLD_PATH, LD_LIBRARY_PATH, etc.) and symbolic links to install each encapsulated software, even if it is dynamically linked. The process of garbage collection, which involves removing unused dependencies or packages, adds complexity to this functionality. nix provides tools specifically designed for this purpose.

• nix-store gc - initiates a garbage collection in the store, deleting all unused packages.
• nix-env --delete-generations old - allows removal of previous versions, termed "generations," in the current environment. This command facilitates version history management.
• nix-collect-garbage -d - removes other builds that are not present in the store or are located elsewhere, effectively eliminating any "garbage."

dir-env

The installation of packages can be automated and associated with a directory. Consequently, when switching to a project directory, all necessary tools for that specific project are automatically installed and made accessible in the shell.

In this case, when I switch to the directory of Stable Diffusion, a Python environment in the correct version is automatically provided. Also, all dependencies are installed [^since Python usually manages its dependencies itself via pip, a little trickery is required]. When I leave the directory, the original state is restored.

nix-shell vs. nix-env

In nix jargon, the user is in their "environment" (or env). And of course, this environment can also be "manipulated". With nix-env --install tree, the package tree is installed and made available in my current environment. This is also available beyond the lifespan of the current shell.

So, this is roughly equivalent to brew install tree on a Mac or apt-get install tree on Linux. That's nice (because of reproducibility and such), but it doesn't necessarily simplify the installation of a new system.

However, nix-env naturally offers all the functions you would expect: installing, uninstalling, listing installed packages, searching for installable packages, etc.

I would rather put the packages that I have permanently installed via nix-env into the home-manager configuration, so that I have a central description.

Nix home-manager

Now we come to the actual tool that started it all for me. I wanted a way to describe which tools I like to have on my system. And since that changes all the time, it would be nice to synchronize it across different machines via, for example, git. The Nix home-manager offers exactly that.

Actually, it's just a file ~/.config/home-manager/home.nix where you write what the home-manager should do. Various things can be managed with it:

• Installing software for the local nix-env
• Settings in zshrc / bashrc
• Setting environment variables
• Starship prompt settings
• Zsh plugins
• Offering certain files via nix (e.g. other config files or scripts)
• Configuration, especially under Linux, for KDE / Gnome, related tools, etc.
• and many more features

The exciting thing about it is again that nix is used as the basis for the home-manager and everything that the Home-manager installs. And that brings another feature with it, which can sometimes be useful:

Once you "install" the current configuration with home-manager switch, the existing environment is saved as a "generation". And I can switch to these "generations" as I please, i.e. if my installation is currently causing problems, I can switch to a previous generation that I know was still working. Great feature, especially on Linux.

Reproducibility vs. Repeatable

As mentioned above, nix tries to make the results of a software build reproducible. In detail, this means that no matter when I run the build, no matter what changes have been made to the dependencies, I always get the same result!

flake vs. nix

In general, a nix installation works just like you would expect from a package manager. So these dependencies are managed "internally".

In a nix file, I can then specify installations of packages, configurations, etc. But every time I run it, it would always choose the latest version of the packages I want.

With the help of so-called flakes, the versions are frozen in the current state.

A flake is basically a nix + current version numbers. In detail, not much more happens than a lockfile being created for the flakefile, listing the used checksums. And if you want to run the flake again, nix uses this lockfile to determine the correct versions.

This can also be used in home-manager or in dir-env. It also provides reproducibility in development - my development environment is always the same, even after switching computers, at home vs. at the office, etc.

In this context, the question arises of how to update the software...

nix flake update updates the flake in the current directory. This can also be used for the Home-manager:

after that, a home-manager switch will install all updates and makes them available.

My Nix Journey

It's been a few months since I came across Nix. In my IT bubble, Nix was occasionally suggested to me and I found it interesting, but I didn't take the time for it.

Eventually, I came across a comment that Nix can also replace brew.sh, or it was promoted as an alternative to brew and MacPorts.

And when I noticed the 500 installed packages on my system, I wanted to try it out.

But the start was a bit scary. The Nix installer wants to install something in /nix, a new filesystem in a way. I wasn't so sure about that, I find it quite dangerous. It even makes an entry for /nix in '/etc/fstab'...

At that point, I stopped and thought I would try it differently first...

NixOS in a VM

Nix can also be used system-wide, and that happens in NixOS. I installed it in a VM for testing. I wanted to see if the "effort" is worth it.

So, in the VM, I installed NixOS and familiarized myself with the concepts (which I described above). I would actually recommend that to anyone who wants to play around with Nix - try it out in a VM first...

I also set up NixOS as a development environment, including a graphical frontend KDE and Plasma, WezTerm, etc.

In that sense, NixOS doesn't really differ from Ubuntu, Debian, or Fedora. Actually, it is most comparable to ArchLinux because both distributions offer "rolling updates," which makes it easier to keep your system up to date.

With Nix, however, you have a few more options than with a "classic" approach: I can install a specific version of something without conflicting with existing software. For example, in the VM, I was able to install an Apache in a very old version that shouldn't actually run anymore (libSSL also had to be available in an old version).

I could play around a bit without putting my system at risk. When I tried all of this in another Linux distribution, I was sure that it would work on Mac too.

Nix package manager on OSX

The installation of the Nix Package Manager is relatively easy:

Then you will have to answer some questions (mostly with YES), and afterwards, everything is basically ready. The commands nix, nix-build, nix-channel, nix-env, and nix-store should now be available (possibly open a new shell!).

With that, you have actually already replaced brew. If I want to install something, I call nix-env --install PACKAGE, and if I want to delete something, it's nix-env --uninstall PACKAGE. With --upgrade, I can update everything or, if specified, only a specific package.

With nix-env --query, I get all the packages that have been installed in my environment... and so on.

However, searching for packages is currently a bit awkward. For that, you have to enable an option for "experimental features" when calling it 🙄 :

Of course, that's not practical, so you can put it in your nix-config.

Just for clarification: nix search nixpkgs searches the standard Nix packages nixpkgs. There are (theoretically) other collections that you could search as well.

The first call is really slow and takes a while until all package descriptions have been downloaded. You might be faster by simply searching directly on nixos.org.

home-manager

After all the prelude, we now come to the actual star of this post. The Nix home-manager is a software that tries to standardize and simplify the installation of a user directory along with all the required software packages and configurations.

Actually, for the Home Manager in its simplest form, you only need the nix package manager (mentioned above), and then you just call this line:

Now you need to create your home-manager configuration in ~/.config/ome-manager/home.nix, for example this here:

dir-env

dir-env is a really useful tool that, with the help of nix, automatically installs the necessary tools, software packages, etc. when switching to a directory.

dir-env simply needs to be installed. To do this, simply enable dir-env in the home-manager configuration.

theoretically, dir-env is working now.

But to get it really working on a specific directory, you need two things:

• there needs to be a file called .envrc in the directory. As I know, this only consists of one line, either use nix or use flake

• when using nix, you need to have a file called shell.nix that will be executed by nix when changing into this directory.

• when using flake, a file called flake.nix needs to be there. This is using flakes as described above. As mentioned, this will stick to the installed binary version.

  and, you'd probably get an errormessage the first time:

As mentioned in the error message, you just need to direnv allow to execute the nix.

The practical thing is that dir-env monitors the flake or nix file all the time, not just when you switch directories. So if I make a change to the file, it will be automatically loaded without having to leave and come back to the directory. Convenient...

WARNING: This is not nix-shell! When you exit or press CTRL-D, you are completely out (happens to me all the time 😉 )

dir-env for Software Developers

dir-env has become the most important tool in my setup, I think. Since I currently switch between a total of about 20 projects at work and privately, all of which have different requirements, it's a blessing!

• I have a project where I absolutely need JDK1.8 (yes, creepy, I know).
• Several projects use JDK11
• Others use JDK17
• And my blog software uses JDK20
• Some projects are in Rust
• Some others are in Python, and in different versions
• And some are a mix of Python and Java

Sure, you can also solve something like this with SDKMAN or similar. But I find the dir-env method to be cleaner.

And yes, if you use an IDE, the problem is also smaller. But I work a lot through the command line because I can get from A to B faster there.

My first Derivation

What was a derivation again? In the end, it's the description of what needs to be installed. The description of how a software is built and installed. These are the descriptions that are also in nixpkgs. And building such a derivation is essentially creating an installation guide for nix. It's comparable to building an apt package (deb) or rpm or something similar. Only, and that's the good thing, much simpler.

This is in a nutshell what such a derivative could look like. It is important here that the variable $out is the only thing that may be described during the install and build phase. Network traffic is also not allowed during the build.

And that brings us to the Java problem

Java Problems

Although this is not a real problem with Java, but rather with Maven. Maven wants to write the dependencies of the software to the local Maven repository. But this is not possible because nix prevents writing to other directories. In addition, the build process does not have access to the internet...

The solution is called "fixed derivative", i.e. a derivative that already defines the checksum in advance. Such a derivative must be created for the dependencies in Java and with it the checksum can be determined.

Deployment of a Java project

It took me a little time to figure this out, NIX is probably not used often by Java developers, or not used to deploy software. But packing the above into a file has provided me with a "new" way for deployment.

For the deployment of JBLOG2, here is the derivative:

Explanation: in the let section, a fixed derivate named deps is created. This is "built" by packing all dependencies for the project into the $out directory. All files that contain any timestamps are removed from there (otherwise the checksum would change with every run!).

And this should then result in the output hash. If there is a discrepancy between the defined hash and the calculated one, the build fails. Normally, this means that a dependency has been changed (added, removed, version number changed), or a "fuzzy" dependency has been defined in Maven (such as LATEST) and there is an update.

For this reason, all dependencies in the pom.xml should be precisely specified. Otherwise, it sometimes gets stuck, and you don't know why.

This dependency derivative is used in the build phase (where ${deps}/.m2 is located). There, the repository is copied to the local $out directory and specified as the mvn repo.

During installation, a wrapper script is created that can start the JAR file and copy the JAR file accordingly.

voila.

And with nix-build jblog.nix I can then build the version. it is then available in the "result" directory.

Or, and this is the best, you can start the project right after building it:

This compiles the code with the dependencies (both JDK and any other required tools) and starts the script in $out/bin.

Conclusion

Nix has been available for a long time and yet is not well-known enough. It can solve some problems in software development in a simple way. However, you need to have a bit of tinkering passion and the willingness to learn a new programming language.

Unfortunately, the documentation is not the best and you often read conflicting information. But once you have it up and running, you have really built something like "Docker ultralight" (and it is easier and more resource-efficient than Docker).

I highly recommend anyone who uses Linux, Unix, or MacOS and frequently uses the command line to take a look at nix. It is just as powerful as brew.sh, but offers more possibilities and even more packages...