As a bit of background, I've had a linux OS on a computer in my home since college, and most of the time its been my primary. My current OS is Ubuntu, and as they have a stripped down version intended for server use, its my go to for remote servers unless I have a reason to choose something else. Unfortunetly, one of my projects just found such a reason.

I still don't have all the details around the root cause, be I'll go over the initial symptom and some of the deeper ones. The project in question has grown to the point that it no longer makes sense to build it from my machine. The project is an Elixir application that uses Distillery to build releases, which are pushed to a server and run. I moved the build process from my machine to a CI service, and the image doing the building there is the official Elixir docker image. The transition was very smooth until I actually went to run the release on the server. The Erlang process immediately crashed, and the erl_crash.dump file included these snippets:

    args: [load_failed,"Failed to load NIF library
           ./lib/crypto-4.4/priv/lib/crypto: 'libcrypto.so.1.1:
           cannot open shared object file: No such file or directory'",
           "OpenSSL might not be installed on this system."]
    format: "Unable to load crypto library.

The initial round of searching online didn't reveal anything that just worked, so I took a step back and worked through what had changed. In tracing the Elixir docker image back several steps, the base OS was Debian. I had been building and still was deploying to Ubuntu...but those typically are close enough to not matter. A bit more digging, however, revealed that the default version of OpenSSL on Debain was 1.1.0j vs 1.0.2g on Ubuntu. I did not delve in further as changing the OS of the server to Debian resolved the issue.