RIP my life with tmem (Jan 2020 - Jan 2020)

Life can truly be comically tragic sometimes. I experienced that this past week and thought I should write about it, share with you this rollercoaster adventure (not really, I’m just being dramatic). Before we go on, we must first introduce the subject of our sad tale.

tmem, or transcendent memory was a feature that was introduced in the Linux kernel sometime back in the days of the dinosaurs i.e. June 2009. The goal of tmem was to provide a way to set aside a separate pool of memory outside of direct kernel control. The contents of the chunk of memory would be completely opaque to the kernel and could only accessed by the kernel via put_page and get_page calls to tmem. tmem dealt with data at page granularity to keep with existing understood page conventions. A tmem frontend would be utilized by core kernel code to send pages to the tmem backend via a ‘put’ call and later retrieve it via a ‘get’ call. Each page would be associated with an id using which it could be later retrieved. The backend would have complete control on how the pages would be organized in the pool of memory that it was set up with. As a result, it could do useful things like deduplicating pages, compressing the pages to save space, sending a pages to a different machine that had extra RAM space, all sorts of stuff. The backend would usually be implemented as a driver. It was a very neat concept and this is just a very surface level description. I highly recommend reading this LWN article for a well written overview of tmem and its benefits.

I stumbled upon the concept of tmem when I was looking into memory compression in Linux for my graduate research. I was (still am) trying to find a way of applying memory compression for containers and wanted to find a way to trigger memory compression from userspace. Typically, in Linux, when the OS sees that memory is reaching its limit, it will start sending pages into the swap space. The problem here is that sending to swap has a great amount of overhead because you are reading from and writing to disk. This can slow down applications when your memory is full because it is constantly reading and writing to swap. To try and mitigate this, you can set up one of a couple of memory compression solutions that will serve as a in memory cache that can hold any pages you want to swap out so that you don’t pay the penalty of disk IO when you need that page back. This allows you to get more effective memory for a small amount of overhead of the compression/decompression. The problem for me was, this is only done in association with the swap subsystem. The kernel makes all the decisions. You can’t control which pages should be sent for compression and retrieved.

This LWN article was the one I read that described all this memory compression stuff to me. It was also where I was introduced to the general idea of transcendent memory. I looked into it a bit more (i.e. this and this. And though the existing memory compression tools I read about didn’t really fit what I needed, I thought tmem and its base implementation was exactly what I needed. If I could perhaps write something in the kernel that would utilize tmem and allow me to trigger memory compression in userspace, I’d be rich! (Well maybe not that. But it would get me a project for my thesis). I was really enamored with the idea, never mind the fact I didn’t know the first thing about kernel hacking but I figured that would be something I can deal with later. But the problem was, all I had was a vague idea. How would I make a concrete implementation? I figured I would maybe find some answers by looking at the API and the kernel code for frontswap and cleancache.

If you had been following along and reading all the articles I’ve been linking, maybe you may have noticed my problem and why this is ultimately a tragic tale. The clue is in the dates of all the articles. They all date back from around ten years ago. Millenia in internet years. Transcendent memory is a really neat idea but in the intervening years it has not seen much use for more ideas beyond the memory compression stuff I described earlier. And so it fell to the wayside. I was first clued in to the fact that something was off when I couldn’t find any calls to the tmem API anywhere when searching through the Linux source code. The only references to tmem were in the frontswap code (in mm/frontswap.c) but there were no calls to any tmem API, no tmem related header files being included. I couldn’t find any files where tmem was defined, no header files, nothing. This seemed really odd. What was going on?

I figured maybe searching through LWN might help, maybe give me some clue as to what is going on. And the first result I find was this.

As it turns out, tmem was on its way out. The kernel folks had decided it was not really something that was worth maintaining and building on, and the original contributor had left the community as well. It was time for some spring cleaning. There’s some more detail on what’s being removed and why here but basically they are removing the tmem API for good but leaving behind frontswap as its own thing separate from tmem. The unused tmem stuff is getting the boot. So using tmem is not really a viable option because its not in use anywhere and is going the way of the dodo. Sigh. So here we are. My hopes and dreams dashed (they were vague and undefined hopes and dreams that I wasn’t strongly attached to so it’s not like it’s a big loss). I’m going to need to start looking at other options and hopefully find what I’m looking for.