CPU: Intel i5-4670. Of the new batch of Haswell chips, this seems to hit the sweet spot for price and performance. There are slightly faster ones, but they start to get much more expensive very fast after this point.

Motherboard: Asus Z87M-Plus uATX. There seems to be a lot less differentiation among motherboards nowadays, perhaps due to more stuff being moved right onto the CPU die. It’s mainly about size and number of USB and PCIe slots, and I don’t really need very many of them. I’m not going SLI, so I don’t need more than 1 PCIe 16x slot, so this board satisfies everything even though it’s on the cheaper end.

Case: SilverStone PS07B uATX. I could reuse my current Antec P180, but it’s a behemoth, and I’d like to try something a bit more compact. I don’t need four external 5 1/4″ bays! It’s my first time looking at uATX cases, so I’m slightly worried about things fitting and cooling, but as far as I can tell from measurements and other peoples’ reports, it should all work.

Video: Asus GTX 770 DirectCUII. Not much differentiation among video cards either, but this new generation should last me a good while and this card is supposed to be particularly cool and quiet. I’m slightly concerned that 2GB might not be enough if games ported from next-gen consoles need more and more texture memory, but it’s easy enough to replace the card down the road if that happens.

Storage: SanDisk Extreme 480GB SSD and Seagate Barracuda 3TB. It’s about time I got an SSD, and this one’s been recommended for being reliable and reasonably performant. It won’t be big enough by itself, so there’s ye olde regular hard drive in there too.

Power Supply: SeaSonic X650 Gold. Not a particularly glamorous component, but I’m going for a modular one to cut down on the mess of cables inside, especially with a smaller case.

I’m also trying to choose quieter parts this time. I didn’t really focus on it last time, and the fans are fairly noticeable even at idle.

Ugh, it’s not dead yet, but at 7 bad sectors and steadily climbing, causing random system lockups, it won’t be long now. This drive’s had a good life, lasting at least five or six years, and everything’s already backed up, so there’s no concern over losing anything.

It’s just really bad timing, since I was planning on putting together a new gaming system in a couple months or so, based on the upcoming Haswell chips and new GPU generation. If the drive dies tomorrow, there’s not much point in spending the time to immediately replace it and restore everything when I’m going to wipe and reinstall in a couple months anyway. But if I don’t, I’ll be left without my main gaming system for that couple months instead.

For now I’m just crossing my fingers and hoping it manages to limp along just long enough to assemble the new system. Come on and hurry up, Intel!

I wasn’t really using it much anyway, and it was just becoming more of a burden than a benefit. I never posted any status updates; nothing in my life really seems worth injecting into other people’s news streams. Reading other people’s updates was just a depressing inadequacy reminder. The constant stream of ads and game updates and ‘suggestions’ were annoying and nearly impossible to fully disable. The web page behaved poorly in Chrome, often chewing up huge amounts of CPU or crashing the tab. The privacy options were murky at best and what they track increasingly invasive. Perhaps what was the last straw was the update they tried to push to their Android client, that practically takes over the device and tracks what you do on it.

“But you’ll miss out on what all your friends are doing!”

I don’t need it.

Yeah, Facebook is great at efficiently distributing the news of your life to all the people you know, but I’ve been finding it increasingly alienating. Nobody communicates to me, it’s all indirect typing past each other. Telling Facebook about how your day went is not the same thing as telling me how your day went. Comments are not a replacement for conversation. If you want to talk to me, talk to me. If you don’t, that’s fine too, social interaction shouldn’t be forced.

And yeah, it’s a bit hypocritical in that this blog is the same kind of indirect interaction, but I’d like to think that the important differences are that I try to keep it to bigger and/or more niche topics, and stuff I’d rant about to no one in particular, not day-to-day stuff; that your decision to come here and read is voluntary, not obligatory just because you labeled someone a friend; and that I have total control here and am not trying to sell you something or track your reading habits.

Subject: They need you weak

Yeah, well, the cyber-samurai aren’t going to catch me off-guard that easily…

Subject: Postgraduate degree in economics is your dream? Get it right now over here.

Man, I don’t remember most of my dreams, but I hope they’re more exciting than that.

Subject: Get more swell down there

You’re going to help me make more friends down in the U.S.?

Subject: Check out my awesome racks

I would, but my keycard won’t let me in your server room.

Subject: Two simple steps,Add the title "Ph.D" to your resume

Hey, that’s only one step! I may not have a Ph.D, but I did take Counting 100.

Subject: Tarzan in bed after 1 doze.

He probably should have taken the NoDoz instead.

Doing research on routers is usually a big pain, but this time I went with some recommendations and got the ASUS RT-N66U. Besides just generally good reviews, it’s one of the models that works well with TomatoUSB, and I’ve had good luck with the Tomato firmware in the past, so the first thing I did was replace the standard firmware with that via these instructions. It’s a bit trickier with this model since you have to use an external utility to flash the firmware, so there’s a slightly higher chance of turning it into an expensive brick, but it went pretty smoothly.

Only time will tell as far as reliability goes, but it’s nice to finally be able to use the 5GHz band separately, as it’s much less congested around here. Being an apartment and condo-dense area, there’s a crazy number of APs around here, and the status page for the 2.4GHz radio says “Interference level: Severe”. I’ll also have to fiddle with the QoS and USB options at some point.

The build quality is pretty great, the screen is awesome, and no problems at all with the touch interface. The only real concerns were that the camera is pretty low-res and grainy in my living room light, but it’s clearly really just for Skype and such and people who try and take photos with tablets are evil anyway; and when holding it in landscape mode to watch a video, the sound distractingly comes from just one side, so have a good set of headphones instead. It’s a little bit heavier than expected, and my arm got a bit tired of holding it up, but that’s really just because I’m not exactly in great shape…

I checked out some comics, and although cramped, I still found them to be reasonably legible at the full-page-fit level. That was admittedly with some DC/Marvel-style bright, large-text pages though, and I can see comics with denser art and text needing more fiddling with zooming in and out. I’d say it’s ‘adequate’, obviously not as good as a full-size tablet but I’d certainly rather view them on this than my iPhone.

Google Play seems nice enough, though I haven’t really probed the depth of their selections yet. The major omission in Canada is the music store and sync/streaming service, so for us it’s just not going to be a very good music device unless you set up your own DLNA server and use something like Subsonic to do your own streaming. The Google integration is nice in that I recently switched to Chrome on the desktop, so it’s convenient already having bookmarks and such shared.

And I’m still exploring the whole Android ecosystem, but it does please the geek inside me that I can get fairly low-level with terminals, file management, etc. even without having rooted it. The app selection may be smaller, but it hasn’t really felt like anything’s missing yet (hell, I even found a .mod music player). I still have to check out the gaming selection in more depth.

I still do way too much fiddly stuff on my laptop for this to ever be a replacement for it, but I’ll just have to keep using it and see what kind of role it naturally falls into. It’s definitely way easier to use than the laptop while lying in bed…

(Though one thing that really bugged me while setting it up is that I had to log in to my Google account four times, typing out my whole crazy-long strong password each time. The first time fails because of two-factor authentication, so it redirects you to a web page to sign in again and enter the mobile code, but then I misclicked something and it took me to another web page with no obvious navigation or gesture controls to get back to the code entry. I wound up having to hard power it off and go through the whole setup process again, requiring another two password prompts. At least it was only a one-time process… The paranoid side of me also isn’t crazy about having another way essential credentials for things like Google could be leaked, but hopefully having a strong PIN on it suffices.)

Well, it finally did it. I’ve been a staunch Firefox user ever since it was in beta, since I liked using the same browser across multiple platforms, but it’s gotten to the point where it’s just too glitchy to tolerate. Pages not responding to the refresh button properly, unexplained choppiness in video playback, the tab bar not scrolling all the way and leaving some tabs accessible only by the tab list, crashes when I use the Flash player full-screen, the tab list missing some tabs at the bottom, high CPU usage on OS X even when lightly loaded, the Firefox window suddenly losing focus so I have to click on it again, not remembering passwords on some pages it should, and so on. Little things, but they add up.

So, I’m going to give Chrome a try for a while. There’s no guarantee that I won’t find a bunch of things about it that’ll annoy me just as much, but it’s worth a shot. Now I just have to find a set of equivalent extensions…

Some experimentation showed that having these different ranges was indeed more efficient than having just a single offset length, but it was hard to tell just what the optimal sizes were for each range. I kept it to only three different ranges because initially I didn’t want the number of identifier symbols to be too large, but after merging the identifiers into the literals, I had a bit more leeway in how many more ranges I could add.

So…why not add a range for every bit length? I set it up so that 256 would correspond to a 6-bit offset, 257 indicated a 7-bit offset, 258 is an 8-bit offset, etc., all the way up to 24-bit offsets. This also had the property that, except for the bottom range, an ‘n’-bit offset could be stored in ‘n-1′ bits, since the uppermost bit would always be ’1′ and could be thrown away (if it was ’0′, it wouldn’t be considered an ‘n’-bit offset, since it fits in a smaller range). Some testing against a set of data files showed that this did indeed improve the compression efficiency and produced smaller files.

With all of these possible bit values and lengths though, there was still the open question of what should be considered reasonable values for things like the default history buffer size and match length. Unfortunately, the answer is that it…depends. I used a shell script called ‘explode’ to run files through the compressor with all possible combinations of a set of buffer sizes and match lengths to see which would produce the smallest files, and the results varied a lot depending on the type and size of input file. Increasing the match length did not necessarily help, since it increased the average size of the length symbols and didn’t necessarily find enough long matches to cancel that out. Increasing the buffer size generally improves compression, but greatly increases memory usage and slows down compression. After some more experimentation with the ‘explode’ script, I settled on defaults of 17 bits for the buffer size, and a match length of 130.

Another idea I’d remembered hearing about was how the best match at the current byte might not necessarily be the most efficient match. It might be more efficient to emit the current byte as a literal instead if the next byte is the start of an even longer match. It was only an intuitive feeling though, so I implemented this and tested it and it did indeed seem to give a consistent improvement in compression efficiency. As an example, in one text document the phrase ‘edge of the dock’ was compressed like so:

Literal: 'e' (101) (4 bits)
Literal: 'd' (100) (6 bits)
Literal: 'g' (103) (8 bits)
10-bit offset: 544   Length: 3 'e o' (16 bits)
 8-bit offset: 170   Length: 6 'f the ' (17 bits)
10-bit offset: 592   Length: 3 ' do' (16 bits)
Literal: 'c' (99) (6 bits)
Literal: 'k' (107) (7 bits)

but with the new test, it generated the following instead:

Literal: 'e' (101) (4 bits)
Literal: 'd' (100) (6 bits)
Literal: 'g' (103) (8 bits)
Literal: 'e' (101) (4 bits) (forced, match len=3)
 8-bit offset: 170   Length: 8 ' of the ' (19 bits)
10-bit offset: 592   Length: 3 ' do' (16 bits)
Literal: 'c' (99) (6 bits)
Literal: 'k' (107) (7 bits)

The ‘forced’ literal normally would have been part of the first match, but by emitting it as a literal instead it was able to find a more efficient match and only two offset/length tokens were needed instead of three, for a difference of 80 bits for the original versus 70 bits for the improved match. Doing these extra tests does slow down compression a fair bit though, so I made it an optional feature, enabled on the command line.

At this point though, it’s getting harder and harder to extract gains in compression efficiency, as it starts devolving into a whole bunch of special cases. For example, increasing the buffer size sometimes makes compression worse, as in the following example:

'diff' output between two runs:
 17-bit offset: 87005   Length: 10 'with the t' (26 bits)
 14-bit offset: 10812   Length: 3 'arp' (18 bits)
-13-bit offset: 7705   Length: 3 ', w' (17 bits)
-13-bit offset: 5544   Length: 8 'ould you' (19 bits)
+18-bit offset: 131750   Length: 4 ', wo' (41 bits)
+13-bit offset: 5544   Length: 7 'uld you' (19 bits)
 16-bit offset: 50860   Length: 7 '?  You ' (22 bits)
 17-bit offset: 73350   Length: 10 'take that ' (26 bits)

The compressor looks for the longest matches, and in the ‘+’ run it found a longer match, but at a larger offset than in the ‘-’ run. In this case, 18-bit offsets are rare enough that their symbol has been pushed low in the Huffman tree and the bitstring is very long, making it even less efficient to use a long offset, and in the end a whopping 24 bits are completely wasted. Detecting these kinds of cases requires a bunch of extra tests though, and this is just one example.

So, I think that’s about all I’m going to do for attempting to improve the compression efficiency. How does it do overall? Well, that 195kB text file that originally compressed to 87.4kB and then made it down to 84.2kB can now be compressed down, with harder searching on and optimal buffer and match length sizes determined, to 77.9kB. That’s even lower than ‘gzip -9′ at 81.1kB!

It’s not all good news, though. If I take the Canterbury Corpus and test against it, the original total size is 2810784 bytes, ‘gzip -9′ reduces them to a total of 730732 bytes (26.0%), and at the default settings, my compressor gets…785421 bytes (27.9%). If I enable the extra searching and find optimal compression parameters for each file via ‘explode’, I can get it down to 719246 bytes (25.6%), but that takes a lot of effort. Otherwise, at the default settings, some of the files are smaller than gzip and others are larger; typically I do worse on the smaller files where there hasn’t really been much of a chance for the Huffman trees to adapt yet, and the Excel spreadsheet in particular does really poorly with my compressor, for some reason I’d have to investigate further.

But I’m not going to. No, the main remaining problem was one of speed…

Unfortunately, that method led to horribly imbalanced trees, since it only considered nearby nodes when rebalancing, when changing the frequency of a symbol can actually affect the relationship of symbols on relatively distant parts of the tree as well. As an example, here’s what a 4-bit length tree wound up looking like with my original adaptive method:

Lengths tree:
    Leaf node 0: Count=2256 BitString=1
    Leaf node 1: Count=1731 BitString=001
    Leaf node 2: Count=1268 BitString=0001
    Leaf node 3: Count=853 BitString=00001
    Leaf node 4: Count=576 BitString=000001
    Leaf node 5: Count=405 BitString=0000001
    Leaf node 6: Count=313 BitString=00000001
    Leaf node 7: Count=215 BitString=000000000
    Leaf node 8: Count=108 BitString=0000000011
    Leaf node 9: Count=81 BitString=00000000101
    Leaf node 10: Count=47 BitString=000000001001
    Leaf node 11: Count=22 BitString=00000000100001
    Leaf node 12: Count=28 BitString=0000000010001
    Leaf node 13: Count=15 BitString=000000001000000
    Leaf node 14: Count=9 BitString=000000001000001
    Leaf node 15: Count=169 BitString=01
    Avg bits per symbol = 3.881052

If you take the same data and manually construct a Huffman tree the proper way, you get a much more balanced tree without the ludicrously long strings:

    Leaf node 0: Count=2256 BitString=10
    Leaf node 1: Count=1731 BitString=01
    Leaf node 2: Count=1268 BitString=111
    Leaf node 3: Count=853 BitString=001
    Leaf node 4: Count=576 BitString=1100
    Leaf node 5: Count=405 BitString=0001
    Leaf node 6: Count=313 BitString=11011
    Leaf node 7: Count=215 BitString=00001
    Leaf node 8: Count=108 BitString=000001
    Leaf node 9: Count=81 BitString=000000
    Leaf node 10: Count=47 BitString=1101000
    Leaf node 11: Count=22 BitString=110100110
    Leaf node 12: Count=28 BitString=11010010
    Leaf node 13: Count=15 BitString=1101001111
    Leaf node 14: Count=9 BitString=1101001110
    Leaf node 15: Count=169 BitString=110101
    Avg bits per symbol = 2.969368

That’s nearly a bit per symbol better, which may not sound like much but with the original method there was barely any compression happening at all, whereas a proper tree achieves just over 25% compression.

So, I simply dumped my original adaptive method and made it construct a Huffman tree in the more traditional way, pairing the highest count nodes in a sorted list. To keep it adaptive, it still does the count check against the parent’s parent’s other child, and when it crosses the threshold it simply rebuilds the entire Huffman tree from scratch based on the current symbol counts. This involves a lot more CPU work, but as we’ll see later, performance bottlenecks aren’t necessarily where you think they are…

My trees also differ from traditional ones in that they prepopulate the tree with all possible symbols with a count of zero, whereas usually you only insert nodes into a Huffman tree if they have a count greater than zero. This is slightly suboptimal, but it avoids a chicken-and-egg problem with the decoder not knowing what symbol a bitstring corresponds to if it doesn’t exist in the tree yet because it’s the first time the symbol has been seen.

Knowing that, and with the improved Huffman trees, another thing became clear: using Huffman trees for the offsets wasn’t really doing much good at all. With most files, the offset values are too evenly distributed, and many are never used at all, and all those zero-count entries would get pushed down the tree and become longer strings, so the first time an offset got used it would often have a string longer than its basic bit length, causing file growth instead of compression. I instead just ripped those trees out and emitted plain old integer values for the offsets.

The way I was constructing my trees also had another limitation: the total number of symbols had to be a power of two. With the proper construction method, an arbitrary number of symbols could be specified, and that allowed another potential optimization: merging the identifier tree and the literals tree. The identifier token in the output stream guaranteed that there would always be at least 1 wasted non-data bit per token, and often two. Merging it with the literals would increase the size of literal symbols, but the expectation is that the larger literal size would on average still be smaller than the sum of the identifier symbols and smaller literal symbols, on average, especially as more ‘special case’ symbols are added. Instead of reading an identifier symbol and deciding what to do based on that, the decoder would read a ‘literal’ symbol, and if it was in the range 0-255, it was indeed a literal byte value and interpreted that way, but if it was 256 or above, it would be treated as having a following offset/length pair.

The range of offsets to handle would also have to change, but that’s for next time… With the Huffman tree improvements, my 195kB test file that compressed to 87.4kB before now compressed to 84.2kB. Still not as good as gzip, but getting there.

Looking at it now it’s kind of embarrassing, since of course it’s riddled with inexperience. No comments at all, leaving me mystified at what some of the code was even doing in the first place, unnecessary global variables, little error checking, poor header/module separation, unnecessary exposure of class internals, poor const correctness, and so on. It kind of irks my pride to leave something in such a poor state though, so I quickly resolved to at least clean it up a bit.

Of course, I have to understand it first, and I started to remember more about it as I looked over the code. It’s a fairly basic combination of both LZ77 pattern matching and Huffman coding, like the ubiquitous Zip method, but the twist I wanted to explore was in making the Huffman trees adaptive, so that the symbols would shift around the tree to automatically adjust as their frequency changed within the input stream. There were two parameters that controlled compression efficiency: history buffer size, and maximum pattern length. The history size controls how far back it would look for matching patterns, and the length controlled the upper limit on the length of a match that could be found.

Compression proceeded by moving through the input file byte by byte, looking for the longest possible exact byte match between the data ahead of the current position and the data in the history buffer just behind the current position. If a match could not be found, it would emit the current byte as a literal and move one byte ahead, and if a match was found, it would emit a token with the offset and length of the match in the history buffer. To differentiate between these cases, it would first emit an ‘identifier’ token with one of four possible values: one for a literal, which would then be followed by the 8-bit value of the literal, and three for offset and length values, with three different possible bit lengths for the offset so that closer matches took fewer bits. Only matches of length 3 or longer are considered, since two-byte matches would likely have an identifier+offset+length string longer than just emitting the two bytes as literals. In summary, the four possible types of bit strings you’d see in the output were:

    | ident 0 | 8-bit literal |

    | ident 1 | 'x'-bit offset    | length |

    | ident 2 | 'y'-bit offset        | length |

    | ident 3 | 'z'-bit offset            | length |

And then I used a lot of Huffman trees. Each of these values were then run through a Huffman tree to generate the actual symbol emitted to the output stream, with separate trees for the identifier token, the literals, the lengths, and the three offset types. HUFFMAN TREES EVERYWHERE! The compression parameters were also written to a header in the file, so the decoder would know what history buffer size to use and maximum length allowed.

It worked…okay… I’ve lost my original test files, but on one example text file of 195kB, my method compresses it down to 87.4kB, while ‘gzip -9′ manages 81.1kB. Not really competitive, but not too bad for a completely amateur attempt either. There’s still plenty of room for improvement, which will come…next time.