Compressing pcap files

Here at Sinodun Towers, we’re often dealing with pcap DNS traffic capture files created on a far distant server. These files need to be compressed, both to save space on the server, and also to speed transfer to our machines.

Traditionally we’ve used gzip for quick but basic compression, and xz when space was more important than CPU and we really needed the best compression we could get. At a recent conference, though, an enthusiastic Facebook employee suggested we take a look at zstd. We’d looked at it quite some time ago, but our contact said it has improved considerably recently. So we thought we’d compare the latest zstd (1.2.0) with current gzip (1.8) and xz (5.2.3) and see how they stack up when you’re dealing with pcap DNS traffic captures.

Compressing

We took what is (for us) a big file, a 662Mb DNS traffic capture, and timed compressing it at all the different compression levels offered by each compressor. We did three timed runs for each and averaged the time. Here’s the results. Each point on the graph is a compression level.

zstd turns in an impressive performance. For lower compression levels it’s both notably quicker than gzip and far more effective at compressing pcap DNS traffic captures. In the same time gzip can compress the input pcap to 25% of its original size, zstd manages 10% of the original size. Put another way, in our test the compressed file size is 173Mb for gzip versus 65Mb for zstd at similar runtimes.

zstd is also competitive with xz at higher compression levels, though xz does retain a slight lead in file size and runtime at higher compression levels.

Decompressing

Of course, being able to compress is only half the problem. If you’re collecting data from a fleet of servers and bringing that data back to a central system for analysis, you may well find that decompressing your files becomes your main bottleneck. So we also checked decompression times.

There’s little to choose between zstd and gzip at any compression level, while xz generally lags.

Resource usage

So, if zstd gives better compression in similar times, what other costs does it have over gzip? The short answer there is memory. Our measurements show that while gzip has much the same working set size regardless of compression level, zstd working sets begin an order of magnitude larger and increases; by the time zstd is competing with xz, its working set size is up to nearly 3x the size of xz.

That being said, by modern standards gzip‘s working set size is absolutely tiny, comparable to a simple ls command. You can very probably afford to use zstd. As ever with resource usage, you pays your money and you takes your choice.

Conclusion

It looks to us that if you’re currently using gzip to compress pcap DNS traffic captures, then you should definitely look at switching to zstd. If you are going for higher compression, and currently using xz, the choice is less clear-cut, and depends on what compression level you are using.

A note on the comparison

We generated the above numbers using the standard command line tools for each compressor. Some capture tools like to build compression into their data pipeline, typically by passing raw data through a compression library before writing out. While attractive for some use cases, we’ve found that for higher compression you risk having the compression becoming the processing bottleneck. If server I/O load is not an issue (which it is not for many dedicated DNS servers), we prefer to write temporary uncompressed files and compress these once they are complete. Given sufficient cores, this allows you to parallelise compression, and employ much more expensive – but effective – compression than would be possible with inline compression.

Comparing drill output

I wanted to compare the output of multiple drill queries. After a bit of trial error, I came up with this sed to remove the parts of drlll’s output that change from query to query.

drill www.sinodun.com | grep -v "Query time" | grep -v WHEN 
        | sed 's/t[[:digit:]]{1,}t|,sid.{1,}$//'

Juniper SRX mucking with DNS

I was getting some strange DNS answers on the servers in a trust zone on my SRX. All the servers are statically NAT’d to external IP’s and run their own caching resolvers but when I tried to query for the servers A RR I kept getting the internal IP address. No name server either internal or external was serving that A RR. Eventually I realised that it was the SRX changing the answer section of the DNS response. I don’t know if it is on by default, or if I switched it on by mistake but it was the DNS Application Layer Gateway (ALG) trying to help by making use of what it knew about the static NATs. Switching DNS ALG off solved the issue. For a detailed description of what the ALG does see the Junos OS Security Configuration Guide.

DNS Spoofing

At IETF 72 in Dublin I gave a demonstration of DNS spoofing based on the attack on DNS described by Dan Kaminsky. I was able to successfully inject a fake DNS record in to the cache of a name server with a fixed port in a few seconds and sometimes in well under a second.

Bert Hubert published a description of the math behind this attack on namedroppers and I have been playing with the spoofer to see how close I can get the experiment and theory.

I ran my spoofer on a network consisting of three machines linked by a cheap gigabit switch. The attacker was on a Mac Pro, the target nameserver was on a Mac Book Pro and the authoritative server, that the attacker was pretending to be, was on a old FreeBSD box (100Mb). I used DUMMYNET to simulate a longer link to the authoritative server (delay = 30ms).

I ran the spoofer 1000 times and plotted a histogram of the frequency of success against time.

The pink bar shows the median of all the times recorded. If I recall my A level maths correctly, this should coincide with the 50% chance of success predicted by the math.

The math presented by Bert Hubert considers the expansion of the binomial

Ps = probability of success on a single attempt

Pf = probability of failure on a single attempt

( Ps + Pf )^n = 1

Expanding this and remembering that the sum of all the terms containing success = (1- the term for always failing) leads to the probability of combined success

Pcs(n) = 1 – (1 – Ps)^(n)

We know that n = T/W so we get

Pcs(t) = 1 – (1 – Ps)^(t/W)

Bert Hubert tells us that Ps = (D*R*W)/(N*P*I) where

I: Number distinct IDs available (maximum 65536)

P: Number of ports used (maximum around 64000 as ports under 1024
are not always available, but often 1)

N: Number of authoritative nameservers for a domain (averages
around 2.5)

R: Number of packets sent per second by the attacker

W: Window of opportunity, in seconds.  Bounded by the response
time of the authoritative servers (often 0.1s)

D: Average number of identical outstanding queries of a resolver
(typically 1, see Section 5)

I used the following values
I=65535
R=36000 – From looking at the traffic I was sending
W=0.030 – From the settings I gave DUMMYNET
N=1.0 (I fixed this)
P=1 (I fixed this)
D=1
Plotting this on the same graph as the histogram gives:

The blue circles are the predicted probability of combined success (Their y axis runs from 0 to 1 and is not shown). As you can see the predicted 50% chance (black cross lines) occurs slightly before the median but it is fairly close.

In order to improve things I added an extra term to the equation to account for the time that the window is closed (This is due to the spoofer taking a bit of time to notice that it has been unsuccessful and to try again). So:

n = T/(W+Wc)

Ps = (D*R*W))/(N*P*I)

where Wc is measured to be about 0.003 seconds. The graph now looks like

That seems like good agreement to me. The median in this case is 1386ms.

BTW: The graphs were plotted using R.  This is the code I used

#Plot a histogram of frequency of success against time
mydata <- read.table("/tmp/speed-test-30ms",header=TRUE)
#Plot both on a single graph
h <- hist(mydata$time,breaks=100,plot=FALSE)
plot(h,freq=FALSE, xlim=range(h$mids),ylim=range(h$density),
    sub="Histogram showing time to success of real spoofer (pink line shows median)",
    main="DNS Spoofer Performance",
    ylab="Density", xlab="Time/ms")
abline(v=median(mydata$time),col=70)
#Plot Bert Hubert's math
D=1.0
R=36000.0
W=0.030
Wc=0.003
N=1.0
P=1
I=65535
Ps <- ((D*R*W)/(N*P*I))
Pcs <- function(t){1 - (1 - Ps)**((t/1000)/(W+Wc))}
par(new=TRUE)
nx <- sample(h$mids)
y=Pcs(nx)
#Scale plot to same as histogram
my=max(y)
ny=y*max(h$density)/my
plot(nx,ny, xlim=range(h$mids),ylim=range(h$density),col="blue", ann=FALSE)
#Calculate time for 0.5 chance
time5 = 1000*(W+Wc)*(log10(0.5)/log10(1 - Ps))
abline(v=time5)
abline(h=0.5*max(h$density)/my)