1 million is so 2011

Happy 2012 everyone!

A few months ago we published a blog post that talked about our servers doing 1 million tcp connections on a single box: http://blog.whatsapp.com/?p=170

Today we have an update for those keeping score at home: we are now able to easily push our systems to over 2 million tcp connections!!

jkb@c123$ sysctl kern.ipc.numopensockets
kern.ipc.numopensockets: 2277845

Best part is that we are able to do it with plenty of CPU and memory to spare and do it sustainably:

CPU: 37.9% user, 0.0% nice, 13.6% system, 6.6% interrupt, 41.9% idle
Mem: 35G Active, 14G Inact, 18G Wired, 4K Cache, 9838M Buf, 27G Free

This time we also wanted to share some more technical details with you about hardware, OS and software:

hw.machine: amd64
hw.model: Intel(R) Xeon(R) CPU X5675 @ 3.07GHz
hw.ncpu: 24
hw.physmem: 103062118400
hw.usermem: 100556451840

jkb@c123$ uname -rps
FreeBSD 8.2-STABLE amd64
jkb@c123$ cat /boot/loader.conf.local
boot_verbose=""
kern.hwpmc.nbuffers=32
kern.hwpmc.nsamples=64
kern.ipc.maxsockets=2400000
kern.maxfiles=3000000
kern.maxfilesperproc=2700000
kern.maxproc=16384
kern.timecounter.smp_tsc=1
net.inet.tcp.tcbhashsize=524288
net.inet.tcp.hostcache.hashsize=4096
net.inet.tcp.hostcache.cachelimit=131072
net.inet.tcp.hostcache.bucketlimit=120

and the last important piece of our infrastracture is Erlang:

8> erlang:system_info(system_version).
"Erlang R14B03 (erts-5.8.4) [source] [64-bit] [smp:24:24] [rq:24] [async-threads:0] [kernel-poll:false]\n"

P.S. - we are hiring in both client and server teams, so send your resume to jobs at whatsapp dot com if you are interested (.. and we are also looking for summer interns)