January 06, 2012
Happy 2012 everyone!
A few months ago we published a blog post that talked about our servers doing 1 million tcp connections on a single box: http://blog.whatsapp.com/?p=170
Today we have an update for those keeping score at home: we are now able to easily push our systems to over 2 million tcp connections!
jkb@c123$ sysctl kern.ipc.numopensockets kern.ipc.numopensockets: 2277845
Best part is that we are able to do it with plenty of CPU and memory to spare and do it sustainably:
CPU: 37.9% user, 0.0% nice, 13.6% system, 6.6% interrupt, 41.9% idle Mem: 35G Active, 14G Inact, 18G Wired, 4K Cache, 9838M Buf, 27G Free
This time we also wanted to share some more technical details with you about hardware, OS and software:
hw.machine: amd64 hw.model: Intel(R) Xeon(R) CPU X5675 @ 3.07GHz hw.ncpu: 24 hw.physmem: 103062118400 hw.usermem: 100556451840
jkb@c123$ uname -rps FreeBSD 8.2-STABLE amd64 jkb@c123$ cat /boot/loader.conf.local boot_verbose="" kern.hwpmc.nbuffers=32 kern.hwpmc.nsamples=64 kern.ipc.maxsockets=2400000 kern.maxfiles=3000000 kern.maxfilesperproc=2700000 kern.maxproc=16384 kern.timecounter.smp_tsc=1 net.inet.tcp.tcbhashsize=524288 net.inet.tcp.hostcache.hashsize=4096 net.inet.tcp.hostcache.cachelimit=131072 net.inet.tcp.hostcache.bucketlimit=120
and the last important piece of our infrastracture is Erlang:
8> erlang:system_info(system_version). "Erlang R14B03 (erts-5.8.4) [source] [64-bit] [smp:24:24] [rq:24] [async-threads:0] [kernel-poll:false]\n"
P.S. - we are hiring in both client and server teams, so send your resume to jobs at whatsapp dot com if you are interested (.. and we are also looking for summer interns)