munmap() is slow on Linux, and it is linear from the chunk size. I have known about it for a while, but haven’t got a chance to investigate why.

Anyone knows the reason? The problem is so serious that when working with large memory volumes Tarantool has to fork off a separate thread just to do munmap(), to avoid serious latency issues in the connection thread.

This morning I got a chance to actually write a test program and measure the impact. An interesting result is that it gets relatively faster with a larger chunk.

kostja@atlas:~$ ./a.out 
   size  	mmap time	munmap time	t1/t2   	t2/size
   1048576	0.000006	0.000281	46.833333	0.026798
   2097152	0.000008	0.000388	48.500000	0.018501
   4194304	0.000006	0.001230	205.000000	0.029325
   8388608	0.000008	0.000680	85.000000	0.008106
  16777216	0.000002	0.001550	775.000000	0.009239
  33554432	0.000002	0.002711	1355.500000	0.008079
  67108864	0.000006	0.005897	982.833333	0.008787
 134217728	0.000007	0.010866	1552.285714	0.008096
 268435456	0.000004	0.020137	5034.250000	0.007502
 536870912	0.000005	0.040046	8009.200000	0.007459
1073741824	0.000011	0.067541	6140.090909	0.006290

PS If you have Solaris/Darwin/FreeBSD at hand, I would love if you repeat the program and paste your numbers.