FreeBSD Documentation – 6.12 Tuning Disks:
6.12.1.2 vfs.write_behindThe vfs.write_behind sysctl variable defaults to 1 (on). This tells the file system to issue media writes as full clusters are collected, which typically occurs when writing large sequential files. The idea is to avoid saturating the buffer cache with dirty buffers when it would not benefit I/O performance. However, this may stall processes and under certain circumstances you may wish to turn it off.
I have been seeing some weird performance issues: slowdowns and high loads with the busy processes in the “inode” run state, all of which are httpd processes (load of 18 on a single CPU machine and 4-5 process all splitting 100% of system time). From my reading of this, I guess there is some contention for disk writes (ie, logging). I toggled this sysctl off and hey, presto, loads dropped like a rock. A symptomatic fix, but it will do for now.