Unfortunately, ZFS is a memory hog. People says the ideal configuration for heavy disk uses is 8G for a really good reason. :-)

Of course the ZFS is marvelous, and all that memory is put for really good uses (definitively, it’s not a “lazy programming” issue) but in my humble opinion we should be able to choose to use the older UFS.

On a 2G system it could be a better choice.

(Yes, I know – back to that little English grammar I used some years ago)

I’m have used OpenSolaris Build 134 for some weeks by now, and faced similar situations, but different conclusions.

The OpenSolaris memory consumption is always near 100% because almost every kbyte of unused memory is being used by ZPF to cache disk access. When you start big applications, the system must “clear” the cache memory back to application use, causing some overhead. This is not a problem with 8G RAM systems, but on 4G (or, in my case, a Atom300 with 2G) it is.

Adjust ZFS to be less memory hungry and you’ll face less overhead on application startup – but at cost of more disk access.

See http://www.solarisinternals.com/wiki/index.php/ZFS_Evil_Tuning_Guide#Limiting_the_ARC_Cache for details.