Question:

I am running the Sumo collector with the default heap memory allocation of 128MB, but the collector appears to use more memory than this. Why?

Answer:

The default Java heap size set for the collector is 128MB, however the total physical memory consumption of the JVM process actually consists of more than just the Java heap. This can increase the total memory used by the collector. For example, other things that the process includes could be:

  • The JVMs' own binaries 
  • Loaded libraries (including JAR and class files) 
  • Control structures for the Java heap 
  • Thread stacks

The memory settings for the Java virtual machine (Xms and Xmx) set the limits for the available heap for the user executable code within the JVM, and not necessarily for the overall memory footprint. The Xmx and Xms flags configure the class loader limits for how the users code. In this case, the collector code that tails logs and other things is managed within the JVM. 

The total physical memory used by the Collector can vary depending on the installed operating system, bit level of the JVM used, the amount of heap initially allocated to the Collector, and the number of Sources and log messages being collected. The total physical memory used by a default configured Collector can range from 180Mb to 500Mb.

Virtual Memory

In many cases you will also notice what appears to be a high amount of "virtual memory" being used by the collector process. This virtual memory is typically unused blocks on disk, which are currently still addressed to the Collector process. This is an exaggeration of the physical memory required at runtime (the memory initially required to get the JVM started). Once the process is running, this virtual memory is usually paged out, or released, since it is no longer required. Just after startup, it is common to see this amount hover from 400M to 500M.

For complete details on collector requirements see Installed Collector Requirements