Java Garbage Collection in a nutshell

[jogger]

Do you think your Java app like i2p sticks to the specified heap size and garbage collection occurs when this heap is filled with unused objects to some degree? Then you need to read on.

A short look at the physical memory occupied by i2p reveals that it allocates much more than the specified heap size visible to i2p and set in wrapper.config.

Basic facts:
- All garbage collectors have a stop-the-world pause that locks everything. Some collectors do some work like object marking before concurrently. The length of the stop-the-world pause statistically adds to all timer values.
- In old times - when Java was slow and throughput a top priority - the goal for this pause was set at 200ms. This goal is the default in all collectors until today (Java 11).
- Today with fast processors the JVM will accumulate lots of unused memory adding to the need for physical RAM because it can be collected so fast.

My insights: 200ms is too long for any network application and also i2p. We want lower latency and are willing to sacrifice some throughput. With the G1 collector (default in Java >=9, needs to be specified with -XX:+UseG1GC otherwise if available) you can specify the pause time goal with -XX:MaxGCPauseMillis=xxx.

Effect on my machine: 20ms as a pause time goal reduced the allocated RAM by 50% and doubled the GC overhead from about 1 to 2% CPU. Well worth trying.

[zlatinb]

The topic of garbage collection in Java and more specifically in the i2p router is difficult to put in a nutshell because there are too many factors at play. I've worked with JVMs at low level for many years and still prefer to avoid making definitive statements about how the JVM will manage its memory as that changes not only from garbage collector to garbage collector but from version to version as well.

Recently, I spent some time profiiling the memory usage of a busy router (1-1.5MB/s participating traffic). I had allowed the router to use 12 GB of RAM and after an initial period of about two days it settled to a steady state of just about 1GB used memory as reported by "top". Minor GCs were frequent, as were major GCs, but luckily the memory usage was steady, i.e. there were no reference leaks. The minor GCs were usually shorter than 10 ms and the major ones less than 100. Of course these numbers mean anything only on my specific setup.

I suggest you enable GC logging to see what is the actual length of the pauses. More details here https://blog.codecentric.de/en/2014/01/ ... c-logging/

[jogger]

you are absolutely right, however from what you are reporting there must be 100 KB of packets piling up in your network buffers after a STW GC. That will cause a CPU burst with additional delays depending on your scheduler. I will have more on that in the next days.

Most people will be more comfortable limiting STW GC to a lower value like 20 ms.

RAM usage can comfortably be seen in jconsole, it varies widely with the major driver being the amount of Snark instances, number of torrents and burstiness of traffic. Even a low end router will accumulate 1GB over some weeks uptime.

[zlatinb]

RAM usage can comfortably be seen in jconsole

I haven't used jconsole so I don't know if that is "heap" memory usage or total jvm usage. Usually there is a big discrepancy between how much heap is allocated and how much the jvm itself is used.

I recommend you give YourKit a try - it's the profiler of choice for many people including myself. It's not free but they offer 30-day trial.