We're updating the issue view to help you get more done. 

ReflectUtil.findClass(String) causing lock contention on web app class loader.

Description

We have noticed that under high load, that when we thread dump, we see lots of threads blocked in RelectUtil.findClass(String) when trying to make the following call (which is synchronzied)

Thread.currentThread().getContextClassLoader().loadClass(name);

INFO | jvm 1 | 2015/03/17 15:06:44 | "qtp560683258-213" prio=10 tid=0x00007fe39c001000 nid=0xa29 waiting for monitor entry [0x00007fe5805cb000]
INFO | jvm 1 | 2015/03/17 15:06:44 | java.lang.Thread.State: BLOCKED (on object monitor)
INFO | jvm 1 | 2015/03/17 15:06:44 | at org.eclipse.jetty.webapp.WebAppClassLoader.loadClass(WebAppClassLoader.java:396)
INFO | jvm 1 | 2015/03/17 15:06:44 | - waiting to lock <0x00000001cef0a870> (a org.eclipse.jetty.webapp.WebAppClassLoader)
INFO | jvm 1 | 2015/03/17 15:06:44 | at org.eclipse.jetty.webapp.WebAppClassLoader.loadClass(WebAppClassLoader.java:389)
INFO | jvm 1 | 2015/03/17 15:06:44 | at net.sourceforge.stripes.util.ReflectUtil.findClass(ReflectUtil.java:158)
INFO | jvm 1 | 2015/03/17 15:06:44 | at net.sourceforge.stripes.tag.StripesTagSupport.getActionBeanType(StripesTagSupport.java:191)
INFO | jvm 1 | 2015/03/17 15:06:44 | at net.sourceforge.stripes.tag.StripesTagSupport.getActionBeanUrl(StripesTagSupport.java:226)
INFO | jvm 1 | 2015/03/17 15:06:44 | at net.sourceforge.stripes.tag.UseActionBeanTag.setBeanclass(UseActionBeanTag.java:198)
INFO | jvm 1 | 2015/03/17 15:06:44 | at

:~$ grep "waiting to lock <0x00000001cef0a870> (a org.eclipse.jetty.webapp.WebAppClassLoader)" wrapper.20150317.log
INFO | jvm 1 | 2015/03/17 15:06:44 | - waiting to lock <0x00000001cef0a870> (a org.eclipse.jetty.webapp.WebAppClassLoader)
INFO | jvm 1 | 2015/03/17 15:06:44 | - waiting to lock <0x00000001cef0a870> (a org.eclipse.jetty.webapp.WebAppClassLoader)
INFO | jvm 1 | 2015/03/17 15:06:44 | - waiting to lock <0x00000001cef0a870> (a org.eclipse.jetty.webapp.WebAppClassLoader)
INFO | jvm 1 | 2015/03/17 15:06:44 | - waiting to lock <0x00000001cef0a870> (a org.eclipse.jetty.webapp.WebAppClassLoader)
INFO | jvm 1 | 2015/03/17 15:06:44 | - waiting to lock <0x00000001cef0a870> (a org.eclipse.jetty.webapp.WebAppClassLoader)
INFO | jvm 1 | 2015/03/17 15:06:44 | - waiting to lock <0x00000001cef0a870> (a org.eclipse.jetty.webapp.WebAppClassLoader)
INFO | jvm 1 | 2015/03/17 15:06:44 | - waiting to lock <0x00000001cef0a870> (a org.eclipse.jetty.webapp.WebAppClassLoader)
INFO | jvm 1 | 2015/03/17 15:06:44 | - waiting to lock <0x00000001cef0a870> (a org.eclipse.jetty.webapp.WebAppClassLoader)
INFO | jvm 1 | 2015/03/17 15:06:44 | - waiting to lock <0x00000001cef0a870> (a org.eclipse.jetty.webapp.WebAppClassLoader)
INFO | jvm 1 | 2015/03/17 15:06:44 | - waiting to lock <0x00000001cef0a870> (a org.eclipse.jetty.webapp.WebAppClassLoader)
INFO | jvm 1 | 2015/03/17 15:06:44 | - waiting to lock <0x00000001cef0a870> (a org.eclipse.jetty.webapp.WebAppClassLoader)
INFO | jvm 1 | 2015/03/17 15:06:44 | - waiting to lock <0x00000001cef0a870> (a org.eclipse.jetty.webapp.WebAppClassLoader)
INFO | jvm 1 | 2015/03/17 15:06:44 | - waiting to lock <0x00000001cef0a870> (a org.eclipse.jetty.webapp.WebAppClassLoader)
INFO | jvm 1 | 2015/03/17 15:06:44 | - waiting to lock <0x00000001cef0a870> (a org.eclipse.jetty.webapp.WebAppClassLoader)
INFO | jvm 1 | 2015/03/17 15:06:44 | - waiting to lock <0x00000001cef0a870> (a org.eclipse.jetty.webapp.WebAppClassLoader)
INFO | jvm 1 | 2015/03/17 15:06:44 | - waiting to lock <0x00000001cef0a870> (a org.eclipse.jetty.webapp.WebAppClassLoader)
INFO | jvm 1 | 2015/03/17 15:06:44 | - waiting to lock <0x00000001cef0a870> (a org.eclipse.jetty.webapp.WebAppClassLoader)
INFO | jvm 1 | 2015/03/17 15:06:44 | - waiting to lock <0x00000001cef0a870> (a org.eclipse.jetty.webapp.WebAppClassLoader)
INFO | jvm 1 | 2015/03/17 15:06:44 | - waiting to lock <0x00000001cef0a870> (a org.eclipse.jetty.webapp.WebAppClassLoader)
INFO | jvm 1 | 2015/03/17 15:06:44 | - waiting to lock <0x00000001cef0a870> (a org.eclipse.jetty.webapp.WebAppClassLoader)
INFO | jvm 1 | 2015/03/17 15:06:44 | - waiting to lock <0x00000001cef0a870> (a org.eclipse.jetty.webapp.WebAppClassLoader)
INFO | jvm 1 | 2015/03/17 15:06:44 | - waiting to lock <0x00000001cef0a870> (a org.eclipse.jetty.webapp.WebAppClassLoader)
INFO | jvm 1 | 2015/03/17 15:06:44 | - waiting to lock <0x00000001cef0a870> (a org.eclipse.jetty.webapp.WebAppClassLoader)
INFO | jvm 1 | 2015/03/17 15:06:44 | - waiting to lock <0x00000001cef0a870> (a org.eclipse.jetty.webapp.WebAppClassLoader)
INFO | jvm 1 | 2015/03/17 15:06:44 | - waiting to lock <0x00000001cef0a870> (a org.eclipse.jetty.webapp.WebAppClassLoader)
INFO | jvm 1 | 2015/03/17 15:06:44 | - waiting to lock <0x00000001cef0a870> (a org.eclipse.jetty.webapp.WebAppClassLoader)
INFO | jvm 1 | 2015/03/17 15:06:44 | - waiting to lock <0x00000001cef0a870> (a org.eclipse.jetty.webapp.WebAppClassLoader)
INFO | jvm 1 | 2015/03/17 15:06:44 | - waiting to lock <0x00000001cef0a870> (a org.eclipse.jetty.webapp.WebAppClassLoader)
INFO | jvm 1 | 2015/03/17 15:06:44 | - waiting to lock <0x00000001cef0a870> (a org.eclipse.jetty.webapp.WebAppClassLoader)
INFO | jvm 1 | 2015/03/17 15:06:44 | - waiting to lock <0x00000001cef0a870> (a org.eclipse.jetty.webapp.WebAppClassLoader)
INFO | jvm 1 | 2015/03/17 15:06:44 | - waiting to lock <0x00000001cef0a870> (a org.eclipse.jetty.webapp.WebAppClassLoader)
INFO | jvm 1 | 2015/03/17 15:06:44 | - waiting to lock <0x00000001cef0a870> (a org.eclipse.jetty.webapp.WebAppClassLoader)
INFO | jvm 1 | 2015/03/17 15:06:44 | - waiting to lock <0x00000001cef0a870> (a org.eclipse.jetty.webapp.WebAppClassLoader)
INFO | jvm 1 | 2015/03/17 15:06:44 | - waiting to lock <0x00000001cef0a870> (a org.eclipse.jetty.webapp.WebAppClassLoader)
INFO | jvm 1 | 2015/03/17 15:06:44 | - waiting to lock <0x00000001cef0a870> (a org.eclipse.jetty.webapp.WebAppClassLoader)
INFO | jvm 1 | 2015/03/17 15:06:44 | - waiting to lock <0x00000001cef0a870> (a org.eclipse.jetty.webapp.WebAppClassLoader)
INFO | jvm 1 | 2015/03/17 15:06:44 | - waiting to lock <0x00000001cef0a870> (a org.eclipse.jetty.webapp.WebAppClassLoader)
INFO | jvm 1 | 2015/03/17 15:06:44 | - waiting to lock <0x00000001cef0a870> (a org.eclipse.jetty.webapp.WebAppClassLoader)
INFO | jvm 1 | 2015/03/17 15:06:44 | - waiting to lock <0x00000001cef0a870> (a org.eclipse.jetty.webapp.WebAppClassLoader)
INFO | jvm 1 | 2015/03/17 15:06:44 | - waiting to lock <0x00000001cef0a870> (a org.eclipse.jetty.webapp.WebAppClassLoader)
INFO | jvm 1 | 2015/03/17 15:06:44 | - waiting to lock <0x00000001cef0a870> (a org.eclipse.jetty.webapp.WebAppClassLoader)
INFO | jvm 1 | 2015/03/17 15:06:44 | - waiting to lock <0x00000001cef0a870> (a org.eclipse.jetty.webapp.WebAppClassLoader)
INFO | jvm 1 | 2015/03/17 15:06:44 | - waiting to lock <0x00000001cef0a870> (a org.eclipse.jetty.webapp.WebAppClassLoader)
INFO | jvm 1 | 2015/03/17 15:06:44 | - waiting to lock <0x00000001cef0a870> (a org.eclipse.jetty.webapp.WebAppClassLoader)
INFO | jvm 1 | 2015/03/17 15:06:44 | - waiting to lock <0x00000001cef0a870> (a org.eclipse.jetty.webapp.WebAppClassLoader)

It does appear to be a limiting factor when trying to scale our application.

Environment

Jetty 9.x
Oracle 1.7.0_75
RHEL x64
Dual socket, 6 cores (12 cores / 24 hyperthreaded)

Status

Assignee

Rick G

Reporter

David Wade

Labels

None

Tester

None

Fix versions

Affects versions

Release 1.5.7
Release 1.5.8

Priority

Major