Stuck thread for CASFilter and SSOFilter class in WCS - GST URL

[Sivakumar8789]

GSF Team,

Appreciate any inputs for the below issue

Issue:We have observed that CASFilter and SSOFilter classes creating stuck threads in WCS production. We are having WCS prod env with 6 managed servers, all went into warning state during business hours and we observed that there was "java.lang.OutOfMemoryError: allocLargeObjectOrArray: [C, size 235929640" error with which we observed many stuck threads as below.

Dump:

From Heap dump observed that Character arrays are taking much space and all these are GST urls

Class Name | Shallow Heap | Retained Heap | Percentage

char[117964810] @ 0xed4ea5e0 /Satellite?virtual-webroot=https://www.celcom.com.my&pagename=GST/Dispatcher&url-path=/Satellite&site=Celcom_PBO&virtual-webroot=https://www.celcom.com.my&pagename=GST/Dispatcher&url-path=/Satellite&site=Celcom_PBO&virtual-webroot=https://www.celcom.com.my...| 235,929,640 | 235,929,640 | 7.46%

Char arrays are taking more space and could anyone confirm us that org.tuckey.web.filters.urlrewrite.* are using for this GST urls?

-- Log File --

FileName

<Nov 24, 2015 4:08:37 PM MYT> <[STUCK] ExecuteThread: '9' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "646" seconds working on the request "weblogic.servlet.internal.ServletRequestImpl@1a0c2cc1[
GET /cs/..%5C..%5C..%5C..%5C..%5C..%5C..%5C..%5Cwindows%5Cwin.ini/out/ HTTP/1.0
Referer: https://www.celcom.com.my/personal
Cookie: JSESSIONID=Gr32WTqYSjrhyJjZZm1N5TC910l71VnTdDHG4njGqLNNrlqwjfhf!-240226222; PHPSESSID=ii6rsqau04ra2kaguqt3o9dhh7; dtCookie=5F1E00607EF3475FAA4B3CF0B1BBBE33|Online+Customer+Service|1|WebCenter+Portal|1; wcpjsessionid=VTkYWTqh6GLLj2yfkPrXMtyWcRyBY2bnQgwLS3YFsxFT1TNDyQw0!-164721746; BIGipServerprod-wcp-app-pool=3926590730.16671.0000
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/28.0.1500.63 Safari/537.36
Accept: /
ECID-Context: 1.0059KHx4jM6Dc_95zfp2iW0000gS00aAqb;kXjE9ZDLIPHCmFBOnVABi59DZUAB_5FOiE99p3QBi59DZUAB_5FOiE99p3QBi59DZUAB_5FOrPQR^VRTnL9DZNTQiU9QiPRBjLSTj0
Connection: Keep-Alive
X-Forwarded-For: 173.247.194.197, 10.202.1.90
Proxy-Client-IP: 10.5.3.254
X-WebLogic-KeepAliveSecs: 30
X-WebLogic-Force-JVMID: -1051495285

]", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:
Thread-173 "[STUCK] ExecuteThread: '9' for queue: 'weblogic.kernel.Default (self-tuning)'" <alive, suspended, priority=1, DAEMON> {
java.lang.StringCoding$StringDecoder.decode(StringCoding.java:132)
java.lang.StringCoding.decode(StringCoding.java:159)
java.lang.String.(String.java:439)
weblogic.utils.http.BytesToString.newNonAsciiString(BytesToString.java:25)
weblogic.utils.http.BytesToString.newString(BytesToString.java:14)
weblogic.utils.http.HttpParsing.unescape(HttpParsing.java:343)
weblogic.servlet.internal.ServletRequestImpl.getContextPath(ServletRequestImpl.java:625)
com.fatwire.wem.sso.cas.filter.util.RequestMatcher.match(RequestMatcher.java:24)
com.fatwire.wem.sso.cas.filter.CASFilter.doFilter(CASFilter.java:139)
com.fatwire.wem.sso.SSOFilter.doFilter(SSOFilter.java:48)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.wrapRun(WebAppServletContext.java:3687)
weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3681)
weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:308)
weblogic.security.service.SecurityManager.runAs(SecurityManager.java:116)
weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2213)
weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2135)
weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1420)
weblogic.work.ExecuteThread.execute(ExecuteThread.java:203)
weblogic.work.ExecuteThread.run(ExecuteThread.java:170)
}

2015-11-24 16:08:51 [5ec24610] info [native] Got ResourceExhausted Notification: allocLargeObjectOrArray: [C, size 61341736, not creating Memory Dump
<Nov 24, 2015 4:08:54 PM MYT> <[ServletContext@190176262[app:cs1 module:cs1 path:/cs spec-version:2.5]] Root cause of ServletException.
java.lang.OutOfMemoryError: allocLargeObjectOrArray: [C, size 61341736
at java.util.Arrays.copyOf(Arrays.java:2882)
at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:100)
at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:390)
at java.lang.StringBuffer.append(StringBuffer.java:224)
at org.tuckey.web.filters.urlrewrite.substitution.MatcherReplacer.substitute(MatcherReplacer.java:105)
Truncated. see log file for complete stacktrace

FileComment

Character arrays are taking more space and not able to allocate space for others which may raise OutOfMemory error and this happens from at org.tuckey.web.filters.urlrewrite.substitution.MatcherReplacer.substitute and this may be some customisation