ColdFusion performance issues and troubleshooting

Performance issues are one of the biggest challenges to expect when designing and implementing web applications. Performance problems can disrupt your business, which can result in short and long term loss of revenue.

The ColdFusion support team at Adobe, has dealt with several performance related issues with ColdFusion and a couple of them, around JDK 1.8 as well.

Based on our experience with customers, the major performance issues could be categorized as – CPU hikes, Website crashes, Processing of slow requests, Memory (for example, OutOFMemory, Memory leaks) issues, Error 503/Service unavailable error, Slow performance while running database queries, SecureRandom seed generation on some linux servers, Network latency and likewise.

ColdFusion is a Java-based application server. Any Java-related change directly impacts ColdFusion. With introduction of Java 1.8, ColdFusion had to be optimized for Java 1.8. Even after the optimization, there could be few parameters, that might cause the performance hit on the ColdFusion server.

Let’s discuss about these performance issues, method to trace them, and their possible resolutions, in a bit more detail.

CPU Hike: Hikes in CPU usage are the most common performance issue, which we experienced. Usually, the absence of load and performance testing fails to predict the impact on existing CPU utilization. CPU hike can occur due to various reasons, such as: –

● Out of memory issues
● Excessive Garbage collection
● Slow database query processing
● Network latency
● Linux random number generation
● Security scanner

OutOfMemory
This is the first parameter we must look at, when CPU surges/spikes are seen in your ColdFusion application. We can check the ColdFusion logs for OutOfMemory entries. There are two possible scenarios further.

● OutOfMemoryError: Heap – Generally OutOfMemory:Heap would not only happen because application usage is higher than the upper limit provided. But, it can also happen, because a lower value of heap, than actual usage could slow down the jvm. OutOfMemory could be because GC is not able to claim memory. This could happen because of strong references to stale objects or aggressive load so that before GC cleans up OOM is thrown. The default value for Maximum JVM Heap Size is 1GB in ColdFusion (2016 release). Based on your application’s memory usage, we can update the maximum heap value.You can change the value in ColdFusion Administrator or in jvm.config (ColdFusionXXXX/instance_name/bin).

● OutOfMemoryError: Metaspace: A new flag is available in Java 1.8 (MaxMetaspaceSize), allowing you to limit the amount of native memory used for class metadata.
In metaspace, most allocations for the class metadata are now allocated out of native memory. By default class metadata allocation is limited by the amount of available native memory. Garbage collection of the dead classes and classloaders is triggered once the class metadata usage reaches the “MaxMetaspaceSize”. Proper monitoring & tuning of the Metaspace will obviously be required in order to limit the frequency or delay of such garbage collections. Excessive Metaspace garbage collections may be a symptom of classes, classloaders memory leak or inadequate sizing for your application. If you don’t specify this flag, the Metaspace will dynamically resize depending of the application demand at runtime.

Excessive Garbage collection:
Extra load on a server triggers increased GC and causes CPU spikes. There are four types of Garbage collectors. We must figure out the one, that best suits your application.

For more information on Garbage collections please refer to below documentation:
http://www.oracle.com/webfolder/technetwork/tutorials/obe/java/gc01/index.html

By default, ColdFusion uses parallel GC. You can change the values in jvm.config (ColdFusionXXXX/instance_name/bin):

-XX:+UseConcMarkSweepGC
-XX:+UseParallelGC
-XX:+UseSerialGC
-XX:+UseG1GC – This is recommended when heap size is large (At least more than 4GB)

For detailed investigation of memory leaks or out of memory errors, a heap dump analysis can be very useful. Add the following jvm arguments in jvm.config(ColdFusionXXXX/instance_name/bin) to obtain heap dump:
-XX:+HeapDumpOnOutOfMemoryError
-XX:HeapDumpPath= <path_to_dump_file>

If you have JDK installed, run the following command from \jdk\bin directory:
jmap -dump:format=b,file=dump.hprof  where pid is the ColdFusion process id.
You can use Eclipse Memory Analyzer Tool (MAT) to review heap dumps.

Slow Database query processing:
ColdFusion logs (Application, Exception, error log) sometimes indicate whether your queries timeout or not. You can then identify slow queries and fix them.

CF does closes the connections after the timeout. CF reuses the idle/unclosed connections. When required unless the connection is still busy executing some query. If the query execution is taking too long, it has to be a problem with either the application or the database. The idle connections are being re-used, as and when required.

Technical details about the timeout:
We take two parameters in admin for this – Timeout and interval.
CF closes a maximum of 5 timed out connections at each interval. Say for example, If we have 20 open connections with timeout being set to 10 and interval being set to 5, then CF will close:-
0 connections after 5 mins
5 connections after 5 more mins
5 more connections after 5 more mins
5 more connections after 5 more mins
5 more connections after 5 more mins

So to close all the connections (as per above calculation), CF will take at least 25 mins to close, all open connections. The maximum limit of closing 5 timeout connections is not configurable and is by design.

The optimized value for timeout can be set 5 and interval to 1. You can configure them further, as per your application requirement. You can change the database timeout value in CF administrator in Advanced settings of Data & Services > Datasources to optimize idle/unclosed connections.

Network latency
If the application code resides and being accessed from a shared drive in ColdFusion Application, network latency can cause slow request processing, resulting in performance issues. This can even cause a server to crash/unresponsive. Its highly recommended to check your internal Network throughput. You can also refer to the information available on below blog:
http://blogs.coldfusion.com/source-code-deployed-on-network-path-identifying-network-latency/

You may try below :
Add the jvm arguments below to speed up the the processing of cfm pages on network/shared location:
-Dsun.io.useCanonPrefixCache=true -Dsun.io.useCanonCaches=true.

Note that 30 sec is default timeout.

This enables canonical cache that caches the canonical path of a file. This helps, when there are a lot of threads waiting to get path from WinNTFileSystem. While accessing files from a network drive, each “getCanonicalPath” would end up going to network and would become quite expensive task. Enabling this cache means that for same file, JVM would never go back to disk (till the time it is in cache) to find its path.

Linux random number generation:
Random number generation and server startup is slow on Unix platforms for some of the servers. This could be because, /dev/random is used in Unix platforms for random number generation.
java.security.SecureRandom is designed to be crypto secure. It provides strong and secure random numbers. SecureRandom should be used when high-quality randomness is important and is worth consuming CPU. We can add the below jvm argument, to get rid of performance issue due to random number generation:
-Djava.security.egd=file:/dev/./urandom

Security scanner:
If you see CPU spikes at some specific time of the day/week, this could be due to a third party security scanner interfering with your ColdFusion application. The scanner hits the server monitoring port 5500 (by default) with 0.0.0.0, which goes to infinite loop and causes server crash.

To fix this issue, we need to modify the jetty.xml at ColdFusionXXXX\cfusion\lib. Change the Server monitoring IP address from 0.0.0.0 to 127.0.0.1 and restart ColdFusion.

Code Cache:
If your program has high codecache memory set via -XX:ReservedCodeCacheSize, you can limit it by disabling code cache flushing. If flushing is disabled, the JIT does not compile methods after the codecache fills up and hence there won’t be CPU hikes. You can add the following jvm argument. This can be used to flush code cache.

XX:-UseCodeCacheFlushing
You can also disable tieredcompilation with below argument:
-XX:-TieredCompilation (Applicable only with Java 1.8. Java versions less than 8 doesn’t have tiered compilation enabled by default.)

Service unavailable error:
503 – Service unavailable is a generic error. Whenever we get this error, the first thing we should check is, whether ColdFusion is started and running or not. In case you experience intermittent 503’s, then its time to investigate the less responsive server, which might be dropping requests. This could be because of Long GC pauses or any reason that could delay response from ColdFusion server. The ColdFusion connector tuning can help us to overcome service unavailable error. Below blog post can be used to tune ColdFusion connector and avoid such errors.
http://blogs.coldfusion.com/coldfusion-11-iis-connector-tuning/

We have also seen some issues because of bugs in few specific update level of java. The best practice would be to keep your ColdFusion Java updated to latest version. Use the below blog to keep your java up to date.
http://blogs.coldfusion.com/installing-and-troubleshooting-java-updates-in-coldfusion/

ColdFusion thread dumps:
ColdFusion thread dumps can be used to analyze New, Runnable, Blocked, Waiting, Timed_Waiting and Running threads.
The issues such as Thread race, Deadlock, Hang IO calls, GC/OutOfMemory exceptions, Infinite Loop can be determined using the thread dumps. Following Blog can be used to take thread dump on a ColdFusion server:
http://blogs.coldfusion.com/taking-thread-dumps-from-coldfusion-server-programmatically/
If you are on CF11 update 12 and CF2016, you can skip copying threaddump.jar. And just use takethreaddump.cfm file to capture the thread dump.

Another issue we have seen in one or two cases, If the performance is impacted by XML parsing, the jvm argument below can fix it:
-Dcom.sun.xml.bind.v2.bytecode.ClassTailor.noOptimize=true

The other causes of performance issue include:
Lack of proper database SQL tuning & capacity planning
Application specific performance problems
Lack of proper data caching
Excessive data caching
Excessive logging

In case the above steps does not resolve the issue, please feel free to contact Adobe support (https://helpx.adobe.com/support/coldfusion.html) for analysis of the issue.

Some key points to remember:-

*Please note that, any changes made to jvm.config, would require a ColdFusion service restart.

Jvm.config: ColdFusionXXXX/instance_name/bin
ColdFusion logs: ColdFusionXXXX/instance_name/logs

Reference:
https://dzone.com/articles/java-8-permgen-metaspace
https://dzone.com/articles/top-10-causes-java-ee

Update 4 for ColdFusion Builder 2016 released

ColdFusion Builder 2016 Update 4 is now available for download.

The update adds support for ColdFusion Builder plugin installation on Eclipse Neon and fixes 22 other bugs.  
The bugs fixed are in areas such as Editor (Dictionary, Code Colorization, Code Folding), Security Code Analyzer and  RDS support. 

For instructions on how to download and install the update, refer this technote.
For the list of bugs fixed with this update, refer this technote.

To access the update directly from the ColdFusion Builder GUI, ensure that it it configured with one of the following URLs depending on the CF Builder variant in use :

Stand-alone installation of ColdFusion BuilderClick here!

Plug-in installation of ColdFusion Builder: Click here!

If you need to download the update and apply it manually, you can access the update at:
Stand-alone Update:
Click here! 
Plugin Update:
Click here! 

ColdFusion 2016 : Support for Windows Server 2016

We have updated the Windows 64-bit installer for ColdFusion 2016, to support Microsoft Windows Server 2016. The Add-on services installer and the .NET service installer for ColdFusion 2016 have also been refreshed.
You can access the server installer by clicking on the "free trial" or the "buy now" link in the ColdFusion product page at Adobe.com. You may download the aforementioned additional installers at the ColdFusion support page

The ColdFusion 2016 support matrix would be updated soon to reflect the support for the new platform.

The refreshed installer comes with the Update 3 baked-in. After installing the server, you may bring it up to the current update level, by installing Update 4. You can follow the instructions at this technote to download and install Update 4. If you need any help with installing ColdFusion server you may refer the installation instructions at this technote.

The installers for ColdFusion Builder 2016 and ColdFusion API Manager would soon be refreshed to support Windows Server 2016.

ColdFusion 2016 Update 4, ColdFusion 11 Update 12 and ColdFusion 10 Update 23 released

This post is to announce the release of the following ColdFusion updates:

ColdFusion 2016 Update 4

ColdFusion 2016 Update 4 upgrades Tomcat to version 8.5.11.0 and fixes 115 bugs (including 52 external bugs) in areas such as Security, Language, Charting and Performance. This update also addresses vulnerabilities mentioned in the security bulletin APSB17-14.  For details and instructions on how to apply this update refer this technote.

ColdFusion 11 Update 12

ColdFusion 11 Update 12 upgrades Tomcat to version 7.0.75. It also addresses vulnerabilities mentioned in the security bulletin APSB17-14 and fixes 59 bugs (including 28 external bugs) related to areas such as AJAX, Charting and Language. For details and instructions on how to apply this update refer this technote.

ColdFusion 10 Update 23

ColdFusion 10 Update 23 upgrades Tomcat version to 7.0.75. This update addresses vulnerabilities mentioned in the security bulletin APSB17-14 and includes a total of 17 bug fixes (including 7 external bugs) related to Language, Charting, Scheduler, Document Management and certain other areas. For details and instructions on how to apply this update refer this technote.

The build number after applying thse updates should be:

2106,0,4,302561 for ColdFusion 2016;
11,0,12,302575 for ColdFusion 11.
10,0,23,302580 for ColdFusion 10.

Note:

  • Support for Windows Server 2016 will be introduced with the refreshed full ColdFusion 2016 server installer which will be made available shortly. Update: The new installer is now available, as of Apr 28.
  • The core support for ColdFusion 10 effectively ends on May 16, 2017. It will, therefore, receive no further updates. For detailed support timelines, see this EOL matrix.

 

 

Security fix for ColdFusion Builder 3 released

An important security fix for ColdFusion Builder 3 is now available for download. For more information on the vulnerability refer APSB16-44.

You can download the patch from here (md5 checksum : b67914e27ca4fb8e0fc5ecd354e9a330). Apply this patch to secure your ColdFusion Server and Builder installation. Follow the installation instructions detailed at this technote

Updates for ColdFusion 2016 and ColdFusion Builder 2016 are available now

This post is to announce the availability of updates for ColdFusion 2016 and ColdFusion Builder 2016.

ColdFusion (2016 release) Update 3

ColdFusion 2016 Update 3 includes support for Windows 10 Version 1607 and IBM Websphere Application Server 9. It also includes approximately 180 bug fixes(including ~100 external bugs) related to Language, Document Management, Serialization, Net Protocols, Database, Administrator and a few other areas.

For more details, refer this tech note.

ColdFusion Builder (2016 release) Update 3

ColdFusion Builder 2016 Update 3 includes:
    1. Important security fixes
    2. Bugs fixes in the areas of Editor and Security Code Analyzer.
    3. Upgraded PhoneGap libraries(from version 5.2.0 to version 6.0.0)
    4. Dictionary (Code Assist) changes to accommodate the changes/enhancements in ColdFusion 2016 Update 3 and earlier.
    
For more details, refer this tech note.

NOTE: We will be back porting all the important/applicable fixes to ColdFusion 10 and 11 version. Pre-release build for the same will be available in a couple of weeks for your feedback.

ColdFusion 2016 Update 3 early access build is now available

ColdFusion 2016 Update 3 early access build is now available for your testing and feedback. It includes support for Windows 10.1 and IBM Websphere Application Server 9 along with approximately 180 bug fixes.

Please note that this is a test build and should not be used in a production environment.

Refer the documents given below for more details on this update:

1. What's New in this update    
2. Issues Fixed in this update
3. Installation Instructions

The build number after applying this update should be 2016.0.03.300232.

In case, you have configured local site for receiving the update notifications, then please take back up of the URL before changing it to the pre-release URL.

We will look forward to your valuable feedback and suggestions.

 

ColdFusion Builder 2016 Update 1 released

This update is a companion update to ColdFusion 2016 Update 1 and primarily addresses issues related to Security Code Analyzer and it’s performance.

The issues fixed for this release are listed in this document

This update is applicable for a standalone as well as a plugin installation of ColdFusion Builder. After applying this update, ColdFusion Builder build number should be 298831.

ColdFusion Builder has an automatic update notification that notifies the user of the updates availability.

Application deployed on Network/Remote Path – Identifying Network Latency – Improve Performance

Many a time, ColdFusion application code is deployed on a network path when your ColdFusion deployments are of large-scale and mandated to use network paths.

After setting up the server for the first time, if there is any performance hit, as the first thing you would want to cross-check few things. One of the things to determine is if there is any network latency.

Though you would have got same network within your organization same as earlier, your OS version also would have changed.

Follow the steps below to see if the performance hit is due to network latency-

When the server is under moderate or full load(with at least 8-10 requests under process), take 2 or 3 thread dumps with 30 seconds interval.

It is not appropriate to take thread dump when the server has negligible load and anlyze that as there may not be any in-process requests.

If you are not sure how to take thread dump, you can simply follow the following blog.

( Taking Thread Dumps From ColdFusion Server Programmatically )

Open the thread dump file:

Under moderate or full load server conditions, if you see more than 5-8% of running ColdFusion threads containing “WinNTFileSystem” in the thread’s stack trace –> It means that there is lot of time being spent in trying to resolve the application file paths.

Following are the sample threads having WinNTFileSystem in its dump.

"ajp-bio-8014-exec-6861" Id=13898 in RUNNABLE
 java.lang.Thread.State: RUNNABLE
 prio=5 blockedtime=28963 blockedcount=6819 waitedtime=421762 waitedcount=115
    at java.io.WinNTFileSystem.getBooleanAttributes(Native Method)
    at java.io.File.isFile(File.java:876)

 

"ajp-bio-8014-exec-6861" Id=13898 in RUNNABLE (running in native)
 java.lang.Thread.State: RUNNABLE
 prio=5 blockedtime=28961 blockedcount=6814 waitedtime=421762 waitedcount=115
    at java.io.WinNTFileSystem.canonicalize0(Native Method)
    at java.io.Win32FileSystem.canonicalize(Win32FileSystem.java:414)
    at java.io.File.getCanonicalPath(File.java:618)

 

(Note: ColdFusion threads can be identified by the name starting with "ajp-" )

For Example, if there are 50 threads with thread name starting "ajp-bio-" in the thread dump, if you see WinNTFileSystem in more than 2-3 threads, it is the time you start looking at minimizing the network latency.

 

Once you know there is latency, you would want to know how much is the latency when compared to the application existing locally.

Created a very basic network latency test program to validate this.

You can take the jar from here.

And run it from command prompt as follows:

> C:ColdFusion11jrebinjava -jar <Path of NetworkPathsTest.jar> <Network or Local Directory Path >

If the network path (Ex:- \orgserverd$) is accessible only to the ColdFusion service user, open command prompt as that user ( runas /user:<cfserviceaccount domainname>cfserviceusername CMD )

 

Examples:

Path Arguments can be one or more. More Path arguments is a good measure to see the difference clearly.

C:ColdFusion11jrebinjava -jar C:ColdFusion11NetworkPathTest.jar \orgserverd$deploycfm

C:ColdFusion11jrebinjava -jar C:ColdFusion11NetworkPathTest.jar \orgserverd$deploycfm \orgserverd$deploycfmapi

Try the same paths keeping the content same on the local machine and see the time differences.

For the same paths on local and remote, the difference in time should not be exponential.

These tests are to be performed on your ColdFusion server machine.

Once you have validations and found any latencies, it is the time to call for network optimization expertise.

 

 

getHeaders – a new attribute in the cfexchangemail tag

With ColdFusion 11 Update 3, we have introduced a new parameter called “getHeaders”, in the “cfExchangeMail” tag. It accepts a boolean value. When set to true, cfExchangeMail returns a query with an additional “InternetHeader” column. This column contains a struct containing key-value pairs of the email-headers associated with each message.

Email message headers provide technical details about the message, such as who sent it, the software used to compose it, the version of the MIME protocol used by the sender etc. 

On Exchange 2010, the fields that are returned are: CC, Content-Transfer-Encoding, Content-Type, Date, From, MIME-Version, Message-ID, Received, Return-Path, Subject, To, X-MS-Exchange-Organization-AuthAs, X-MS-Exchange-Organization-AuthSource, X-Mailer.

You may reference this weblink for the detailed list of the fields and their description.

You can put this new feature to any good use that suites your purpose. I will dwell on one such use case below.

In MS Exchange 2010 and later, the “ToId” column in the retrieved messages query contains the primary email address of the sender. A primary email address can have multiple aliases. If you need to retrieve the email-alias the message was sent to, you can make use of this new attribute.

Here’s an example that demonstrates the usage the tag in the context of the use case discussed above:

<cfmail from=”#frm_usr_email#” to=”#to_usr_alias#” cc=”#cc_usr_alias#” subject=”#msg_sub#”  server= “#exchangeServerIP#” port = “25”>

———– testing mail to an alias address ————

</cfmail>

<cfset sleep(5000)>

<cfexchangeConnection action=”open” username =”#to_usr#” password=”#password#” server=”#exchangeServerIP#” serverversion=”#version#” protocol=”#protocol#” connection=”excon”>

<cfexchangemail action=”get” name=”usr_msgs” connection=”excon” getheaders=true folder=”Inbox”>

<cfexchangefilter name=”fromID” value=’#frm_usr#’>

<cfexchangefilter name=”subject” value=”#msg_sub#”>

</cfexchangemail>

<cfif usr_msgs.recordcount GTE 1>

info from cfexchangemail fields:<br>

<cfloop query=”usr_msgs”>

<cfoutput>

#usr_msgs.subject#<br> 

#usr_msgs.CC#<br> 

#usr_msgs.fromId#<br>

</cfoutput>

</cfloop>

info from cfexchangemail.internetHeaders fields:<br>

<cfloop query=”usr_msgs”>

<cfoutput>

#ReplaceList(usr_msgs.internetHeaders[“from”][1], “>,<“, “,”, “,”, “,”)#<br>

#ReplaceList(usr_msgs.internetHeaders[“to”][1], “>,<“, “,”, “,”, “,”)#<br>

#ReplaceList(usr_msgs.internetHeaders[“cc”][1], “>,<“, “,”, “,”, “,”)#<br>

</cfoutput>

</cfloop>

</cfif>

 

You can reference the bugbase, for the enhancement request originally logged for this feature.