Security Updates for ColdFusion 2016 and ColdFusion 11 released

This article announces the release of ColdFusion 2016 Update 6 and ColdFusion 11 Update 14.

These updates –

  • address security vulnerabilities mentioned in the security bulletin APSB18-14,
  • upgrade the Tomcat engines & OpenSSL jars, and
  • contain few other bug fixes.

ColdFusion 2016 Update 6

ColdFusion 2016 Update 6 addresses the vulnerabilities mentioned in the security bulletin APSB18-14.
It also includes a Tomcat version upgrade to 8.5.28, OpenSSL upgrade to 1.0.2n and bug fixes in few other areas.
For the security fixes to take effect, ColdFusion should be on JDK 1.8.0_121 or higher. Post update the build number of ColdFusion 2016 should be 2016.0.06.308055.

For detailed installation instructions and the list of bugs fixed with this update, refer this technote.

ColdFusion 11 Update 14

ColdFusion 11 Update 14 addresses the vulnerabilities mentioned in the security bulletin APSB18-14.
It also includes a Tomcat version upgrade to 7.0.85, OpenSSL upgrade to 1.0.2n and fixes in few other areas.
For the security fixes to take effect, ColdFusion should be on JDK 1.7.0_131 or JDK 1.8.0_121 or higher. Post update the build number of ColdFusion 11 should be 11,0,14,307976.

For detailed installation instructions and the list of bugs fixed with this update, refer this technote.

ColdFusion performance issues and troubleshooting

Performance issues are one of the biggest challenges to expect when designing and implementing web applications. Performance problems can disrupt your business, which can result in short and long term loss of revenue.

The ColdFusion support team at Adobe, has dealt with several performance related issues with ColdFusion and a couple of them, around JDK 1.8 as well.

Based on our experience with customers, the major performance issues could be categorized as – CPU hikes, Website crashes, Processing of slow requests, Memory (for example, OutOFMemory, Memory leaks) issues, Error 503/Service unavailable error, Slow performance while running database queries, SecureRandom seed generation on some linux servers, Network latency and likewise.

ColdFusion is a Java-based application server. Any Java-related change directly impacts ColdFusion. With introduction of Java 1.8, ColdFusion had to be optimized for Java 1.8. Even after the optimization, there could be few parameters, that might cause the performance hit on the ColdFusion server.

Let’s discuss about these performance issues, method to trace them, and their possible resolutions, in a bit more detail.

CPU Hike: Hikes in CPU usage are the most common performance issue, which we experienced. Usually, the absence of load and performance testing fails to predict the impact on existing CPU utilization. CPU hike can occur due to various reasons, such as: –

● Out of memory issues
● Excessive Garbage collection
● Slow database query processing
● Network latency
● Linux random number generation
● Security scanner

OutOfMemory
This is the first parameter we must look at, when CPU surges/spikes are seen in your ColdFusion application. We can check the ColdFusion logs for OutOfMemory entries. There are two possible scenarios further.

● OutOfMemoryError: Heap – Generally OutOfMemory:Heap would not only happen because application usage is higher than the upper limit provided. But, it can also happen, because a lower value of heap, than actual usage could slow down the jvm. OutOfMemory could be because GC is not able to claim memory. This could happen because of strong references to stale objects or aggressive load so that before GC cleans up OOM is thrown. The default value for Maximum JVM Heap Size is 1GB in ColdFusion (2016 release). Based on your application’s memory usage, we can update the maximum heap value.You can change the value in ColdFusion Administrator or in jvm.config (ColdFusionXXXX/instance_name/bin).

● OutOfMemoryError: Metaspace: A new flag is available in Java 1.8 (MaxMetaspaceSize), allowing you to limit the amount of native memory used for class metadata.
In metaspace, most allocations for the class metadata are now allocated out of native memory. By default class metadata allocation is limited by the amount of available native memory. Garbage collection of the dead classes and classloaders is triggered once the class metadata usage reaches the “MaxMetaspaceSize”. Proper monitoring & tuning of the Metaspace will obviously be required in order to limit the frequency or delay of such garbage collections. Excessive Metaspace garbage collections may be a symptom of classes, classloaders memory leak or inadequate sizing for your application. If you don’t specify this flag, the Metaspace will dynamically resize depending of the application demand at runtime.

Excessive Garbage collection:
Extra load on a server triggers increased GC and causes CPU spikes. There are four types of Garbage collectors. We must figure out the one, that best suits your application.

For more information on Garbage collections please refer to below documentation:
http://www.oracle.com/webfolder/technetwork/tutorials/obe/java/gc01/index.html

By default, ColdFusion uses parallel GC. You can change the values in jvm.config (ColdFusionXXXX/instance_name/bin):

-XX:+UseConcMarkSweepGC
-XX:+UseParallelGC
-XX:+UseSerialGC
-XX:+UseG1GC – This is recommended when heap size is large (At least more than 4GB)

For detailed investigation of memory leaks or out of memory errors, a heap dump analysis can be very useful. Add the following jvm arguments in jvm.config(ColdFusionXXXX/instance_name/bin) to obtain heap dump:
-XX:+HeapDumpOnOutOfMemoryError
-XX:HeapDumpPath= <path_to_dump_file>

If you have JDK installed, run the following command from \jdk\bin directory:
jmap -dump:format=b,file=dump.hprof  where pid is the ColdFusion process id.
You can use Eclipse Memory Analyzer Tool (MAT) to review heap dumps.

Slow Database query processing:
ColdFusion logs (Application, Exception, error log) sometimes indicate whether your queries timeout or not. You can then identify slow queries and fix them.

CF does closes the connections after the timeout. CF reuses the idle/unclosed connections. When required unless the connection is still busy executing some query. If the query execution is taking too long, it has to be a problem with either the application or the database. The idle connections are being re-used, as and when required.

Technical details about the timeout:
We take two parameters in admin for this – Timeout and interval.
CF closes a maximum of 5 timed out connections at each interval. Say for example, If we have 20 open connections with timeout being set to 10 and interval being set to 5, then CF will close:-
0 connections after 5 mins
5 connections after 5 more mins
5 more connections after 5 more mins
5 more connections after 5 more mins
5 more connections after 5 more mins

So to close all the connections (as per above calculation), CF will take at least 25 mins to close, all open connections. The maximum limit of closing 5 timeout connections is not configurable and is by design.

The optimized value for timeout can be set 5 and interval to 1. You can configure them further, as per your application requirement. You can change the database timeout value in CF administrator in Advanced settings of Data & Services > Datasources to optimize idle/unclosed connections.

Network latency
If the application code resides and being accessed from a shared drive in ColdFusion Application, network latency can cause slow request processing, resulting in performance issues. This can even cause a server to crash/unresponsive. Its highly recommended to check your internal Network throughput. You can also refer to the information available on below blog:
http://blogs.coldfusion.com/source-code-deployed-on-network-path-identifying-network-latency/

You may try below :
Add the jvm arguments below to speed up the the processing of cfm pages on network/shared location:
-Dsun.io.useCanonPrefixCache=true -Dsun.io.useCanonCaches=true.

Note that 30 sec is default timeout.

This enables canonical cache that caches the canonical path of a file. This helps, when there are a lot of threads waiting to get path from WinNTFileSystem. While accessing files from a network drive, each “getCanonicalPath” would end up going to network and would become quite expensive task. Enabling this cache means that for same file, JVM would never go back to disk (till the time it is in cache) to find its path.

Linux random number generation:
Random number generation and server startup is slow on Unix platforms for some of the servers. This could be because, /dev/random is used in Unix platforms for random number generation.
java.security.SecureRandom is designed to be crypto secure. It provides strong and secure random numbers. SecureRandom should be used when high-quality randomness is important and is worth consuming CPU. We can add the below jvm argument, to get rid of performance issue due to random number generation:
-Djava.security.egd=file:/dev/./urandom

Security scanner:
If you see CPU spikes at some specific time of the day/week, this could be due to a third party security scanner interfering with your ColdFusion application. The scanner hits the server monitoring port 5500 (by default) with 0.0.0.0, which goes to infinite loop and causes server crash.

To fix this issue, we need to modify the jetty.xml at ColdFusionXXXX\cfusion\lib. Change the Server monitoring IP address from 0.0.0.0 to 127.0.0.1 and restart ColdFusion.

Code Cache:
If your program has high codecache memory set via -XX:ReservedCodeCacheSize, you can limit it by disabling code cache flushing. If flushing is disabled, the JIT does not compile methods after the codecache fills up and hence there won’t be CPU hikes. You can add the following jvm argument. This can be used to flush code cache.

XX:-UseCodeCacheFlushing
You can also disable tieredcompilation with below argument:
-XX:-TieredCompilation (Applicable only with Java 1.8. Java versions less than 8 doesn’t have tiered compilation enabled by default.)

Service unavailable error:
503 – Service unavailable is a generic error. Whenever we get this error, the first thing we should check is, whether ColdFusion is started and running or not. In case you experience intermittent 503’s, then its time to investigate the less responsive server, which might be dropping requests. This could be because of Long GC pauses or any reason that could delay response from ColdFusion server. The ColdFusion connector tuning can help us to overcome service unavailable error. Below blog post can be used to tune ColdFusion connector and avoid such errors.
http://blogs.coldfusion.com/coldfusion-11-iis-connector-tuning/

We have also seen some issues because of bugs in few specific update level of java. The best practice would be to keep your ColdFusion Java updated to latest version. Use the below blog to keep your java up to date.
http://blogs.coldfusion.com/installing-and-troubleshooting-java-updates-in-coldfusion/

ColdFusion thread dumps:
ColdFusion thread dumps can be used to analyze New, Runnable, Blocked, Waiting, Timed_Waiting and Running threads.
The issues such as Thread race, Deadlock, Hang IO calls, GC/OutOfMemory exceptions, Infinite Loop can be determined using the thread dumps. Following Blog can be used to take thread dump on a ColdFusion server:
http://blogs.coldfusion.com/taking-thread-dumps-from-coldfusion-server-programmatically/
If you are on CF11 update 12 and CF2016, you can skip copying threaddump.jar. And just use takethreaddump.cfm file to capture the thread dump.

Another issue we have seen in one or two cases, If the performance is impacted by XML parsing, the jvm argument below can fix it:
-Dcom.sun.xml.bind.v2.bytecode.ClassTailor.noOptimize=true

The other causes of performance issue include:
Lack of proper database SQL tuning & capacity planning
Application specific performance problems
Lack of proper data caching
Excessive data caching
Excessive logging

In case the above steps does not resolve the issue, please feel free to contact Adobe support (https://helpx.adobe.com/support/coldfusion.html) for analysis of the issue.

Some key points to remember:-

*Please note that, any changes made to jvm.config, would require a ColdFusion service restart.

Jvm.config: ColdFusionXXXX/instance_name/bin
ColdFusion logs: ColdFusionXXXX/instance_name/logs

Reference:
https://dzone.com/articles/java-8-permgen-metaspace
https://dzone.com/articles/top-10-causes-java-ee

ColdFusion 2016 Update 5 and ColdFusion 11 Update 13 released

This post is to announce the release of updates for ColdFusion 2016 and ColdFusion 11. These updates address a common vulnerability mentioned in security bulletin APSB 17-30.

ColdFusion 2016 Update 5

In addition to addressing the vulnerabilities in the security bulletin APSB17-30 this update includes 13 bug fixes in language, database and AJAX some other areas. For the installation instructions and details on the bugs fixed, refer this technote.

ColdFusion 11 Update 13

In addition to addressing the vulnerabilities in the security bulletin APSB17-30 this update includes 8 bug fixes in charting, AJAX and some other areas. For the installation instructions and details on the bugs fixed, refer this technote,

For the security fixes in these updates to be effective, ColdFusion 2016 should be on JDK 8 u121 or a higher version, and ColdFusion 11 should be on JDK 8 u121 or JDK 7 u131 or a higher version of JDK. The use of latest JDK update is recommended.

On a standalone installation of ColdFusion, you can upgrade Java by editing the jvm.config file at <cf_root>/cfusion/bin. For a JEE installation of ColdFusion, refer the documentation for the host application server.

Update 4 for ColdFusion Builder 2016 released

ColdFusion Builder 2016 Update 4 is now available for download.

The update adds support for ColdFusion Builder plugin installation on Eclipse Neon and fixes 22 other bugs.  
The bugs fixed are in areas such as Editor (Dictionary, Code Colorization, Code Folding), Security Code Analyzer and  RDS support. 

For instructions on how to download and install the update, refer this technote.
For the list of bugs fixed with this update, refer this technote.

To access the update directly from the ColdFusion Builder GUI, ensure that it it configured with one of the following URLs depending on the CF Builder variant in use :

Stand-alone installation of ColdFusion BuilderClick here!

Plug-in installation of ColdFusion Builder: Click here!

If you need to download the update and apply it manually, you can access the update at:
Stand-alone Update:
Click here! 
Plugin Update:
Click here! 

ColdFusion 2016 Update 4, ColdFusion 11 Update 12 and ColdFusion 10 Update 23 released

This post is to announce the release of the following ColdFusion updates:

ColdFusion 2016 Update 4

ColdFusion 2016 Update 4 upgrades Tomcat to version 8.5.11.0 and fixes 115 bugs (including 52 external bugs) in areas such as Security, Language, Charting and Performance. This update also addresses vulnerabilities mentioned in the security bulletin APSB17-14.  For details and instructions on how to apply this update refer this technote.

ColdFusion 11 Update 12

ColdFusion 11 Update 12 upgrades Tomcat to version 7.0.75. It also addresses vulnerabilities mentioned in the security bulletin APSB17-14 and fixes 59 bugs (including 28 external bugs) related to areas such as AJAX, Charting and Language. For details and instructions on how to apply this update refer this technote.

ColdFusion 10 Update 23

ColdFusion 10 Update 23 upgrades Tomcat version to 7.0.75. This update addresses vulnerabilities mentioned in the security bulletin APSB17-14 and includes a total of 17 bug fixes (including 7 external bugs) related to Language, Charting, Scheduler, Document Management and certain other areas. For details and instructions on how to apply this update refer this technote.

The build number after applying thse updates should be:

2106,0,4,302561 for ColdFusion 2016;
11,0,12,302575 for ColdFusion 11.
10,0,23,302580 for ColdFusion 10.

Note:

  • Support for Windows Server 2016 will be introduced with the refreshed full ColdFusion 2016 server installer which will be made available shortly. Update: The new installer is now available, as of Apr 28.
  • The core support for ColdFusion 10 effectively ends on May 16, 2017. It will, therefore, receive no further updates. For detailed support timelines, see this EOL matrix.

 

 

ColdFusion 10 “Core Support” ends in May 2017

I made a post last week regarding this and now would like to pass this on to wider audience, as well.

The “Core Support” for ColdFusion 10 ends on May 16, 2017. That means, no more Security patches/updates by Adobe for this version of ColdFusion after mid of May 2017. The detailed timelines are mentioned here in the EOL Matrix.

What is Core Support then? Core support is the time frame wherein the product and the support programs are available. This provides, five years of product support from the general availability date of a product.

General availability  is the date when the product and the support programs are announced and available for purchase.

Extended support provides an additional two years of Platinum Maintenance and Support services after the end of Core Support. Extended Maintenance and Support provides the extra time you may need, to plan your migration to Adobe’s latest technology. Here is the source.

So, if you are on version 10 or prior then, its the correct time for you to upgrade. This will ensure your eligibility of getting Security updates and patches timely from Adobe, for the supported versions of ColdFusion, as and when released.

ColdFusion 11 Update 11 and ColdFusion 10 Update 22 released

This post is to announce the release of ColdFusion 11 Update 11 and ColdFusion 10 Update 22.
Update 11 and Update 22 fix approximately 164 and 45 bugs respectively. For the list of bugs fixed in these updates, refer the following documents:
Bugs fixed with Update 11
Bugs fixed with Update 22

Follow the steps below to apply the updates:

  1. Navigate to ColdFusion Administrator -> Server Updates -> Updates.
  2. Switch to the "Settings" tab.
  3. Ensure that the update site URL is set to the right value by clicking on the "Restore Default URL" button.
  4. Click on "Submit changes" to save your changes.
  5. Switch to "Available Updates" tab. Click on "Check for Updates".
  6. "ColdFusion 11 Update 11" or "ColdFusion 10 Update 22" should be listed under the "Available updates" tab. 
  7. Click on the "Download and Install" button to install the update.

Refer the following technotes for instructions and other details related to the updates:

ColdFusion 11 Update 11 technote 
ColdFusoin 10 Update 22 technote

To apply these updates manually, download the required update by clicking on one of the applicable links below:

ColdFusion 11 Update 11 jar
ColdFusoin 10 Update 22 jar
 
To run the downloaded jar, execute the following command:
java -jar <jar-file-dir>/hotfix_0xx.jar
You should use the JRE used by ColdFusion for running the update jar (for standalone CF, it should be <cf_root>/jre/bin)
For further details on the manual application of the updater follow this help article.
 
The build number after applying this update should be:
11,0,11,301867 for ColdFusion 11;
10,0,22,301868 for ColdFusion 10.

Security fix for ColdFusion Builder 3 released

An important security fix for ColdFusion Builder 3 is now available for download. For more information on the vulnerability refer APSB16-44.

You can download the patch from here (md5 checksum : b67914e27ca4fb8e0fc5ecd354e9a330). Apply this patch to secure your ColdFusion Server and Builder installation. Follow the installation instructions detailed at this technote

ColdFusion 2016 installer refreshed.

The server and express installers for Adobe ColdFusion (release 2016) have been refreshed. The installers are available for download at the ColdFusion product page at www.adobe.com. The new installer includes the following changes:

  • The API Manager installer is decoupled from the ColdFusion Server installer.
  • The new API Manager installer incorporates certain new features such as multi-tenancy, enhanced security, configurable policies, a dedicated update mechanism and support for Redis cluster and request/response compression. For a detailed description of these new features follow the links embedded in this technote. The API Manager installer would be made available very soon. We will update this post to share the location where the installer would be hosted.
  • The ColdFusion installer incorporates ColdFusion 2016 Update 3 and updates JDK to version 1.8.0_112. For details on the changes that went out with Update 3 refer the Update 3 Release Notes document. The build number for this installation should be 2016,00,03,301771.
  • The features listed below have been retired from the product and no longer ship with ColdFusion. For a detailed overview of the affected areas, refer the "Portlets" and "YUI and Spry" sections of the coldfusion-deprecated-features technote. In case you need to use any of these libraries you can download them from locations mentioned below.
    • Portlets. download (md5 checksum : 93273a7b4ab8c650e5fa9cece518e099);
    • YUI. download (md5 checksum : 827e0f8395d176ac28f46ed5e78004fd);
    • Spry. download (md5 checksum : 750c275c20b291f00c1ba92c855a09d7).

 To integrate the downloaded library, follow the instructions below:

  1. Stop ColdFusion sever.
  2. Download the libraries from the links mentioned above.
  3. Extract the downloaded files to the following locations:
    • Extract portlets.zip file to <cf_root>/cfusion directory. Update the web.xml file at <CF_HOME>/cfusion/wwwroot/WEB-INF to re-introduce the mappings mentioned in the "Portlets" section of this technote.
    • Extract yui.zip and spry.zip to ColdFusion's webroot at <cf_root>/cfusion. If your scripts directory is mapped to a non-default location (setting at CF admin > Settings > Default ScriptSrc Directory), unpack the zipped package manually and place it in the custom location following the structure in the package.
  4. Restart ColdFusion server.

If you are restoring just the YUI or Spry libraries, restarting the ColdFusion server is not required.

Revisions

20 Dec, 2016 –  added the web.xml mappings step in restoring portlets instruction. added reference to coldfusion-deprecated-features article.

 

 

ColdFusion 11 Update 11 and ColdFusion 10 Update 22 PreRelease build available for download

NOTE: THIS POST has been made obsolete with the final release of these updates in Dec 2016. Please see the post announcing that.

The information below, this post and its comments, is left for history sake.

ColdFusion 11 Update 11 and ColdFusion Update 22 early access builds are now available for your testing and feedback. Please note that these are test builds and should not be used in a production environment.

For the list of bugs fixed with these updates, refer the following documents:

Follow the steps below to apply the update.

  1. Navigate to ColdFusion Administrator -> Server Updates -> Updates.
  2. Under Settings tab, check "Automatically Check for Updates" check box
  3. Change the Site URL to https://cfdownload.adobe.com/pub/adobe/coldfusion/PR/updates.xml. 
  4. Click Submit to save your changes.
  5. Under the "Available Updates" tab, click on the “Check for Updates” button.
  6. "ColdFusion 11 Update 11" or "ColdFusion 10 Update 22" should be listed under the "Available updates" tab. 
  7. Click on the "Download and Install" button to install the update.
To apply this update manually, download the required update by clicking on one of the applicable links below:
 
To run the downloaded jar, execute the following command:
java -jar <jar-file-dir>/hotfix_0xx.jar
You should use the JRE used by CF for running the update jar (for standalone CF, it should be <cf_root>/jre/bin)
For further details on the manual application of the updater follow this help article.
 
The build number after applying this update should be
11,0,11,300779 (Pre-Release) for ColdFusion 11;
10,0,22,300783 (Pre-Release) for ColdFusion 10.
 

In case, you have configured local site for receiving the update notifications, then please take back up of the URL before changing it to the prerelease URL.

We will look forward to your valuable feedback and suggestions.