Security Updates for ColdFusion 2016 and ColdFusion 11 released

This article announces the release of ColdFusion 2016 Update 6 and ColdFusion 11 Update 14.

These updates –

  • address security vulnerabilities mentioned in the security bulletin APSB18-14,
  • upgrade the Tomcat engines & OpenSSL jars, and
  • contain few other bug fixes.

ColdFusion 2016 Update 6

ColdFusion 2016 Update 6 addresses the vulnerabilities mentioned in the security bulletin APSB18-14.
It also includes a Tomcat version upgrade to 8.5.28, OpenSSL upgrade to 1.0.2n and bug fixes in few other areas.
For the security fixes to take effect, ColdFusion should be on JDK 1.8.0_121 or higher. Post update the build number of ColdFusion 2016 should be 2016.0.06.308055.

For detailed installation instructions and the list of bugs fixed with this update, refer this technote.

ColdFusion 11 Update 14

ColdFusion 11 Update 14 addresses the vulnerabilities mentioned in the security bulletin APSB18-14.
It also includes a Tomcat version upgrade to 7.0.85, OpenSSL upgrade to 1.0.2n and fixes in few other areas.
For the security fixes to take effect, ColdFusion should be on JDK 1.7.0_131 or JDK 1.8.0_121 or higher. Post update the build number of ColdFusion 11 should be 11,0,14,307976.

For detailed installation instructions and the list of bugs fixed with this update, refer this technote.

ColdFusion performance issues and troubleshooting

Performance issues are one of the biggest challenges to expect when designing and implementing web applications. Performance problems can disrupt your business, which can result in short and long term loss of revenue.

The ColdFusion support team at Adobe, has dealt with several performance related issues with ColdFusion and a couple of them, around JDK 1.8 as well.

Based on our experience with customers, the major performance issues could be categorized as – CPU hikes, Website crashes, Processing of slow requests, Memory (for example, OutOFMemory, Memory leaks) issues, Error 503/Service unavailable error, Slow performance while running database queries, SecureRandom seed generation on some linux servers, Network latency and likewise.

ColdFusion is a Java-based application server. Any Java-related change directly impacts ColdFusion. With introduction of Java 1.8, ColdFusion had to be optimized for Java 1.8. Even after the optimization, there could be few parameters, that might cause the performance hit on the ColdFusion server.

Let’s discuss about these performance issues, method to trace them, and their possible resolutions, in a bit more detail.

CPU Hike: Hikes in CPU usage are the most common performance issue, which we experienced. Usually, the absence of load and performance testing fails to predict the impact on existing CPU utilization. CPU hike can occur due to various reasons, such as: –

● Out of memory issues
● Excessive Garbage collection
● Slow database query processing
● Network latency
● Linux random number generation
● Security scanner

OutOfMemory
This is the first parameter we must look at, when CPU surges/spikes are seen in your ColdFusion application. We can check the ColdFusion logs for OutOfMemory entries. There are two possible scenarios further.

● OutOfMemoryError: Heap – Generally OutOfMemory:Heap would not only happen because application usage is higher than the upper limit provided. But, it can also happen, because a lower value of heap, than actual usage could slow down the jvm. OutOfMemory could be because GC is not able to claim memory. This could happen because of strong references to stale objects or aggressive load so that before GC cleans up OOM is thrown. The default value for Maximum JVM Heap Size is 1GB in ColdFusion (2016 release). Based on your application’s memory usage, we can update the maximum heap value.You can change the value in ColdFusion Administrator or in jvm.config (ColdFusionXXXX/instance_name/bin).

● OutOfMemoryError: Metaspace: A new flag is available in Java 1.8 (MaxMetaspaceSize), allowing you to limit the amount of native memory used for class metadata.
In metaspace, most allocations for the class metadata are now allocated out of native memory. By default class metadata allocation is limited by the amount of available native memory. Garbage collection of the dead classes and classloaders is triggered once the class metadata usage reaches the “MaxMetaspaceSize”. Proper monitoring & tuning of the Metaspace will obviously be required in order to limit the frequency or delay of such garbage collections. Excessive Metaspace garbage collections may be a symptom of classes, classloaders memory leak or inadequate sizing for your application. If you don’t specify this flag, the Metaspace will dynamically resize depending of the application demand at runtime.

Excessive Garbage collection:
Extra load on a server triggers increased GC and causes CPU spikes. There are four types of Garbage collectors. We must figure out the one, that best suits your application.

For more information on Garbage collections please refer to below documentation:
http://www.oracle.com/webfolder/technetwork/tutorials/obe/java/gc01/index.html

By default, ColdFusion uses parallel GC. You can change the values in jvm.config (ColdFusionXXXX/instance_name/bin):

-XX:+UseConcMarkSweepGC
-XX:+UseParallelGC
-XX:+UseSerialGC
-XX:+UseG1GC – This is recommended when heap size is large (At least more than 4GB)

For detailed investigation of memory leaks or out of memory errors, a heap dump analysis can be very useful. Add the following jvm arguments in jvm.config(ColdFusionXXXX/instance_name/bin) to obtain heap dump:
-XX:+HeapDumpOnOutOfMemoryError
-XX:HeapDumpPath= <path_to_dump_file>

If you have JDK installed, run the following command from \jdk\bin directory:
jmap -dump:format=b,file=dump.hprof  where pid is the ColdFusion process id.
You can use Eclipse Memory Analyzer Tool (MAT) to review heap dumps.

Slow Database query processing:
ColdFusion logs (Application, Exception, error log) sometimes indicate whether your queries timeout or not. You can then identify slow queries and fix them.

CF does closes the connections after the timeout. CF reuses the idle/unclosed connections. When required unless the connection is still busy executing some query. If the query execution is taking too long, it has to be a problem with either the application or the database. The idle connections are being re-used, as and when required.

Technical details about the timeout:
We take two parameters in admin for this – Timeout and interval.
CF closes a maximum of 5 timed out connections at each interval. Say for example, If we have 20 open connections with timeout being set to 10 and interval being set to 5, then CF will close:-
0 connections after 5 mins
5 connections after 5 more mins
5 more connections after 5 more mins
5 more connections after 5 more mins
5 more connections after 5 more mins

So to close all the connections (as per above calculation), CF will take at least 25 mins to close, all open connections. The maximum limit of closing 5 timeout connections is not configurable and is by design.

The optimized value for timeout can be set 5 and interval to 1. You can configure them further, as per your application requirement. You can change the database timeout value in CF administrator in Advanced settings of Data & Services > Datasources to optimize idle/unclosed connections.

Network latency
If the application code resides and being accessed from a shared drive in ColdFusion Application, network latency can cause slow request processing, resulting in performance issues. This can even cause a server to crash/unresponsive. Its highly recommended to check your internal Network throughput. You can also refer to the information available on below blog:
http://blogs.coldfusion.com/source-code-deployed-on-network-path-identifying-network-latency/

You may try below :
Add the jvm arguments below to speed up the the processing of cfm pages on network/shared location:
-Dsun.io.useCanonPrefixCache=true -Dsun.io.useCanonCaches=true.

Note that 30 sec is default timeout.

This enables canonical cache that caches the canonical path of a file. This helps, when there are a lot of threads waiting to get path from WinNTFileSystem. While accessing files from a network drive, each “getCanonicalPath” would end up going to network and would become quite expensive task. Enabling this cache means that for same file, JVM would never go back to disk (till the time it is in cache) to find its path.

Linux random number generation:
Random number generation and server startup is slow on Unix platforms for some of the servers. This could be because, /dev/random is used in Unix platforms for random number generation.
java.security.SecureRandom is designed to be crypto secure. It provides strong and secure random numbers. SecureRandom should be used when high-quality randomness is important and is worth consuming CPU. We can add the below jvm argument, to get rid of performance issue due to random number generation:
-Djava.security.egd=file:/dev/./urandom

Security scanner:
If you see CPU spikes at some specific time of the day/week, this could be due to a third party security scanner interfering with your ColdFusion application. The scanner hits the server monitoring port 5500 (by default) with 0.0.0.0, which goes to infinite loop and causes server crash.

To fix this issue, we need to modify the jetty.xml at ColdFusionXXXX\cfusion\lib. Change the Server monitoring IP address from 0.0.0.0 to 127.0.0.1 and restart ColdFusion.

Code Cache:
If your program has high codecache memory set via -XX:ReservedCodeCacheSize, you can limit it by disabling code cache flushing. If flushing is disabled, the JIT does not compile methods after the codecache fills up and hence there won’t be CPU hikes. You can add the following jvm argument. This can be used to flush code cache.

XX:-UseCodeCacheFlushing
You can also disable tieredcompilation with below argument:
-XX:-TieredCompilation (Applicable only with Java 1.8. Java versions less than 8 doesn’t have tiered compilation enabled by default.)

Service unavailable error:
503 – Service unavailable is a generic error. Whenever we get this error, the first thing we should check is, whether ColdFusion is started and running or not. In case you experience intermittent 503’s, then its time to investigate the less responsive server, which might be dropping requests. This could be because of Long GC pauses or any reason that could delay response from ColdFusion server. The ColdFusion connector tuning can help us to overcome service unavailable error. Below blog post can be used to tune ColdFusion connector and avoid such errors.
http://blogs.coldfusion.com/coldfusion-11-iis-connector-tuning/

We have also seen some issues because of bugs in few specific update level of java. The best practice would be to keep your ColdFusion Java updated to latest version. Use the below blog to keep your java up to date.
http://blogs.coldfusion.com/installing-and-troubleshooting-java-updates-in-coldfusion/

ColdFusion thread dumps:
ColdFusion thread dumps can be used to analyze New, Runnable, Blocked, Waiting, Timed_Waiting and Running threads.
The issues such as Thread race, Deadlock, Hang IO calls, GC/OutOfMemory exceptions, Infinite Loop can be determined using the thread dumps. Following Blog can be used to take thread dump on a ColdFusion server:
http://blogs.coldfusion.com/taking-thread-dumps-from-coldfusion-server-programmatically/
If you are on CF11 update 12 and CF2016, you can skip copying threaddump.jar. And just use takethreaddump.cfm file to capture the thread dump.

Another issue we have seen in one or two cases, If the performance is impacted by XML parsing, the jvm argument below can fix it:
-Dcom.sun.xml.bind.v2.bytecode.ClassTailor.noOptimize=true

The other causes of performance issue include:
Lack of proper database SQL tuning & capacity planning
Application specific performance problems
Lack of proper data caching
Excessive data caching
Excessive logging

In case the above steps does not resolve the issue, please feel free to contact Adobe support (https://helpx.adobe.com/support/coldfusion.html) for analysis of the issue.

Some key points to remember:-

*Please note that, any changes made to jvm.config, would require a ColdFusion service restart.

Jvm.config: ColdFusionXXXX/instance_name/bin
ColdFusion logs: ColdFusionXXXX/instance_name/logs

Reference:
https://dzone.com/articles/java-8-permgen-metaspace
https://dzone.com/articles/top-10-causes-java-ee

ColdFusion 2016 Update 5 and ColdFusion 11 Update 13 released

This post is to announce the release of updates for ColdFusion 2016 and ColdFusion 11. These updates address a common vulnerability mentioned in security bulletin APSB 17-30.

ColdFusion 2016 Update 5

In addition to addressing the vulnerabilities in the security bulletin APSB17-30 this update includes 13 bug fixes in language, database and AJAX some other areas. For the installation instructions and details on the bugs fixed, refer this technote.

ColdFusion 11 Update 13

In addition to addressing the vulnerabilities in the security bulletin APSB17-30 this update includes 8 bug fixes in charting, AJAX and some other areas. For the installation instructions and details on the bugs fixed, refer this technote,

For the security fixes in these updates to be effective, ColdFusion 2016 should be on JDK 8 u121 or a higher version, and ColdFusion 11 should be on JDK 8 u121 or JDK 7 u131 or a higher version of JDK. The use of latest JDK update is recommended.

On a standalone installation of ColdFusion, you can upgrade Java by editing the jvm.config file at <cf_root>/cfusion/bin. For a JEE installation of ColdFusion, refer the documentation for the host application server.

Adobe ColdFusion Developer Week :: July 31st – Aug 4th

We are pleased to bring you ColdFusion Developer Week once again this year from 31st July – 4th August with some great topics. ColdFusion Developer week is a full week of completely FREE webinars (2-a-day) put on by the Adobe team. Register Now at https://cfdevweek.meetus.adobeevents.com/.

We got some great topics from what is planned for the future versions, REST, API manager, Language changes and much more. All the sessions would be held remotely and the recordings would be available later in case you couldn’t make it at the scheduled time. Please try to attend the live sessions as it would be more engaging and you would be able to talk directly to the developers who worked on the features.

The session timings would be 9:00 AM and 12:00 PM Pacific Time, Monday through Friday

Schedule and Topics:

Monday, July 31, 2017 9:00 – 10:00 am The now and the next of Adobe ColdFusion
12:00 – 1:00 pm API Economy: Realizing The Business Value of APIs Through Adobe API Management
Tuesday, August 1, 2017 9:00 – 10:00 am Design of Rest API and Rest in ColdFusion
12:00 – 1:00 pm Upgrade to ColdFusion 2016
Wednesday, August 2, 2017 9:00 – 10:00 am Finding security vulnerabilities in your code base using security analyzer
12:00 – 1:00 pm Load Balancing, Scalability and Failover with ColdFusion Clustering
Thursday, August 3, 2017 9:00 – 10:00 am PDF enhancements in ColdFusion 2016
12:00 – 1:00 pm Language enhancements in ColdFusion 2016
Friday, August 4, 2017 9:00 – 10:00 am Scale with ColdFusion 2016

 

Register Now!

ColdFusion 2016 API Manager Update 1 released

Update 1 for ColdFusion 2016 API Manager was released on 9th June 2017.

It introduces new features such as a JavaScript connector for configuring User Store, a new Token-based authentication support and packages enhancements like additional Swagger import options and logging for API request and response.

It also fixes several bugs in security, publisher portal and server core and workflows related to SOAP proxy, Swagger and SOAP to REST. For further details on the new features, the installation instructions and the manual download link refer this technote. For the list of bugs fixed with this update refer this webpage.

To be able to apply this update, ensure that you are using an API Manager installation from the latest API Manager installer. The installer was refreshed on Dec 13, 2016. An API Manager installation that has been installed with the new installer would be on build no. 301768.

Post update installation, the build for the API Manager should change to ColdFusion API Manager 2016,0,1,302960. You can validate this by using the apimanagerinfo (.sh/.bat) utility in <CFAPIM_root>/bin directory.

ColdFusion 2016 : Support for Windows Server 2016

We have updated the Windows 64-bit installer for ColdFusion 2016, to support Microsoft Windows Server 2016. The Add-on services installer and the .NET service installer for ColdFusion 2016 have also been refreshed.
You can access the server installer by clicking on the "free trial" or the "buy now" link in the ColdFusion product page at Adobe.com. You may download the aforementioned additional installers at the ColdFusion support page

The ColdFusion 2016 support matrix would be updated soon to reflect the support for the new platform.

The refreshed installer comes with the Update 3 baked-in. After installing the server, you may bring it up to the current update level, by installing Update 4. You can follow the instructions at this technote to download and install Update 4. If you need any help with installing ColdFusion server you may refer the installation instructions at this technote.

The installers for ColdFusion Builder 2016 and ColdFusion API Manager would soon be refreshed to support Windows Server 2016.

ColdFusion 2016 Update 4, ColdFusion 11 Update 12 and ColdFusion 10 Update 23 released

This post is to announce the release of the following ColdFusion updates:

ColdFusion 2016 Update 4

ColdFusion 2016 Update 4 upgrades Tomcat to version 8.5.11.0 and fixes 115 bugs (including 52 external bugs) in areas such as Security, Language, Charting and Performance. This update also addresses vulnerabilities mentioned in the security bulletin APSB17-14.  For details and instructions on how to apply this update refer this technote.

ColdFusion 11 Update 12

ColdFusion 11 Update 12 upgrades Tomcat to version 7.0.75. It also addresses vulnerabilities mentioned in the security bulletin APSB17-14 and fixes 59 bugs (including 28 external bugs) related to areas such as AJAX, Charting and Language. For details and instructions on how to apply this update refer this technote.

ColdFusion 10 Update 23

ColdFusion 10 Update 23 upgrades Tomcat version to 7.0.75. This update addresses vulnerabilities mentioned in the security bulletin APSB17-14 and includes a total of 17 bug fixes (including 7 external bugs) related to Language, Charting, Scheduler, Document Management and certain other areas. For details and instructions on how to apply this update refer this technote.

The build number after applying thse updates should be:

2106,0,4,302561 for ColdFusion 2016;
11,0,12,302575 for ColdFusion 11.
10,0,23,302580 for ColdFusion 10.

Note:

  • Support for Windows Server 2016 will be introduced with the refreshed full ColdFusion 2016 server installer which will be made available shortly. Update: The new installer is now available, as of Apr 28.
  • The core support for ColdFusion 10 effectively ends on May 16, 2017. It will, therefore, receive no further updates. For detailed support timelines, see this EOL matrix.

 

 

ColdFusion Nginx Connector – Initial Performance Numbers

We recently announced the availability of Nginx connector for ColdFusion 2016. This post talks about the initial performance numbers we have seen with ColdFusion and Nginx. The numbers seen are only indicative. Continued efforts are being put to improve performance on Nginx Connectors.

The Specifics on installing and configuring connector are listed in the following document,
http://cfdownload.adobe.com/pub/adobe/coldfusion/nginx/prerelease/v7/Configuring_Nginx_with_ColdFusion.pdf

 

Nginx Optimizations:
Before collecting performance numbers, a handful of optimizations were done to the Nginx Configurations. The Nginx Tuning Guide was referred before making these changes.

  • Updating the number of worker processes to use one worker per CPU.
    • worker_processes  auto;
  • Worker connections count updated to an appropriate value.
    • worker_connections 1024;
  • Keep Alive request count updated to 150.
    • keepalive_requests  150;
  • Connection queue size updated in /etc/sysctl.conf,
    • net.core.somaxconn = 65536
    • net.ipv4.tcp_max_tw_buckets = 1440000

 

Baselines and Performance Numbers:

To comparatively measure the performance of the Nginx connector, we collected baselines for the following configurations:

  • ColdFusion – Vanilla ColdFusion 2016 with requests served by the bundled Tomcat server.
  • Nginx Proxy – The traditional method to configure ColdFusion with Nginx. The nginx.conf file used for this purpose can be downloaded here.  
  • Apache Connector – Amongst the supported webservers, Apache connector is the most suitable due to parity with platforms.

Baselines were captured for two CFM pages, one, a simple Hello World, and another, a more complex CFM page which uses getRealPath, CGI variables, references static content and invokes a CFC via REST.

All requests were executed with 100 concurrent threads for the duration of 180 seconds and averaged over 3 executions. The results are listed below.

 

Simple CFM – Hello World     

 

Throughput
(req/sec)

ART
(ms)

ColdFusion

10088.80

8.67

Nginx Proxy

10301.27

9

Apache Connector

9933.70

9.33

Nginx Connector

9427.77

9.66

Nginx Connector with CFM handler only

10097.13

9

 

Complex CFM – Includes static content, getRealPath, CGI, POST Request

 

Throughput
(req/sec)

ART
(ms)

ColdFusion

36.90

2658.67

Nginx Proxy

37.13

2650.67

Apache Connector

37.53

2620.33

Nginx Connector

37.37

2635.67

 

 

 

 

 

 

 

Going by the initial numbers, performance on Apache stands uncontested. Nginx Proxy shows better performance than the Nginx connector when smaller payloads are being processed, with ColdFusion doing all the processing. This is because ColdFusion registers a large number of handlers, as seen in the connector configuration file, ‘ajp_location.conf’. Having only the CFM handler registered, results in numbers very similar to Nginx Proxy.

The other benefits that the Nginx connector provides, that Nginx Proxy does not, are CGI scope variables support and Search-Engine-Safe URL support. Additional restrictions such as having a unified webroot for ColdFusion and Nginx also come into play with Nginx Proxy, making it an impractical solution for deployments.

While we continue to work on the connector performance, do share your feedback and inputs on using the Nginx connector. 

ColdFusion 2016 installer refreshed.

The server and express installers for Adobe ColdFusion (release 2016) have been refreshed. The installers are available for download at the ColdFusion product page at www.adobe.com. The new installer includes the following changes:

  • The API Manager installer is decoupled from the ColdFusion Server installer.
  • The new API Manager installer incorporates certain new features such as multi-tenancy, enhanced security, configurable policies, a dedicated update mechanism and support for Redis cluster and request/response compression. For a detailed description of these new features follow the links embedded in this technote. The API Manager installer would be made available very soon. We will update this post to share the location where the installer would be hosted.
  • The ColdFusion installer incorporates ColdFusion 2016 Update 3 and updates JDK to version 1.8.0_112. For details on the changes that went out with Update 3 refer the Update 3 Release Notes document. The build number for this installation should be 2016,00,03,301771.
  • The features listed below have been retired from the product and no longer ship with ColdFusion. For a detailed overview of the affected areas, refer the "Portlets" and "YUI and Spry" sections of the coldfusion-deprecated-features technote. In case you need to use any of these libraries you can download them from locations mentioned below.
    • Portlets. download (md5 checksum : 93273a7b4ab8c650e5fa9cece518e099);
    • YUI. download (md5 checksum : 827e0f8395d176ac28f46ed5e78004fd);
    • Spry. download (md5 checksum : 750c275c20b291f00c1ba92c855a09d7).

 To integrate the downloaded library, follow the instructions below:

  1. Stop ColdFusion sever.
  2. Download the libraries from the links mentioned above.
  3. Extract the downloaded files to the following locations:
    • Extract portlets.zip file to <cf_root>/cfusion directory. Update the web.xml file at <CF_HOME>/cfusion/wwwroot/WEB-INF to re-introduce the mappings mentioned in the "Portlets" section of this technote.
    • Extract yui.zip and spry.zip to ColdFusion's webroot at <cf_root>/cfusion. If your scripts directory is mapped to a non-default location (setting at CF admin > Settings > Default ScriptSrc Directory), unpack the zipped package manually and place it in the custom location following the structure in the package.
  4. Restart ColdFusion server.

If you are restoring just the YUI or Spry libraries, restarting the ColdFusion server is not required.

Revisions

20 Dec, 2016 –  added the web.xml mappings step in restoring portlets instruction. added reference to coldfusion-deprecated-features article.