Entries Tagged as 'Tuning'

Memory management is the process of recognizing, when allocated objects are no longer needed, deallocating (freeing) the memory used by such objects, and making it available for subsequent allocations. Since ColdFusion is an application server, ColdFusion requires memory management for effective performance improvement.

 

Sun Java differentiates memory across five generations:

  1. Young Generation: This generation holds objects with a high mortality rate. In ColdFusion, usually this space holds a variable that only exists in one page, for example – variable scope.
  2. Tenured Generation: This generation holds objects that live for a longer duration. For example, in ColdFusion, client and session scope variables.
  3. Permanent Generation: This generation holds all the reflective data of the virtual machine itself, such as class and method objects.
  4. Minor collections: This type of Garbage Collection (GC) occurs in the Young generation. In ColdFusion, minor collection collects unreferenced local scope and variable scope variables
  5. Major collection: This type of GC occurs in tenured generation. In ColdFusion, this collection collects all unreferenced session scope variables.

 

We can optimize the amount of memory allocated to ColdFusion server, for better performance. Let us discuss some of the parameters, which will help in Memory tuning.

  1. -Xmx – Defines maximum Java heap size.
  2. -Xms – Defines minimum Java heap size.
  3. MaxPermSize / MaxMetaspaceSize: The JDK 8 JVM uses native memory for the representation of class metadata and is called Metaspace. Users with JDK1.7 or lower will see the attribute MaxPermSize, and users with JDK 8 will see the attributes MaxMetaspaceSize.

 

Garbage Collection

Garbage collection refers to the process of recycling memory that was assigned to objects no longer referenced by a program. Garbage collection can be divided into 2 types:

  • Minor collection: Garbage collection that occurs in the young generation. In ColdFusion, minor collections will collect unreferenced local scope and variable scope variables
  • Major collection: Garbage collection that occurs in tenured generation. In ColdFusion, this will collect all unreferenced session scope variables.

 

Concurrent Collector

If the sites on your server are primarily lightweight request/response applications where response times are critical, then the concurrent garbage collector is a viable choice for your server. This collector does its best to collect dead objects from memory while applications on the server are still processing data. This adds a small bit of overhead for transactional applications, but it helps applications maintain their response-time SLAs.  To enable concurrent collector You need to add one of the following jvm arguments in jvm argument -XX:+UseConcMarkSweepGC  or -XX:+UseG1GC

For more information on concurrent collector please click here

Parallel Collector

If the applications on your server are memory-intensive and allocate a lot of objects, then the parallel garbage collector is a viable choice for your server. Any application performing image manipulation or creating a lot of objects is a great candidate for the throughput collector. To enable this garbage collector, add the following JVM arguments in the jvm.config: -XX:+UseParallelGC

 For more information on Parallel collector please click here  

 

Where to make the chnages?

You can make the changes at ColdFusion administrator (Settings >>Java and JVM >> JVM Arguments). 

Alternatively, you can tune memory in jvm.config, located at ColdFusion install directory (cf_root >> instance name >> bin >> jvm.config). 

Note: It’s always recommended to take a backup of jvm.config before making any changes.

 

Sever Monitor

If you have an Enterprise edition of ColdFusion, you can use Sever Monitor to monitor your server. You need to go to ColdFusion Administrator >> Server Monitor. To Start and Stop Monitoring, profiling, and memory tracking, click the appropriate buttons o of the server monitor.

Let us discuss about the Server Monitor in detail.

  1. Start Monitoring: This feature gathers information about all requests, including active requests, slowest requests, active sessions, cumulative server, etc. The Server Monitor does not gather information for requests that are excluded on the filter settings page.
  2. Start Profiling: It starts tag and function timing information for the slowest request reports, active queries, slowest queries, cached queries, and query cache status. You can view about each request that is slow or consumes a lot of memory. You can use this feature on development server. We do not recommend you to use this feature in a production environment.
  3. Start Memory Tracking: It gathers information about memory consumption, including all memory usage, queries and session that use most memory. You must enable profiling to view query-related reports.

 

You need to see if there is any effect on non-heap or heap memory at all. What needs to be deduced from the graph is, if the non-heap memory increases then, the perm gen/metaspace value needs to be recalculated. Whereas, if the heap memory increases, then the Xmx needs to changed.

You can also take help from some 3rd party software to monitor your server like GC viewer. You can also use Jconsole.  It is another memory management tool provided within JDK. The usage information is available at https://docs.oracle.com/javase/8/docs/technotes/guides/management/jconsole.html.

Follow the same procedure, mentioned above for non-heap or heap memory monitoring and tuning.

 

Many a time, ColdFusion application code is deployed on a network path when your ColdFusion deployments are of large-scale and mandated to use network paths.

After setting up the server for the first time, if there is any performance hit, as the first thing you would want to cross-check few things. One of the things to determine is if there is any network latency.

Though you would have got same network within your organization same as earlier, your OS version also would have changed.

 

Follow the steps below to see if the performance hit is due to network latency-

When the server is under moderate or full load(with at least 8-10 requests under process), take 2 or 3 thread dumps with 30 seconds interval.

It is not appropriate to take thread dump when the server has negligible load and anlyze that as there may not be any in-process requests.

If you are not sure how to take thread dump, you can simply follow the following blog.

( Taking Thread Dumps From ColdFusion Server Programmatically )

Open the thread dump file:

Under moderate or full load server conditions, if you see more than 5-8% of running ColdFusion threads containing “WinNTFileSystem” in the thread’s stack trace --> It means that there is lot of time being spent in trying to resolve the application file paths.

Following are the sample threads having WinNTFileSystem in its dump.

"ajp-bio-8014-exec-6861" Id=13898 in RUNNABLE
 java.lang.Thread.State: RUNNABLE
 prio=5 blockedtime=28963 blockedcount=6819 waitedtime=421762 waitedcount=115
    at java.io.WinNTFileSystem.getBooleanAttributes(Native Method)
    at java.io.File.isFile(File.java:876)


"ajp-bio-8014-exec-6861" Id=13898 in RUNNABLE (running in native)
 java.lang.Thread.State: RUNNABLE
 prio=5 blockedtime=28961 blockedcount=6814 waitedtime=421762 waitedcount=115
    at java.io.WinNTFileSystem.canonicalize0(Native Method)
    at java.io.Win32FileSystem.canonicalize(Win32FileSystem.java:414)
    at java.io.File.getCanonicalPath(File.java:618)


(Note: ColdFusion threads can be identified by the name starting with "ajp-" )

For Example, if there are 50 threads with thread name starting "ajp-bio-" in the thread dump, if you see WinNTFileSystem in more than 2-3 threads, it is the time you start looking at minimizing the network latency.

 

Once you know there is latency, you would want to know how much is the latency when compared to the application existing locally.

Created a very basic network latency test program to validate this.

You can take the jar from here.

And run it from command prompt as follows:

> C:\ColdFusion11\jre\bin\java -jar <Path of NetworkPathsTest.jar> <Network or Local Directory Path >

If the network path (Ex:- \\orgserver\d$) is accessible only to the ColdFusion service user, open command prompt as that user ( runas /user:<cfserviceaccount domainname>\cfserviceusername CMD )

Examples:

Path Arguments can be one or more. More Path arguments is a good measure to see the difference clearly.

C:\ColdFusion11\jre\bin\java -jar C:\ColdFusion11\NetworkPathTest.jar \\orgserver\d$\deploy\cfm\

C:\ColdFusion11\jre\bin\java -jar C:\ColdFusion11\NetworkPathTest.jar \\orgserver\d$\deploy\cfm\ \\orgserver\d$\deploy\cfm\api\

Try the same paths keeping the content same on the local machine and see the time differences.

For the same paths on local and remote, the difference in time should not be exponential.

These tests are to be performed on your ColdFusion server machine.

Once you have validations and found any latencies, it is the time to call for network optimization expertise.

 


Connector tuning is an essential part of setting up a ColdFusion server. There are various configurations in connector that needs to be tuned. Incorrect values may lead to “Service Unavailable” or “Server too busy”. In this blog, we will discuss how to handle such errors caused by incorrect tuning and how to tune the connectors for the site correctly.

The connector setting may vary from site to site. It is very important to configure the connectors for your application appropriately. This blog will include connector tuning parameters for IIS. During installation, user can choose to configure connector for “Individual Site” or “ALL” sites in IIS connector configuration.

Configure Web Server

After the installation, the user can launch the “Web Server Configuration tool” and has the availability to create the connector for “Individual Site” or “ALL” sites in IIS.

Add Web Server configuration

When connector is configured with individual sites, separate connector for each site will be placed under {CF-Home}/config/wsconfig/{some no}/. Similarly for “ALL” configuration the connector is configured at global level, which means the same connector binary will be used across multiple sites.

The three most important parameters will be discussed here and will help us to understand the role of the same:-

  • Reuse Connections
  • Connection pool size
  • Connection pool timeout

Re-use connections: - This setting determines the count of connections that can be re-used. When Tomcat connector makes a connection with Tomcat server, it does not closes the connection even after it finished serving the request. Instead it keeps the connection active, so that for the next request, the same connection can be re-used. This increases the performance by minimizing the overhead of creating new connection with tomcat server for every request. This settings needs to be tuned for connector configured with multiple sites. The max value for the re-use connection is determined based on the number of sites configured with same CF server and the load on each site.

The default re-use connection is 250.

Connection pool size: - This setting determines the maximum number of connections that can be created in the connection pool. When multiple requests arrive to the connector from IIS, connector creates new connections in the connection pool only if there are no free connections available in the pool. The connector will not create a new connection if connections reach the connection pool size limit. When connector is configured with “ALL” sites, the same connection pool will be used to serve the request for all sites. So the default value of the connection pool size, works well with the single site configuration, but fails to work well with “ALL” site configuration in some scenarios. Hence this value should be increased carefully based on the need and number of sites that are present within IIS.

The default connection pool size is same as, which is 250.

Connection pool timeout: - This setting determines the timeout value (in seconds) for idle connections in connection pool. This value must be in sync with the connectionTimeout attribute of your AJP connector in Tomcat's server.xml.

The default timeout for connection is indefinite, if not set in server.xml explicitly.

There are other parameters which CF connector inherits from Tomcat AJP connector. Please find the details of those settings from AJP documentation (http://tomcat.apache.org/connectors-doc/reference/workers.html)

The worker.properties is available at {CF-Home}/config/wsconfig/{some no}/ and the server.xml can be found at {CF-Home}/cfusion/runtime/conf/. Below are the changes required to tune the Site:-

  • Open the worker.properties file, add below line as new entry worker.cfusion.connection_pool_size=500 (This is connection pool size inside connector which are available to handle request)
  • Tune the entry for max_reuse_connections to appropriate value based on number of site. Optimal value is connection_pool_size / {no of site}
  •  Add another entry in new line worker.cfusion.connection_pool_timeout=60 (This value is idle connection timeout (in seconds), when sites are not under load connections will be recycled back to IIS)
  • Now open the server.xml from {cf-home/cfusion/runtime/conf}, add/update the maxThreads=500 and connectionTimeout="60000" to connection node containing the AJP entry.
    <Connector port="8012" protocol="AJP/1.3" redirectPort="8445" tomcatAuthentication="false"/>
  • Now the AJP entry in server.xml should look like 
    <Connector port="8012" protocol="AJP/1.3" redirectPort="8445" tomcatAuthentication="false" maxThreads="500" connectionTimeout ="60000"> </Connector>

There can be multiple use cases. Let us consider three most widely used scenarios:-

  •  Connector created with “ALL” OR with “Individual” Site and single site in IIS
  • Connector created with “ALL” and multiple sites in IIS
  • Connector created with “Individual” site and multiple sites site in IIS

Use Case# 1: Connector created with “ALL” OR with “Individual” Site and single site in IIS

In an idle scenario, where the user has only one site (configured with ALL or individual connector) and not running under high load, the worker.properties, can look like this

worker.list=cfusion

worker.cfusion.type=ajp13

worker.cfusion.host=localhost

worker.cfusion.port=8012

worker.cfusion.max_reuse_connections=250

worker.cfusion.connection_pool_size=500

worker.cfusion.connection_pool_timeout=60

And server.xml should look like

<Connector port="8012" protocol="AJP/1.3" redirectPort="8445" tomcatAuthentication="false" maxThreads="500" connectionTimeout ="60000"> </Connector>

So, we added the connection_pool_size and connection_pool_timeout (in seconds) in the worker.properties. The corresponding connectionTimeout (in milliseconds) is added to server.xml along with maxThreads whose value is equivalent to the connection_pool_size in the worker.properties.

Use Case# 2: Connector created with “ALL” and multiple sites in IIS

Consider a scenario that the connector is created with “ALL” and there is only one site which is running under load. The default 250 re-use connections are utilized by site 1. Later on, the user adds another site in IIS.

Site 1 will make all 250 re-usable connections with ColdFusion and any request for new connection from site 2 will be ignored by ColdFusion. Hence it is required, to increase the re-use connection count to optimal value, so that site 2 does not starve for new connections. This can be achieved by configuring optimal value of max_reuse_connections count. Considering that the site 2 is not running under high load, 100 re-use connection will work. So the max_reuse_connections becomes 350 {250 (for site 1) + 100 (for site 2)}. But, it is a good practice, to start tuning the connection_pool_size first, and then the max_reuse_connections appropriately.

This case would require connection_pool_size=700, as max_reuse_connections= connection_pool_size / {no of site}. So, the worker.properties will look like this

worker.list=cfusion

worker.cfusion.type=ajp13

worker.cfusion.host=localhost

worker.cfusion.port=8012

worker.cfusion.max_reuse_connections=350

worker.cfusion.connection_pool_size=600

worker.cfusion.connection_pool_timeout=60

And server.xml should look like

<Connector port="8012" protocol="AJP/1.3" redirectPort="8445" tomcatAuthentication="false" maxThreads="700" connectionTimeout ="60000"> </Connector>

Note: The connectionTimeout is in milliseconds

Use Case# 3: Connector created with “Individual” site and multiple sites site in IIS

Consider a scenario that the individual connectors are created for each site. There are three sites - Site 1 is running under high load, site 2 and site 3 running are under low load. For all the sites, there are individual connectors. Now, ideally in this scenario, we should start tuning with the site running under high load first. We can disable the timeout for high traffic sites, if we are not sure for timeout. If not defined, the default timeout for connection is indefinite. To start with, don’t specify the re-use parameter. Set the connection_pool_size=500 and monitor the site. Gradually increase the value by 100 and likewise, till the site is stable. Say, at connection_pool_size=800, the site is stable. Now, set the max_reuse_connections=270 (connection_pool_size / {no of site} i.e. 800/3=270 approx)

Site 1

worker.list=cfusion

worker.cfusion.type=ajp13

worker.cfusion.host=localhost

worker.cfusion.port=8012

worker.cfusion.max_reuse_connections=270

worker.cfusion.connection_pool_size=800

worker.cfusion.connection_pool_timeout=60

Site 2 and site 3 are running under low traffic, but are bind to same ColdFusion instance (cfusion in this case). The below settings should be optimal:-

Site 2

worker.list=cfusion

worker.cfusion.type=ajp13

worker.cfusion.host=localhost

worker.cfusion.port=8012

worker.cfusion.max_reuse_connections=100

worker.cfusion.connection_pool_size=250

worker.cfusion.connection_pool_timeout=60

Site 3

worker.list=cfusion

worker.cfusion.type=ajp13

worker.cfusion.host=localhost

worker.cfusion.port=8012

worker.cfusion.max_reuse_connections=100

worker.cfusion.connection_pool_size=250

worker.cfusion.connection_pool_timeout=60

And server.xml should look like

<Connector port="8012" protocol="AJP/1.3" redirectPort="8445" tomcatAuthentication="false" maxThreads="1300" connectionTimeout ="60000"> </Connector>

Note: The connectionTimeout is in milliseconds and the maxThreads is the value equivalent to summation of all the connection_pool_size(s). So, in this case maxThreads=1300 {800 (for site 1) + 250 (for site 2) + 250 (for site 3)}.

Some key points to remember:-

  • The connector tool should always run with “Run as Administrator” feature, even if the user is from the Administrator group i.e.

    Using the command line:-

    Connector Tool cmd

    Using the GUI

    Connector Tool gui

  • Any changes made to {CF-Home}/config/wsconfig/{some no}/, including isapi_redirect.dll or worker.properties, would require an IIS restart.
  • Any changes made to {CF-Home}/cfusion/runtime/conf/server.xml requires “ColdFusion 11 Application Server” service restart.
  • max_reuse_connections should always be less than or equivalent to connection_pool_size. It can’t be larger than the connection_pool_size.
  • The above use cases are scenario based and may vary from site to site, depending upon the load, architecture and traffic on the site.

In a previous video by Asha from Adobe team,we had discussed the various details about the Apache connector in ColdFusion 10.

In this video , Adobe engineer Kiran talks about configuring IIS with ColdFusion 10

 

You can find out more about it here

-Viny

Replacing JRun with Tomcat is a fundamental change in ColdFusion 10 and with that comes the new connectors for IIS and Apache. That also makes it very important to know the different configuration options available for the connectors as an incorrectly configured connector can easily lead to “Service Not available” or “Server is too busy” errors for your site.

During IIS connector configuration, user can choose to configure connector for individual sites or for “ALL” sites.

When connector is configured with individual sites, separate connector for each site will be placed under {CF-Home}/config/wsconfig/{some no}/ , Similarly for “ALL” configuration the connector is configured at global level, which means the same connector binary will be used across multiple sites.

In most of the cases server administrators wish to go for configuring connector for “ALL” sites as it is very convenient.

In this blog post we will talk what are the parameters needs to be tuned appropriately to make CF connector works flawlessly.

I will be covering three most important parameters which decide the scalability of the connector.

  • Reuse Connections
  • Connection pool size
  • Connection pool timeout

There are other parameters which CF connector inherits from Tomcat AJP connector. Please find the details of those settings from AJP documentation (http://tomcat.apache.org/connectors-doc/reference/workers.html )

Connection pool size: - This setting determines the maximum number of connections that can be created in the connection pool. When multiple requests arrive to the connector from IIS, connector creates  new connections in the connection pool only if there are no free connections available in the pool.  The connector will not create a new connection if connections reach the connection pool size limit.

Re-use connections: - This setting determines the count of connections that can be re-used.  . When Tomcat connector makes a connection with Tomcat server, it does not close the connection even after it finished serving the request. Instead it keeps the connection alive, so that for the next request, the same connection can be re-used. This increases the performance by minimizing the overhead of creating new connection with tomcat server for every request.

Note: - The default re-use connection count is same as connection pool size which is 200.

Connection pool timeout: - This setting determines the timeout value (in seconds) for idle connections in connection pool.  This value must be in sync with the connectionTimeout attribute of your AJP connector in Tomcat's server.xml.

Let us discuss more about how these 3 parameters can be tuned properly to make your server with varying load on your server.

1.       Connection pool size: - When connector is configured with “ALL” sites, the same connection pool will be used to serve the request for all sites.  So the default value of the connection pool size, works well with the single site configuration, but fails to work well with “ALL” site configuration in some scenarios. Hence this value should be increased carefully based on the need and number of sites that are present in IIS. 

2.       Max Re-use connections: - This settings needs to be used for connector configured with multiple sites.  The max value for the re-use connection is determined based on the number of sites configured with same CF server and the load on each site.

Let us consider use case where site 1 configured with CF server, now site 2 is also configured with the same CF server. By default each connector will have 200 re-use connections.

Now consider site 1 is running under heavy load make all 200 re-usable connections with CF server. Now CF server is not left with any new connection in its connection pool, hence any request for new connection from site 2 will be ignored by CF server.

 Hence it is required to make site 1 re-use connection count  to optimal value, so that site 2 does not starve for new connection. This can be achieved by configuring optimal value of max re-use connection count.

For the above use case, if site 1 is allowed to use 100 re-usable connections, there will be 100 more connections available for site 2 at the tomcat server connection pool.

3.       Connection pool timeout: - This timeout value helps in recycling connections that are being re-used when they are not used for a long time.  This is proved very useful when sites runs under varying load and makes overall improvement in the server performance. By default timeout for connection is indefinite.

Below we will discuss steps to follow to configure the above parameters

  • Go to worker.properties file (inside connector installer folder) add below line as new entry worker.cfusion.connection_pool_size = 500 (This is connection pool size inside connector which are available to handle request)
  • Add another entry in new line worker.cfusion.connection_pool_timeout = 60 (This value is idle connection timeout, when sites are not under load connections will be recycled back to IIS)
  • Tune the entry for max_reuse_connections to appropriate value based on number of site. Optimal value is connection_pool_size / {no of site}
  • Now open the server.xml from {cf-home/cfusion/runtime/conf}, add/update the maxThreads=500 and connectionTimeout="60000" to connection node containing the AJP entry. Now the AJP entry in server.xml should look like  <Connector port="8012" protocol="AJP/1.3" redirectPort="8445" tomcatAuthentication="false" maxThreads="500"  connectionTimeout ="60000">  </Connector>
  • Restart the IIS and ColdFusion server to reflect the above changes.