ColdFusion 11 IIS Connector Tuning

May 22, 2014 / Anit Kumar Panda 41 Comments

  Performance | Administrator | Adobe ColdFusion 11 | Connector | General | Splendor | Tuning

Connector tuning is an essential part of setting up a ColdFusion server. There are various configurations in connector that needs to be tuned. Incorrect values may lead to “Service Unavailable” or “Server too busy”. In this blog, we will discuss how to handle such errors caused by incorrect tuning and how to tune the connectors for the site correctly.

The connector setting may vary from site to site. It is very important to configure the connectors for your application appropriately. This blog will include connector tuning parameters for IIS. During installation, user can choose to configure connector for “Individual Site” or “ALL” sites in IIS connector configuration.

Configure Web Server

After the installation, the user can launch the “Web Server Configuration tool” and has the availability to create the connector for “Individual Site” or “ALL” sites in IIS.

Add Web Server configuration

When connector is configured with individual sites, separate connector for each site will be placed under {CF-Home}/config/wsconfig/{some no}/. Similarly for “ALL” configuration the connector is configured at global level, which means the same connector binary will be used across multiple sites.

The three most important parameters will be discussed here and will help us to understand the role of the same:-

Re-use connections: - This setting determines the count of connections that can be re-used. When Tomcat connector makes a connection with Tomcat server, it does not closes the connection even after it finished serving the request. Instead it keeps the connection active, so that for the next request, the same connection can be re-used. This increases the performance by minimizing the overhead of creating new connection with tomcat server for every request. This settings needs to be tuned for connector configured with multiple sites. The max value for the re-use connection is determined based on the number of sites configured with same CF server and the load on each site.

The default re-use connection is 200.

Connection pool size: - This setting determines the maximum number of connections that can be created in the connection pool. When multiple requests arrive to the connector from IIS, connector creates new connections in the connection pool only if there are no free connections available in the pool. The connector will not create a new connection if connections reach the connection pool size limit. When connector is configured with “ALL” sites, the same connection pool will be used to serve the request for all sites. So the default value of the connection pool size, works well with the single site configuration, but fails to work well with “ALL” site configuration in some scenarios. Hence this value should be increased carefully based on the need and number of sites that are present within IIS.

The default connection pool size is same as, which is 200.

Connection pool timeout: - This setting determines the timeout value (in seconds) for idle connections in connection pool. This value must be in sync with the connectionTimeout attribute of your AJP connector in Tomcat's server.xml.

The default timeout for connection is indefinite, if not set in server.xml explicitly.

There are other parameters which CF connector inherits from Tomcat AJP connector. Please find the details of those settings from AJP documentation (http://tomcat.apache.org/connectors-doc/reference/workers.html)

The worker.properties is available at {CF-Home}/config/wsconfig/{some no}/ and the server.xml can be found at {CF-Home}/cfusion/runtime/conf/. Below are the changes required to tune the Site:-

There can be multiple use cases. Let us consider three most widely used scenarios:-

Use Case# 1: Connector created with “ALL” OR with “Individual” Site and single site in IIS

In an idle scenario, where the user has only one site (configured with ALL or individual connector) and not running under high load, the worker.properties, can look like this

worker.list=cfusion

worker.cfusion.type=ajp13

worker.cfusion.host=localhost

worker.cfusion.port=8012

worker.cfusion.max_reuse_connections=250

worker.cfusion.connection_pool_size=500

worker.cfusion.connection_pool_timeout=60

And server.xml should look like

<Connector port="8012" protocol="AJP/1.3" redirectPort="8445" tomcatAuthentication="false" maxThreads="500" connectionTimeout ="60000"> </Connector>

So, we added the connection_pool_size and connection_pool_timeout (in seconds) in the worker.properties. The corresponding connectionTimeout (in milliseconds) is added to server.xml along with maxThreads whose value is equivalent to the connection_pool_size in the worker.properties.

Use Case# 2: Connector created with “ALL” and multiple sites in IIS

Consider a scenario that the connector is created with “ALL” and there is only one site which is running under load. The default 200 re-use connections are utilized by site 1. Later on, the user adds another site in IIS.

Site 1 will make all 200 re-usable connections with ColdFusion and any request for new connection from site 2 will be ignored by ColdFusion. Hence it is required, to increase the re-use connection count to optimal value, so that site 2 does not starve for new connections. This can be achieved by configuring optimal value of max_reuse_connections count. Considering that the site 2 is not running under high load, 100 re-use connection will work. So the max_reuse_connections becomes 300 {200 (for site 1) + 100 (for site 2)}. But, it is a good practice, to start tuning the connection_pool_size first, and then the max_reuse_connections appropriately.

This case would require connection_pool_size=600, as max_reuse_connections= connection_pool_size / {no of site}. So, the worker.properties will look like this

worker.list=cfusion

worker.cfusion.type=ajp13

worker.cfusion.host=localhost

worker.cfusion.port=8012

worker.cfusion.max_reuse_connections=300

worker.cfusion.connection_pool_size=600

worker.cfusion.connection_pool_timeout=60

And server.xml should look like

<Connector port="8012" protocol="AJP/1.3" redirectPort="8445" tomcatAuthentication="false" maxThreads="600" connectionTimeout ="60000"> </Connector>

Note: The connectionTimeout is in milliseconds

Use Case# 3: Connector created with “Individual” site and multiple sites site in IIS

Consider a scenario that the individual connectors are created for each site. There are three sites - Site 1 is running under high load, site 2 and site 3 running are under low load. For all the sites, there are individual connectors. Now, ideally in this scenario, we should start tuning with the site running under high load first. We can disable the timeout for high traffic sites, if we are not sure for timeout. If not defined, the default timeout for connection is indefinite. To start with, don’t specify the re-use parameter. Set the connection_pool_size=500 and monitor the site. Gradually increase the value by 100 and likewise, till the site is stable. Say, at connection_pool_size=800, the site is stable. Now, set the max_reuse_connections=270 (connection_pool_size / {no of site} i.e. 800/3=270 approx)

Site 1

worker.list=cfusion

worker.cfusion.type=ajp13

worker.cfusion.host=localhost

worker.cfusion.port=8012

worker.cfusion.max_reuse_connections=270

worker.cfusion.connection_pool_size=800

worker.cfusion.connection_pool_timeout=60

Site 2 and site 3 are running under low traffic, but are bind to same ColdFusion instance (cfusion in this case). The below settings should be optimal:-

Site 2

worker.list=cfusion

worker.cfusion.type=ajp13

worker.cfusion.host=localhost

worker.cfusion.port=8012

worker.cfusion.max_reuse_connections=100

worker.cfusion.connection_pool_size=250

worker.cfusion.connection_pool_timeout=60

Site 3

worker.list=cfusion

worker.cfusion.type=ajp13

worker.cfusion.host=localhost

worker.cfusion.port=8012

worker.cfusion.max_reuse_connections=100

worker.cfusion.connection_pool_size=250

worker.cfusion.connection_pool_timeout=60

And server.xml should look like

<Connector port="8012" protocol="AJP/1.3" redirectPort="8445" tomcatAuthentication="false" maxThreads="1300" connectionTimeout ="60000"> </Connector>

Note: The connectionTimeout is in milliseconds and the maxThreads is the value equivalent to summation of all the connection_pool_size(s). So, in this case maxThreads=1300 {800 (for site 1) + 250 (for site 2) + 250 (for site 3)}.

Some key points to remember:-


41 comments so far ↓

  • 1 Aaron Neff // May 23, 2014 at 1:15 AM
    Hi Anit,

    Thanks for updating this IIS connector tuning info for CF11. Just noting I also filed ER #3429152 for a 3rd option: "All - Individually" (to configure all sites each with their own connector).

    Thanks!,
    -Aaron
  • 2 Richard Herbert // May 28, 2014 at 6:27 AM
    Do you consider Apache hosted sites subject to the same issues and advice?

    If not, do you plan to produce a similar article for those of us using Apache.
  • 3 Anit Kumar Panda // May 28, 2014 at 3:16 PM
    @Richard, Apache would also need tuning with ColdFusion 10 and 11. We are planning to have a similar article for Apache as well, soon.
  • 4 Richard Herbert // May 29, 2014 at 3:22 AM
    Thank you. That would really help with a stability issues we're trying to get to the bottom of.
  • 5 Aaron // May 29, 2014 at 5:29 AM
    I don't understand max_reuse_connections. how is it different from connection_pool_size?

    why is it connection_pool_size / { number of sites}
    why does Use Case 3 not follow this rulw for sites 2 & 3?

    Is max_reuse_connections a custom Adobe extension as it is not on the AJP connector page
  • 6 Anit Kumar Panda // May 30, 2014 at 6:28 AM
    reuse_connections keeps the connection active, so that for the next request, the same connection can be used again, where as, connection_pool_size is a pool of connection that can be used, when multiple requests comes to connector. So one is no of connections in a pool and the other one is how many connections can be reused. Definitely, they are different from each other.

    The ajp13 protocol is packet-oriented. The web server communicates with the servlet container over TCP connections. To cut down on the expensive process of socket creation, the web server will attempt to maintain persistent TCP connections to the servlet container, and to reuse a connection for multiple request/response cycles.

    There is a reuse flag in AJP Tomcat, which is set to true. This TCP connection is then used to handle new incoming requests, within the same already establised connection, keeping it active. Since the "reuse" value is set to true for ColdFusion, there needs to be a parameter, that should take care of the maximum value for the same. That is called as "max_reuse_connections"
  • 7 Anit Kumar Panda // May 30, 2014 at 8:37 AM
    Regarding the Use Case3, the site with heavy traffic and load should follow the steps mentioned above (max_reuse_connections= connection_pool_size / {no of site}). Sites with low traffic, configured with same instance, should use the standard parameters or the one mentioned in the Use Case3 above, as the traffic is less.
  • 8 Helge Hetland // Jun 5, 2014 at 9:06 AM
    My scenario is:
    300+ sites on IIS 8 where some have very low traffic (even zero traffic) and some have high traffic.

    Connector created with “ALL” and multiple sites in IIS

    It´s impossible to know which site has the most traffic as some sites are running events where traffic is very high in short periods of time (during registration etc.)

    My setup now is
    worker.CL.max_reuse_connections=100
    worker.CL.connection_pool_size=600
    worker.CL.connection_pool_timeout=60

    If I were to follow the "formula" my max_reuse_connections would be very low (say 1-2) or my connection_pool_size would be very high.

    So what would your advice be on such scenarios?

    Thanks,
    Helge
  • 9 Anit Kumar Panda // Jun 5, 2014 at 9:58 AM
    @Helge, If you set the max_reuse_connection value to the least, your site will still work. But, every time a new request arrives, a new thread would be created (although existing threads are available). That would impact the performance.
    In your scenario, keep the max_reuse_connection to 200(default value). To start with, increase the connection_pool_size to a value between 800-1000 and monitor your site. If you experience "Service unavailable", gradually increase the pool size and tune the connector.

    Note: Remember to set the maxThreads in server.xml, as you change the connection_pool_size.
  • 10 Helge Hetland // Jun 5, 2014 at 11:28 PM
    I will change per your recommendations and see if things get more stable, we have a few "hangs" weekly where a server is unresponsive and not serving any requests.

    Thank you

    Regards,
    Helge
  • 11 Richard Herbert // Jun 10, 2014 at 1:11 AM
    Given that the suggested value of "max_reuse_connections" is a function of "connection_pool_size", how do you determine a value for "connection_pool_size"? What would be the consequences of setting it to a ridiculous value of "100000" or "2"? What factors influence the choice of the "connection_pool_timeout" value and what would be the consequences of similar extreme values?
  • 12 Anit Kumar Panda // Jun 11, 2014 at 10:44 AM
    @Richard "max_reuse_connections" is not a function of "connection_pool_size", they are dependent on each other. Change in one, would result in change in the settings of other. Setting a value "very high" would leave the resources engaged, even when they are not required. On the other hand, setting a value "too low", would establish a new connection for every request, instead of using the existing connection (as there are none or less available). This would impact the response time.

    Connection pool size is determined by your developer, based on the connection requests you are receiving. If you are not able to get an idea about the relevant/nearest value, then enable Metric logging (http://helpx.adobe.com/coldfusion/kb/coldfusion10-enable-metric-logging.html). This is the same for ColdFusion 11 as well. This would give you an idea about the “connection pool size”. And then based on that, you can tune the connection_pool_size.

    "connection_pool_timeout" indeed is a very important parameter. Extreme high values, will queue up the requests, without timing them out and the server will hang, throwing "Service Unavailable". A "very low" value in turn will kill the request, without even processing them.
  • 13 Richard Herbert // Jun 12, 2014 at 3:08 AM
    @Anit - Thanks for the prompt response.

    Does Adobe have any writings on how to interpret the metrics.log output and how that output would influence the "connection pool size" setting? After several hours of logging all I'm seeing is loads of...

    Max threads: 150 Current thread count: 4 Current thread busy: 0
  • 14 Anit Kumar Panda // Jun 13, 2014 at 8:40 AM
    Below is a sample output of metric log

    "Information","scheduler-0","01/02/14","14:11:23",,"Max threads: 500 Current thread count: 85 Current thread busy: 12 Max processing time: 29266063 Request count: 31045 Error count: 583 Bytes received: 642863 Bytes sent: 1061422778 Free memory: 529568096 Total memory: 1801977856 Active Sessions: 81"
    "Information","scheduler-3","01/02/14","14:12:23",,"Max threads: 500 Current thread count: 85 Current thread busy: 4 Max processing time: 29679670 Request count: 31682 Error count: 584 Bytes received: 653848 Bytes sent: 1085861435 Free memory: 256575784 Total memory: 1803091968 Active Sessions: 80"

    The Current thread count gives you an estimated value for connection pool size. For e.g, in the above scenario, based on the two threads, I would set the connection_pool_size >100 (more than 85). And then i will tune it further, if needed.
  • 15 Richard Herbert // Jun 16, 2014 at 1:31 AM
    @Anit - Interesting, thanks for sharing.

    So for my situation, where 99% of entries over a 120 hour period are as above and only 1 entry is > 9, your analysis would infer a value for connection_pool_size of, say, 12?

    When you say "tune it further if needed" do you mean, if you noticed that the "Current thread count" was starting to hit your new "connection_pool_size" value you would revise it upwards?

    Does the "Active Sessions" value have any bearing? I see yours are in the same order as your "Current thread count" value. Mine are 1200-2500. Does that sound right? E.g.

    "Information","scheduler-2","06/16/14","07:47:04",,"Max threads: 150 Current thread count: 4 Current thread busy: 0 Max processing time: 83425 Request count: 9072 Error count: 72 Bytes received: 1649 Bytes sent: 22768583 Free memory: 306065064 Total memory: 1052966912 Active Sessions: 1218"

    "Information","scheduler-2","06/16/14","08:37:04",,"Max threads: 150 Current thread count: 7 Current thread busy: 2 Max processing time: 84617 Request count: 9133 Error count: 72 Bytes received: 1649 Bytes sent: 22920761 Free memory: 147950448 Total memory: 1051000832 Active Sessions: 2565"

    I appreciate your support here but if there's a better medium for this discussion let me know.
  • 16 Anit Kumar Panda // Jun 17, 2014 at 6:56 AM
    Sure. I am sending you an email for further discussion.
  • 17 Charlie Arehart // Jun 25, 2014 at 7:09 AM
    Folks (Anit, and everyone reading this), about scenario 1 above, beware that it's not at all clear that the changes there entail a change from the default values currently in those files on initial configuration.

    Anit, I really think it would help people a lot if you made that more clear, even if just a sentence after the displayed changes. I have talked to more than one person who has misinterpreted things (they did not compare it to their current settings, so did not realize they ought to make any change at all, even if theirs was scenario 1).

    This is less an issue in scenario 2, as the text conveys a need to think through what values to use.

    I have some more thoughts on all this, again from helping people deal with trying to use the info, which I'll share later (hopefully today).
  • 18 Dave Cordes // Jun 25, 2014 at 11:21 AM
    I'm still confused about Use Case #2. If I have about 50 websites on my server and my connection pool size set to 1,000. Doing the math I should set my max reuse connections to 20. However, the majority of the websites on my server don't get a ton of traffic so I adjusted the value higher to 50. My thoughts are I could most likely set it to even higher like 100 but I don't know if that would be a bad decision or not. Anyone have an environment like myself and can comment on some appropriate settings?
  • 19 Carl // Jun 25, 2014 at 6:35 PM
    @Anit - Do you think rotating the connector log would be useful to apply? One could apply log rotating to minimise the log size which makes it easier to read either current or past log. Does a large log have an impact on performance over time once it grows say beyond 200Mb?

    EG sample isapi_redirect.properties:

    extension_uri= /jakarta/isapi_redirect.dll
    log_file= c:\ColdFusion11\config\wsconfig\1\isapi_redirect_%d-%m-%Y.log
    log_level= info
    log_rotationtime= 86400
    worker_file= c:\ColdFusion11\config\wsconfig\1\workers.properties
    worker_mount_file= c:\ColdFusion11\config\wsconfig\1\uriworkermap.properties
    iis_buffer_enable= true

    where log_rotationtime is specified in seconds so 24 hours in this case.

    Thanks in advance, Carl M.
  • 20 Anit Kumar Panda // Jun 26, 2014 at 7:44 AM
    @Charlie, I have added additional information for scenario 1, so that, it adds more sense to the users. Thanks for your suggestion and we welcome your inputs on this.

    @Dave Cordes, As mentioned above, In such scenarios, keep the max_reuse_connection to 200 (default value). To start with, increase the connection_pool_size to a value between 800-1000 and monitor your site. If you experience "Service unavailable", gradually increase the pool size and tune the connector.

    Once you are able to identify an appropriate value for connection_pool_size, then set the max_reuse_connection to somewhere between 100 to 200. If needed, increase it to a higher value, and monitor the site, till it becomes stable.

    Note: Remember to set the maxThreads in server.xml, as you change the connection_pool_size.

    @Carl, Log rotation can be implemented for connector.

    You can use both log_rotationtime and log_filesize, as per your requirement. This should not impact the performance, unless the logs become larger in size and occupy memory highly.
  • 21 BeekerMD03 // Jun 27, 2014 at 5:18 AM
    I recently enabled metrics and have been monitoring the log for over a week now. My log seems to be a bit different from what has been reported above by @Anit.

    For instance, please see a line from my log below:
    "Information","scheduler-1","06/27/14","08:10:14",,"Max threads: 150 Current thread count: 0 Current thread busy: 0 Max processing time: 0 Request count: 0 Error count: 0 Bytes received: 0 Bytes sent: 0 Free memory: 1746275528 Total memory: 2081423360 Active Sessions: 111"

    The "Current Thread Count", "Current Thread Busy", "Max processing Time", "Request Count", "Error Count", "Bytes Received", and "Bytes Sent" fields are always 0. Is there another setting that has to be enabled in order to record that data?

    The memory and Active Session columns appear to be accurately recording data.
  • 22 Anit Kumar Panda // Jun 27, 2014 at 5:30 AM
    @BeekerMD03, Can you ensure that the site is listening on the correct port. Please verify the Connector Port under Debugging & Logging > Debug Output Settings and restart ColdFusion 11 service.
  • 23 Charlie Arehart // Jun 28, 2014 at 10:46 PM
    @Anit, first, thanks for responding to my observation and adding that additional commentary in the blog. Great to see you responding (quickly and positively) to such suggestions. Indeed, thanks for being so responsive to all the questions folks are asking.

    This seems to be an issue that is striking a LOT of people, whether on CF10 or CF11.

    **One of the challenges, I think, is that we currently have no insight into the connection pooling: no metrics at all, whether from CF or Tomcat (or the web server, whether IIS or Apache) to tell us how many connections are used, by what site, etc.**

    If we had that info, I think it would be MUCH easier both for people to understand then when they are having the problem, and why, as well as perhaps how better to try to address it with the tuning knobs discussed here.

    Can you guys either expose something, or show folks how they could? I've tried to do a little digging but so far to no avail.

    For now, trying to do these web server tweaks is like the early days of CF6 and above, when it was first based on Java, and people were often finding their servers hanging up in ways it had not with CF 5 and before. This was before CF8 added the CF Server Monitor or tools like FusionReactor or SeeFusion became available, where we could really see what was going on.

    And instead people were for years taking stabs in the dark to solve problems: "change this jvm argument, change your heap size, change your gc algorithm, change your CF admin settings".

    Only when we had better diagnostics to help us understand what was REALLY going on in our servers. Perhaps it was that we were hitting the heap size limit, and now we had tools to tell us that (many could not get JVM-provided tools working for various reasons). And even then, with better tools they found out WHY the heap was high: too many sessions (often from spiders), or heavy use of query caching, or ehcache caching, etc.

    Or perhaps a different cause of a CF outage was hung requests, perhaps waiting for a CFHTTP call or CFQUERY, etc, which had nothing to do with jvm tweaking.

    My point is, only with better diagnostics did we really tend to get to the bottom of the problems. (Sadly, many still operate in the "point and shoot" mode of troubleshooting, and don't avail themselves of those valuable tools.)

    So I think we're in the same boat here, and I REALLY, REALLY hope that you (Adobe) or someone in the community will help us to really see what the true situation is on our servers with respect to these connections. Right now, it's again "throwing darts against the wall hoping to pop the balloon with a prize". We might win, we may have to keep throwing, and people are getting frustrated with that approach.

    Most important, this is only people who are finding this blog entry (or the old one for CF10) and trying these settings. There are many (perhaps most using CF 10 and 11) who may have this problem and NOT know of this blog entry.

    We really need CF not only to help us diagnose the problem, but even better would be if it would help us in some PROACTIVE way. Perhaps tracking the key connection metrics over time and providing a report in either the logs or the CF Admin (or even an email) to pre-empt the problem by recommending tweaks based on analysis over time.

    I realize the latter idea is total pie-in-the-sky dreaming. But this problem is so significant for so many, that it seems radical reconsideration of the problem and its solutions is warranted. Until them, anything will be better than the current situation.

    I have some more thoughts related to the tuning process and the blog entry, but I'll raise them in a separate comment, as I'm sure I lost some readers 10 paragraphs ago. :-(
  • 24 Charlie Arehart // Jun 28, 2014 at 11:49 PM
    @Anit, given that you have confirmed that you are in fact proposing a change to the last 3 lines you list for workers.properties, will you guys be changing how CF configures these value by default, going forward?

    I realize not everyone is in any single user case, but you clearly configure the current default values, which again all these recommendations here change, so it seems best to change the default, right? It seems you likely currently create them to default as for a use case 1, so why not set CF to create the values you propose here for that?

    And along those same lines, is there any chance you guys may consider evolving the web server configuration tool (and CF installer, which runs it implicitly if external web server integration is selected) so that they prompt folks to consider these various scenarios, to help them set better values than the current default, or even the single new default values as I just suggested?

    I realize I asked in the last comment for a tool to help determine better values over time. But this is about at least giving people a heads-up, at initial configuration, to consider the very ideas that you are proposing in this blog entry.

    And if you may consider changing that web server configuration tool, could you (while you're at it) change it so that (if on Windows) you require the user to use "run as administrator"? Right now, people have to KNOW to do that for it to work completely, and people often don't realize it, to their downfall (as it gives no error or warning. It just may do the job incompletely.).

    This idea of the tool detecting whether the user is running as admin (on Windows) is not outlandish: the CF10 Mandatory Update tool (also a java app) did the same thing.

    Finally, and perhaps most important, everyone reading this blog entry and applying the changes Anit suggests should note that THOSE CHANGES WILL BE LOST if the web server configuration tool is run again. It always sets them back to the current defaults (which Anit is telling us to change.) This is why I think it's important that the tool create better defaults, and/or prompt us to consider use cases and make better default choices.

    But perhaps better still, related to this last point, perhaps the tools should detect if users have made non-standard changes to these files (workers.properties and server.xml) with respect to these settings, and it should prompt them to either preserve them or let them be overwritten.

    As I said in the last comment, the problems caused for folks by the issues raised in this blog entry are really hurting a lot of CF10 and 11 users. Desperate times call for desperate measures: the web server config tool should be made a LOT smarter to help with this problem--and not hurt it by losing the hard-won tweaks that people may make.
  • 25 Dave Cordes // Jun 29, 2014 at 10:20 AM
    @Charlie - You make some great points. The worst part of running ColdFusion is that you don't really know what's going on under the covers. If ColdFusion had some way of alerting or logging these issues and providing recommendations it would certainly make the tuning process a lot easier. The Adobe recommendation of trying a certain setting and then waiting for it to fail sucks because I have clients to answer to. Once the server becomes unresponsive my phone rings off the hook. Again, if ColdFusion was more pro-active I think it would be a lot better. Maybe these features will be considered for ColdFusion 11 or possibly the next version.
  • 26 Anit Kumar Panda // Jun 30, 2014 at 7:08 AM
    @Charlie, It’s not easy to write paragraph(s) as notes on behalf of the users, and we appreciate your concerns w.r.t. ColdFusion and the users.

    The max_reuse_connection and the timeout settings are already available during the connector configuration (Web server configuration tool > "Add" > Advanced Settings), but usually users skip that part. The connection_pool_size is an explicit parameter that needs to be added. The value of which can be identified by metric.log or monitoring tools.

    The reason, we don't set the “default value” in worker.properties is, the environment varies from user to user. And that is the reason, I have mentioned clearly in the blog that, "The above use cases are scenario based and may vary from site to site, depending upon the load, architecture and traffic on the site." Thus, we leave it for the users to tune the values, based on their application and adjust the parameters.

    Yes, I agree that the web server configuration tool can be enhanced and the connector tuning could be simpler and optimized. Please log an enhancement request and we will look into it.

    Once again, we appreciate everyone’s thought on this one and thank everyone to make the product better and better.
  • 27 Charlie Arehart // Jun 30, 2014 at 8:37 AM
    @Anit, thanks for that, and I have now created two feature requests:

    https://bugbase.adobe.com/index.cfm?event=selectBug&CFGRIDKEY=3783361

    and

    https://bugbase.adobe.com/index.cfm?event=selectBug&CFGRIDKEY=3783364

    Interested folks should please go vote for and/or comment on them.
  • 28 Charlie Arehart // Jun 30, 2014 at 8:41 AM
    @Anit, I'm not sure I agree with your point against changing the defaults that CF creates.

    You say "the environment varies from user to user", and of course that's true, But again, in all 3 of your use cases you show adding 2 values which are never enabled by default, and you change a 3rd.

    Given that, and that you have to pick SOME value for the default, I'm simply asking why you don't put these other values in as the default initially? I know it doesn't know, but why not change the connector to do that? (Perhaps you may prefer to respond to that in the feature request I just posted.)
  • 29 Charlie Arehart // Jun 30, 2014 at 8:42 AM
    In my last comment, I meant to say "I know it doesn't *now*", not "know". :-)
  • 30 Dave Cordes // Jun 30, 2014 at 8:56 AM
    @Charlie - Thanks for creating those feature requests, I have voted for both. Having insight into the inner workings of ColdFusion and customizing the configuration tool would be two great additions to an already great product.
  • 31 Helge Hetland // Jul 1, 2014 at 2:01 AM
    Hello,
    I´ve been monitoring the discussion here and I have also tried to do some changes on our servers, and things is getting more stable.

    One thing I had troubles with was the cfstat utility - it gave "strange" results.

    It turns out that the CFSTAT settings in neo-metric.xml was the same on all servers (for the setting cfstatport). This is a bug (https://bugbase.adobe.com/index.cfm?event=bug&id=3361929) I don´t know if its still a bug in CF11.

    The solution is to change the port address and start cfstat with the port parameter (e.g. cfstat 10 -x -port 7997).
    (The connector port in cfadmin also needs to be set to the AJP port in server.xml - that setting must also be updated manually when there is more than one instance of CF running on the same server)

    The strange thing about this is that after changing the stat port it seems that Coldfusion (10 Enterprise) is also serving requests much faster after restart. Earlier the reqs would queue up for 2-3 minutes before the server was "stable", but now I don´t see any delay.

    It might be a coincidence (why would these things be connected..)
    But it might be worth looking into if you have a similar setup.
  • 32 Chris Gaelic // Oct 14, 2014 at 3:11 AM
    What an irony - I attempted to go to the bugbase.adobe.com site to read Charlie's feature requests, and got a 'This page can't be displayed' error!! Could it be that it's suffering from the same problems that we're all having?
  • 33 charlie arehart // Oct 14, 2014 at 8:50 AM
    Hey @Chris, I'll note that sometimes I get errors going to that site (and some others at Adobe), where I find that if I just go to another browser it works. Something to try.

    BTW, I don't think it's that there's a problem with Adobe supporting a given browser. Rather I think it's a cookie problem, and by going to another browser you may not have there the same cookies as the one that's failing for you.

    Or you can use browser features like "new incognito window" in Chrome, or "new private window" in Firefox, which should open the page without passing in any previously created cookies. I've found that to work also when my normal browsing of the page did not. (Just experienced it in FF, and it worked in a "new private window".)

    And finally one could also clear their cookies for Adobe and maybe find it would work normally.

    Of course, it could have been just that there was a server-side burp. I see your comment was at 3:11am, which if that's eastern time may have been a maintenance window for them. To be clear, I was able to visit the page now, at about 11:30am ET.

    Hope that's helpful. And thanks for considering trying to look at those feature requests. :-)
  • 34 Chris Gaelic // Oct 14, 2014 at 3:02 PM
    Thanks Charlie - I got in using Firefox, and added my votes to both requests. Adobe need to address this urgently.

    CF10 and 11 have some really cool features - which doesn't mean a thing if ColdFusion doesn't reliably serve pages.
  • 35 Anit Kumar Panda // Oct 14, 2014 at 9:16 PM
    @Charlie & Chris, we appreciate you thoughts and efforts. we will work upon these.
  • 36 Jeff // Nov 17, 2014 at 12:25 PM
    After installing CF11 and having the server crash once or twice a day with the Warning message, I see we need to do some tuning. However, having read through dozens of CF/Tomcat/IIS blog and forum posts I'm still left confused.

    We're running Windows 2012 with CF11 and have about 50 sites running in IIS. We have individual connectors for each site which is where I'm running into trouble, I think. Should I just go back and choose 'All'?

    I initially thought doing each individually would give me better performance customization per site thinking we could give more resources to our more-trafficed sites, but this seems like a huge headache to have to go through each workers.properties file. Also, I'm not sure what I should be using for values with so many sites defined. Should I, going by Use Case #3, eg:

    maxThreads=750 in server.xml
    leaving only:
    max_reuse_connections=1 (connection_pool_size / number of sites)
    connection_pool_size=15 (15+15+15+....+15 at 50 sites = 750)
    connetion_pool_timeout=60

    I realize that Definitely isn't the correct answer, but I'm not finding any good responses when hosting a large number of sites using multiple connectors. Is this just a bad idea?

    And if so, can I just run wsconfig.exe and select All to convert them to one big connection? Then if so, as Tom asked here (http://blogs.coldfusion.com/post.cfm/tuning-coldfusion-10-iis-connector-configuration), what would reasonable values be? I realize there won't be a specific number here, but ballpark and an explanation would be very helpful.

    Thanks for any help and direction!
  • 37 Chris Gaelic // Nov 17, 2014 at 1:20 PM
    Hi Jeff,

    I've just been through this, with around 100 websites, and upgrading from cf8 to cf11, using cf10 as an intermediate stage.

    While I had multiple issues to work through, the intermittent cf hangs appear to have been because the migration program didn't update all the jvm references from cf10 to cf11.

    Both the java.home and application.home were still pointing to ColdFusion10.

    I then checked through all my settings, and found that Java Ext Dirs and /gateway were also pointing to ColdFusion10 addresses.

    After correcting these, my webserver has been running for the last 3 weeks without a problem.

    There are some other changes you can try, but this would be the first thing to check.
  • 38 charlie arehart // Nov 17, 2014 at 1:42 PM
    @Jeff, as to the question about these settings (and in addition to my lengthy comments on many related issues above), I'll add that I have helped many people where the ONLY change that was necessary was the timeout (60 secs and 60000 ms in each file, as discussed in the blog entry above).

    And note that this is about IDLE connection timeout. It has NOTHING to do with how long you think a page may run (many people misconstrue this and are afraid to set the value "this low".

    I'll add also that folks can use the IIS "worker process monitor" feature (at the server level in the IIS interface) to see when requests are piling up in the app pools (a "worker process" is an instance of a given app pool).

    While requests can be shown piling up there because they ARE running in CF for a long time (which would be its own separate problem to solve), if you see them piling up there and there are NO requests running in CF (as shown by CFSTAT, the metrics log, the CF Server Monitor, FusionReactor, SeeFusion, or a java thread monitoring tool), then I'd argue those are idle connections that ought to be timed out. As the blog entry notes, the default is infinite (they say "indefinite" but the Tomcat docs do clarify it should be "infinite".)

    Finally, I'll note that it has never yet been necessary (in any assistance I've offered) to change the pool_size or max_reuse. I think that is a total mystery and until we get diagnostics to know what those should be, I doubt anyone can tell you the "right numbers". (Please vote up the bug entry I mention above, 3783364, to get Adobe to help us to better monitor and troubleshoot that setting.)
  • 39 Anit Kumar Panda // Nov 18, 2014 at 3:56 PM
    @Jeff, it would be a lot of task to create individual site level connectors. If you wish to, you are more than welcome. However, i will suggest creating a connector with "ALL". "Use Case# 2" should be your scenario. How did you identified that the issue is caused due to connector tuning and not server tuning, probably memory tuning? Please look at http://blogs.coldfusion.com/post.cfm/coldfusion-11-iis-connector-tuning#comment-7849D9C1-F887-67D0-F7EFA395EAAF603C for Metric logging.
  • 40 Jeff // Nov 20, 2014 at 4:23 PM
    Thank you all for your feedback and suggestions.

    @Chris, we did a fresh install of CF11 on a new instance of Windows 2012, so migration shouldn't be a factor but it's a good thing to look at when I have a chance.

    @Charlie, Anit - interesting! I was hoping the installation would be pretty forgiving and not require tuning, but I realize that going with the individual connectors per site is a bad idea. I'm a little hesitant to switch to All since we had a bad accidental experience with that during ramp-up, but once we get proper backups and schedule some downtime I think we'll move to a single connector and see if that fixes matters.

    I'll post back once we have a chance to try these out!
  • 41 Helge Hetland // Dec 2, 2014 at 6:01 AM
    I have been working with instability issues on our servers for some time now.

    My main issue turned out to be the number of templates in the system.

    With the setting "save class files" turned on we have had a lot of problems where the server simply "locks up" and starts queuing requests/locking up (and more often than not never recovers so a restart is needed).

    My initial thougts and efforts went into connector tuning but I newer got the servers to behave as "expected".

    This link in stack overflow got me in the right direction finally: http://stackoverflow.com/questions/10941977/how-to-resolve-deadlock-issue-in-coldfusion-9-coldfusion-util-abstractcacheloc

    This indicates that the locking problem has something to do with the saved class files getting corrupted in some way.

    When I deselected the "save class files" on my servers the behaviour became more normal and understandable.

    This might be something to look out for for those who have many sites / applications and many .cfm templates on the server.
    (our number is close to 100 000 templates total)

    Regards,
    Helge

Leave a Comment

Leave this field empty:

Blue Mango Theme Design By Mark Aplet

Super Powered by Mango Blog