May 07 2006
ColdFusion Upgrade: 4.5 to MX 7.01
The last few months of my job has me focused on upgrading our ColdFusion web application servers from version 4.5 to 7.01. Skipping two full versions is a mighty leap, especially for the size of the application that is running on these systems.
We attempted an upgrade from version 4.5 to 6.1 about two years ago and it ended in spectacular failure — there were flames, loud crashes, screams, terror. That upgrade consisted of shutting down the application, upgrading ColdFusion, and turning restarting the application. That’s it. No testing, no planning, no analysis, and NO ROLLBACK PLAN.
What we experienced was scary. The JRun server would crash and restart itself after running anywhere from a few minutes to an hour or so. We were never able to identify the cause of the problem and we ended up having to shut down the entire system on a weekday afternoon to rebuild each ColdFusion server with the 4.5 version. What a nightmare. Our clients, needless to say, were not happy.
The circumstances surrounding the current upgrade attempt are much different. The business is under better managment, different lead developers (myself being one of them), and we are leveraging an Adobe Systems support plan which has benefited us with the support services of Steven Erat. Steven’s support was invaluable in overcoming our struggles with this upgrade, perhaps because it is his job to support us, or perhaps because he genuinely enjoys it. I would like think it’s because of the latter. Nevertheless, we couldn’t have done it w ithout his research and recommendations. I’m going to need to thank him once we deem the upgrade a success.
On with it, already… The big news is that we finally upgraded two servers last week with ColdFusion MX 7.01. The two servers are running in the cluster alongside eight ColdFusion 4.5 servers. Our plan is to upgrade two servers at a time until the entire system is running on 7.01.
I won’t get into the chronology of the events that got us to where we are now. Just know that it took a lot of work, a lot of people, and a ton of failure to finally get our application running in 7.01. Here’s a summary of what we found and how we fixed it.
1) URL.requestTimeout
We made several minor coding changes to our appliciation, but only one of note. In ColdFusion 4.5, the default timeout of page requests is overridden with a URL parameter (requestTimeout). Starting with version 6.1, the ability to control the timeout of page requests with the requestTimeout variable was removed (a good thing, from a security standpoint). Timeout is now controlled using the <cfsetting> tag. This was a problem for us since there are hundreds, if not thousands, of areas where we override the default timeout using the requestTimeout URL variable.
The solution to this problem was simple. We simply created a file that contains a condition that checks for the existence of url.requestTimeout and sets the requestTimeout attribute of the <cfsetting> tag if url.requestTimeout is found (the value of the attribute is set to the value of url.requestTimeout). That file is included in application.cfm if, and only if, the version of ColdFusion processing the request is 6.1 or higher. ColdFusion 4.5 cannot process the <cfsetting> tag, so the the ColdFusion version must be checked before including the file containing the <cfsetting> tag. This solutions allows the default page timeout to overridden using the url.requestTimeout variable in all versions of ColdFusion (4.5 or higher, anyway).
2) .jar File Locations
Our application is extended using Java. There are three .jar files that, for reasons that I won’t bother to explain, were placed in a subdirectory of the ColdFusion installation that we created and titled classes. We updated the classpath in ColdFusion adminstrator to point to each of the .jar files in the classes directory. This wasn’t a problem in ColdFusion 6.1, but in 7.01 we were having problems with one of the Java applications. After many hours of research, many hours on the phone with the vendor of the Java application, several failed attempts at workarounds (a wrapper class to the .jar, rewriting the ColdFusion interface to the .jar) and some conversations with Mike Dinnowitz of House of Fusion (HoF is a fair source of information for all things related to ColdFusion), another developer in our group found that all we needed to do was move all the .jar files to [CFusionMX7]/runtime/lib/ and remove the classpath entries to the .jar files. That resolved the problem.
Steven Erat confirmed this in an email and provided some additional information:
Regarding jar files, the standard location for additional jar files for a J2EE web application is WEB-INF/lib. While runtime/lib will work, that location has more visibility than is necessary.
Package visibility:
- runtime/lib: all server instances and all web applications
- runtime/servers/lib: same as above
- runtime/servers/<server>/lib: all web applications for just that instance
- runtime/servers/<server>/<webapp>/WEB-INF/lib: just that one web app on that one server instance
3) Sun JVM HotSpot Runtime Optimizer
I mentioned earlier that our original attempt at the upgrade from 4.5 to 6.1 two years ago ended in failure and rollback due to JRun repeatedly crashing and restarting. We experienced the same problem with the upgrade from 4.5 to 7.01 (although, this time we experienced the problem in our development envirnnment, not production).
90% of our time with Steven Erat was spent trying to identify why JRun was crashing and restarting. The tough part was that we were unable to reproduce the crash and didn’t have a clue how or when it would occur. Over the course of several weeks Steven had us logging every conceivable piece of the ColdFusion server. There were several paths that Steven took to try to identify the cause of the problem, all of them lead nowhere.
Finally, another developer and I noticed a pattern to the crashes and were able to reproduce the crash at will. On a hunch, Steven asked us to disable the HotSpot runtime optimizer by adding the -Xint switch to the JVM arguments in ColdFusion Administrator. We did as instructed, and our problems with JRun disappeared.
I am a little in the dark as to why turning off the HotSpot runtime optimizer works. It seems that Steven Erat was as well, although he has an excellent post that begins to shed some light on the topic.
Exception with <cfmail> Tag and Attachments
The issue we were having was an exception when trying to include an attachment in emails sent using the <cfmail> tag. We would see an error similar to this:
tried to access class javax.mail.internet.ContentDisposition from class coldfusion.mail.RFC2231Util
Well, remember the problem with the locations of the .jar files? Moving the .jar files to the [CFusionMX7]\runtime\lib problem fixed a problem with one of them. I then mentioned that Steven Erat provided to us the recommended (and standard) locations for .jar files. [CFusionMX7]\runtime\lib is not one of those locations. So, while our problem with the one of the Java applications was fixed by moving the .jar file to this directory, the problem with the <cfmail> tag was introduced by moving a different .jar file to the directory.
It was a simple case of conflicting classes with the same name; one of the .jar files that we moved to the [CFusionMX]\runtime\lib directory contained a class with the same name as a class that ColdFusion was trying to access.
The solution: put .jar files in the appropriate directories.
Are you sure about the RequestTimeout stuff? I know the ‘Migrating ColdFusion 5 Applications’ document says “ColdFusion MX no longer supports the RequestTimeout attribute in the URL.”, but my reading from elsewhere is that it’s just deprecated (i.e. still works).
Running my own tests backs this up. e.g. take this code:
<cfset x = 0>
<cfset start=GetTickCount()>
<cfloop index=”i” from=”1″ to=”100000000″>
<cfset x = x + 1>
</cfloop>
<cfset end=GetTickCount()>
<cfset totaltime = end - start>
<cfoutput>
#totaltime# milliseconds<br>
#totaltime/1000# seconds
</cfoutput>
Running it without any timeout, it times out at 30 seconds (the CF Admin default). Sticking requesttimeout=500 on the url, the page completes in about 300 seconds. Setting the requesttimeout=100 on the url, the page times at 100 seconds.
I am sure about the url.requestTimeout stuff. Take a look at the second paragraph. It’s the <cfsetting> tag that sets the requestTimeout. It would like something this:
<cfif isDefined(”url.requestTimeout”) and isNumeric(url.requestTimeout)>
<cfsetting requesttimeout=”#url.requestTimeout#” />
<cfif>