HA – Clustering ColdFusion Part 1 – Installing CF
This will be the first post in a series relating to clustering ColdFusion. In this first series of posts we will be looking at clustering CF at a software level using ColdFusion 8 Enterprise. Hopefully later on, we can move to a Hardware-Software set-up with examples. I mentioned in a previous post that what I will detail is drawn from my experiences from either creating clusters for clients or working on existing clusters. There are no doubt other ways to do this. Firstly, I always create what I call a “master instance”, typically the first instance which is created from a multiple-instance install. Here are some important steps from that… As the install progresses select “Multiserver configuration”. 
At the point where you are asked to select a webserver select the “Built-in webserver”. We will use this to run CF Admin and eventually use with wsconfig utility to connect to our Production web server… 
We let the installation complete, successfully and at this point we have one ColdFusion instance (cfusion) and a JRun instance (admin). We do not need the JRun admin instance so we can go into Windows services and set it to manual start. Next we take a look at the ColdFusion Administrator GUI on the single instance created during the install. The thing to note about this instance is the bottom section on the left navigation pane “ENTERPRISE MANAGER” with that section there are two sub sections “Instance Manager” and “Cluster Manager”. This section and it’s sub sections will not be present in the instances we create. This is why I consider this first instance created during the install of CF8 to be a master instance. Its job from now on will be to manage the cluster. We will next create our first instance, go into ENTERPRISE MANAGER > Instance Manager to create the first instance, obviously give this instance whatever name you wish, I tend to make it meaningful to the web site it will support and then number {instance_name_1}, {instance_name_1} etc… 
Repeat these steps to create your second instance, once again you use the master cfusion instance to do that. Once this is complete you will have two instances in Instance Manager… 
At this point we have installed ColdFusion and created two new instances, the next step is to create a cluster for these to sit in. Before we do that one thing that is good to do at this point is to connect each instance, individually to your Production web server just to make sure that they are fully functional individually… 
Once we have verified that all instances function as expected/needed we are ready to move on to clustering them, which will be the subject of the second article in this series.
@Xung: “If you still have problem you might want to try setting up S2 to be exactly like S1″
I tried that setup but I don’t understand how sessions will replicate across clusters. If S1 fails, then a user will get sent to the other cluster in S2 which is not “aware” of S1 as far as I know.
I have J2EE session variables enabled, and I know sessions are replicating inside the cluster since when I stop S1_i1, S2_i2 will pick up and I can still see the session. But when I send all requests to S2, S2_i2 picks up as well (since I am pointing dev.test.com in S2 to ClusterS1 and S2_i2 is part of that cluster) but this time sessions have not replicated.
Am I missing something to either get the cluster to cluster communication happening (if it’s possible) or to enable S2_i2 from sharing the session in the case of server (not instance) failover?
@Fernando you should not be able to get Cluster to Cluster replication working in my opinion. A users session should only be replicated within the instances in the Cluster they exist in. I hope that makes sense. Here is another important thought point, apart from the two physical servers being different S1 – S2 the JVM is the other key difference, in other words you have JVM1 on S1 and JVM2 on S2. There is no way to have the same JVM spanning two different physical servers. This is just a thought point not a solution.
I also had a question I assume you registered the remote instance in CF Admin on S1 to the instance on S2. If you already stated this I apologize for asking.
@Mike, I figured there that sessions would not replicate across clusters since they are not “buddies”. Only instances are.
I did register the S2 instance on S1 CF Admin. The session replication does work across both instances (local and remote) as long as users are sent to S1 (as expected). So this aspect of the clustering works fine. But when I send all requests to S2 is when I do not see sessions replicate.
Whenever I send the requests to S2 (in which I have the test site assigned to the ClusterS1), I would have thought that sessions would be available there as well since the user is still going to the original cluster. This is not the case.
Should the sessions be replicating in this latter scenario?
@Fernando, thanks for your response, yes in theory they most certainly should be replicating. My suspicion is that something in the architecture of ColdFusion at the Application server level, which is where it resides, is not functioning correctly; it may be as simple as not writing to an underlying file correctly or more difficult to diagnose as with an Application scope quirkiness. I am still trying to dig through this myself and will keep you posted via this blog.
hi all i am in trouble and please need help. i need to config cof8 64 bit in a distributed environment.
cf80 in on box and web server in another.
i have doubts regarading where we nneed to actually put the files ??
i tried the manuall connector stuff. but its not working for me
please help…
@dev is your aim to do any clustering in your deployment? If so, will it be hardware and software or just software? This thread is aimed at HA-clustering rather than distributed mode issues. If you will be clustering can you let us know the following…
Operating system – web server – ColdFusion version (standard or enterprise). Also, what files are you meaning – web site content – ColdFusion-JRun?
@Fernando I just came across an interesting blog post by Sean Corfield which talks about Application scopes as they pertain to Java-J2EE. I don’t know if you solved your issue yet but this looks like a possible clue to what is going on.
@Fernando sorry here is the blog post
http://corfield.org/blog/index.cfm/do/blog.entry/entry/Sharing_Application_Scope
@Mike Thank you for the link to the post, it is very insightful and I might be able to apply it to other session issues I have been having.
As far as my original issue I was able to resolve it. It was actually quite simple:
Since I do not have a HLB in place yet, I was manually shutting down one server to simulate a server (physical box) fail over and then changing the host file to point to the other server. For this to happen correctly in Windows machines one has to usually close down the browser, flush the DNS cache, then try again at which point the site should now go to the new server. This worked fine, the problem was that by closing the browser I was killing the J2EE sessions, as far as the browser was concerned, since the browser is what holds the J2EE session information in a cookie. Even though the session was still alive as far as JRUN was concerned, the browser was no longer “aware” of them. I had forgotten about this from my JSP days.
Once I figured this out, I realized that my configuration was actually working as some further testing revealed.
If I am incorrect in any of this, please let me know so that I may continue my research.
I want to thank all of you for the help you have given and I will try to continue posting my experiences so everyone may benefit from them.
I am having a few issues we are currently trying to use hardware load balancing with Apache and CF8 multiserver installs on Jrun i.e LB in front of the webserver and an LB infront of the CF servers. However we can’t seem to get the web servers to communicate with the CF servers when they are in layer 4 clusters via the proxy ports.
Is there a particular setting in Jrun we need to enable or does jrun use a different protocol for the proxy service.
@Alonso sorry for a slightly delayed response here from me. So am I right in assuming that Apache is on different physical servers to ColdFusion?
If you have 2 instances in a cf cluster and have a factory stored in Application scope, how could I run a script via a url to ensure BOTH instances have Application.factory reset?
Maybe I’m getting confused by the whole clustering thing but we’re going to do this at work for getting sticky sessions and I wondered how Application scope works with this setup.
Thanks
@Alan when we get to the Application scope level things can get difficult to deal with. This scope is not and cannot be replicated because on each instance the items in the Application scope have no knowledge of each other. My suggestion would be run scripts for each instance.
Thanks Mike.
When you say ‘run scripts for each instance’ can you give an idea what you mean and what I would use in the script to target each instance and where I would run them from?
@Mike Brunt Hi sorry about the late reply, I managed to fix the issue. The load balancer seemed to have stopped working so a reboot sorted the connection issues out(coyotepoint firmware version 8.0.0b seems to have wierd issues every so often).
The way I have set it up is using 4 solaris x86 X6250 blades on the presentation layer each running 2 centos unbranded zones with apache and the jrun connector so these boxes only contain static content such as images etc. The ip address in the apache config files corresponds to the ip address of the ColdFusion cluster which contains the instances of ColdFusion and their related jrun proxy ip addresses.
So the httpd conf will have an ip & port of say 10.0.5.3:52800 which is the HLB ColdFusion cluster ip address and then this has 8 instances of coldfusion running across 4 sparc T6320 blades. So the setup runs really well it does take a bit of JVM tuning to get the sparcs to really shine but it is worth a few days playing with things.
Just want to say this thread was great help !
Alonso, we have a similar setup. Do you have session replication across the blades?
@Alonso, thanks for your comments and also the good information you shared here which I am sure has benefited many.
@Mark are you using Session replication and if so do you find it reliable?
@Alan, I have another question before responding. Are all of your application scope variables set in one global file or are they set in many places?
@Mike, yes all in Application.cfm. It’s a typical ‘if we get a reload = true in a url then re-create the factory and set it to Application.factory’ sort of thing.
I also beleive my colleague Shalja has asked you the same question at ‘wee dram’ the other day. We’re all on your case
Hi Mike
Good talk yesterday at ‘wee dram’.
I have a question I need clarification on concerning doing things like resetting factories on a Coldfusion cluster.
Lets say I have the following setup
Two instances of Coldfusion clustered on the same machine. Instance A (port 8091) and instance B (port 8092).
If I want to reset a factory on instance A would the url be IP_ADDRESS:8091/resetSomething.cfm and then of course for instance B IP_ADDRESS:8092/resetSomething.cfm
Thanks in advance
@Mark are you using Session replication and if so do you find it reliable?
We havent tested it under load, but session replication with a cluster that has all instances in the same container is working.
(Of course, we want it between containers on different servers)
Guys and Gals – calling from across the pond in the UK.
Thanks for all the information in the blog but I am struggling with a more basic clustering problem as we don’t use session variables, relying totally on domain cookies instead and using a single central SQL database. The background is I have a single application running on a single box using CF8 Enterprise (multiserver configuration with the defaul EAR file) and IIS with the SQL Server 2000 database on a separate box. As customer load has grown response times have worsened so our strategy was to install an exact duplicate box again using CF8 Enterprise (multiserver configuration with the defaul EAR file) and IIS with the database remaining for now on a separate box and being accessed by both CF servers. The new CF server is fully working and has been running the live workload without any issues.
Now we are trying to implement clustering to share the customer accesses across the two servers. We have therefore set up an instance called Host1 on the first server and another instance called Host2 on the second server in line with the initial steps of Mike Brunt’s blog on Clustering ColdFusion Part 1 – Installing CF. I also made sure that the Host1 and Host2 instances on their respective servers were started.
So far so good. Our next step was then on server 1 to use Register Remote Instance under Instance Manager to register Host2 using the IP address of server2 and the remote port number from the server2 Host2 instance – however having pressed Submit all I continue to get is the remote instance recorded but with ‘Network error’ shown in the Cluster column. We have tried leaving ticked and unticked the Enable JWS (Internal Web Server) but so far have just left ticked the Directory Browsing on the editing screen of the server instance. Note we also haven’t moved to the next steps of either creating the cluster or running the Web Server Configuration tool as I assume I have to get the remote instance properly accessible.
Any suggestions about what I am doing wrong or what ‘Network error’ can mean would be most appreciated. Thanks.
@Ken that’s funny I was just over in England and am now back in the USA, I got back late yesterday. Thanks for posting your issue here and what I would like to see is the host1-out.log and host2-out.logs to see if there are any clues there. You can send to mbrunt at go2ria.net
A basic question also, could any network device be blocking the ports you need to use?
Lastly, I am trying to picture what sort of cluster you want to create? In CF-J2EE clustering an instance can only exist in one cluster. So if you have one instance on each server (I assume that is what you have?) you can only create a cluster on one of the two servers, which may not be what you want. I am making another assumption here, that there is only one production website in IIS?
@Tara, thank you for your kind comments regarding a WeeDram.
Am I right in assuming that you are trying to hit those ports, 8091 and 8092 remotely (not from the actual physical server they reside on) am I correct?
I am also assuming you would configure those ports from inside CFAdmin > Enterprise Manager > Instance Manager am I also correct there?
Thanks for your help with this.
I have emailed the two log files but they are not called host1-out.log and host2-out.log as those files do not exist but they are called easehost1-event.log and easehost2-event.log. I trimmed the names for the blog missing out the ease bit and I assume these -event logs are those that you need. Note I have tried to connect to the other server from each of them so there are multiple entries.
I have no network issues between the two servers using Remote Desk Connection and copying files between the servers, accessing the database server using ODBC and bouncing web page access between the two servers.
In terms of your other query, yes I only have one instance on each of the two servers and the aim is to create a single cluster on one of the servers. At present the main production server has only one production website in IIS receiving all web page accesses and in order to validate the additional server it is currently bouncing all user login requests which come to an index.html using http://88.208.231.2/usa/4-5/login.cfm to get to the second server.
Let me know if you need anything else and thanks again.
Hi Mike
Yes I am trying to hit the ports remotely. And yes you are right in assuming I would configure the ports inside CFAdmin.
Has anything been found on this issue?
I am having the same problem. I try and register a remote instance but i have the network error in the Cluster column.
@Jim,
Can you test basic connectivity between the two hosts?
For example, if you have the following JRun server defined on HOST2:
Name: cfusion2
JNDI: 2941
HTTP: 8401
Proxy: 51401
From the command line on HOST1, try this:
telnet HOST2 2941
You will either get a ‘connection refused’ message or a blank prompt. If you get a blank prompt, that’s good. Type ‘CTRL ]’ to quit the telnet connection. Then type ‘quit’ to exit the telnet application.
If you can telnet to all of the ports, then you do not have a ‘connectivity’ issue. If you cannot telnet to these ports, check with your firewall administrator.
Also, I’m not sure this will help at all, but I tend to register the remote ‘admin’ server first, before I register other jrun instances. See if you can add the ‘admin’ server instance on HOST2 as well.
i can telnet to them but still have the “network error”.
Also have it with the admin instance.
@Jim,
Send me your contact info at: damon dot gentry at gmail dot com.
Perhaps we can solve this one over the phone.
@Mark Ireland
Hi Mark sorry about the late reply, the way we have the setup in our test environment is having the 8 instances in to pairs of two per cluster. So each physical server has two identical instances which are clustered together. As far I as I can tell it is impossible ( I really hope it isn’t but it looks like it is) to get the sessions to replicate across different physical servers and the only way we managed to do it was having a cluster of two or more on the same physical machine.
@Mark Ireland
I am going to play around with it this weekend and see if I have an Einstein moment and I manage to get session replication across the blades working, I’ll keep you posted on my progress.
@Mike
We are trying to setup the Cluster Load balancing in CF8 server with 4 instances on 2 different physical boxes. Below is our server configuration details:
Fedora 9 64bit, CF 8 64 bit Ent-trail edition.
Setup Details:
2 Linux boxes, 2 CF instances on each box. CF is setup as given by you, in Multiserver environment. We created 2 instances of CF in both boxes called, CF1, CF2 in BOX1 and CF3, CF4 in Box 2.
Issue:
The issue we are having here is, when we add a remote server Box2 instance to Box1, it shows “Network Error”. Where as when we check the same using JRun Admin console, it says “The server “cf3″ is unavailable for administration on its specified host “10.10.0.14″. Please make sure that the server exists, and that at least one server is running and registered with the JMC on the given machine.”
We have done following things after going through some posts,
- jrun.subnet.restriction=*
- jrun.trusted.hosts=*
- We telnet’ed to the running instances port nubmers on other box2 from box1 and vice versa and we are successfully able to connect to each of them from either boxes.
- We have disabled firewall on both the servers.
Can you please guide us on the above. Let us know if
- we need to do anything else as this is a 64bit server?
- Is there any issue with 64bit edition of CF8 on Linux?
- Do we need to first join the CF to web server then configure the instances?
Thank you in advance.
Atul, it may seem obvious but just to make sure, have you confirmed that the hostname you’re entering when registering the remote instance is in fact identical (case included) to the one defined for the instance on the remote instance manager. (This is needed for Windows users, too, for other readers.)
Also, have you considered the info in the following entry:
http://cfprimer.blogspot.com/2005/06/jrun-clustering-with-windows-2003-nlb.html
Again, I realize it’s about Windows NLB and you’re on Linux, but perhaps you may see some connection.
Is anyone having issues with MX7 and clusters, when the server is rebooted the cluster doesn’t respond?
We end up having to remove the cluster from the web configuration tool and re add it again before it will respond to requests again.
Please help this is causing significant downtime for a “high availability” feature.
Hi Mike,
Is there a reason you recommend using the built-in web server for the main instance?
Why not use IIS for the main instance also? Specially since Adobr recommends not using the built-in webserver for production machines.
Thanks
@DK, I’m sure Mike will chime in, but I’ll share a thought: the recommendation against using the built-in web server (BIWS) for production has more to do with the question of load, or of other features that an external web server (like IIS or Apache) may offer that the BIWS doesn’t have.
Still, it’s not unusual for folks to enable the BIWS only, if only for accessing the administrator. One benefit there is that the port for the BIWS can be restricted differently than would be possible if the Admin were accessible via the external web server.
Of course, some do go ahead and provide access to the Admin through their external web server. There are certainly additional security mechanisms (like Digest Authentication) that may suffice for protection.
With respect to multiple instances, and where one leaves the cfusion instance to be used only for accessing its Admin to use the Enterprise Manager (meaning they don’t don’t use it to serve up code, don’t define datasources within it, etc., but only use it to manage the other isntances), I could see them regarding that as something to access only through the BIWS. But there’s no technical reason to favor it or not, I don’t think.
Let me rephrase one of those paragraphs: “Still, it’s not unusual for folks to enable the BIWS, if only for accessing the administrator. One benefit is that the port for the BIWS can be firewall-restricted differently than would be possible if the Admin were accessible via the external web server, along with all other apps, via port 80.”
Thanks Charlie; it does makes sense. I will use the inbuilt webserver and give it a try.
@Alonso
Make any progress getting session replication across the blades working?
Nice Post, thanks.
I have read and learned a lot about IP, software and computer programming through your blog.
I’m subscribing
Thanks again and keep the good Information comming.
Craigslist IP Software
Nice Post, thanks.
I have read and learned a lot about IP, software and computer programming through your blog.
I’m subscribing
Thanks again and keep the good Information comming.
Craigslist IP Software
I am having 2 DNS server, 2 web server and 2 CF servers. 2 DNS servers acting as LB, 2 CF servers are in Multi-instance cluster setup.
I have following queries:
- Where do I run the WSCONFIG command? on external Web Server or CF server?
- If it should be run on external Web Server, is the below command correct?, -cluster CLU1 -coldfusion -cfwebroot /var/cfroot/ -v”
“./wsconfig -ws apache -dir /etc/httpd/conf -host
Is it correct?
Please advise.
I’ve been beating my head against a wall trying to register a remote instance and kept getting “Network error”. I’ve found several people in the same predicament without a solution. I then found step 18:
http://livedocs.adobe.com/coldfusion/8/htmldocs/help.html?content=clustering_6.html
My servers were not on the same subnet and therefore, per the article, I had to “add the IP addresses of the other JRun servers in the cluster to the jrun.trusted.hosts property”. This threw me at first; even though I updated the local server’s property I still got the same error. When I added my local server’s IP (the server hosting the instance on which I’m using Instance Manager) to the remote server, AND restarted the instance(s), I met with success. HTH
@Mark
Hi Mark sorry for the extreme delay getting back to you, I got caught up in other projects for the majority of the year but have moved back on this again. So far we are trying top get session replication working across 4 instances in 2 different Solaris containers on the same machine. Once this works we will clone the environment and try across 2 physical machines.
At the moment using a single physical server the sessions seem to be replicating fine all though we do experience some faults every so often where the replication seems to stop or take forever i.e. we can sign in and we seem to be signed in on all servers; however the sign out seems to bug out every so often. This in turn means we may be signed out of one instance yet the other 3 will still regard us as being sign in. I’m pretty sure this is more to do with our application then anything specific to ColdFusion.
I hope you folks can help me.
I’ve got 4 sites set up on IIS 6, each with its own IP address/domain name. These are currently html sites. (Partech.com, parlms.com, pargovernment.com and springermiller.com). I have these same sites set up in a development area set up with the same configuration.
They were written by an outside firm and we’ve found that it takes to long to update them. Press releases take at least an hour to update all of the pages over all of the sites. So I’m converting them to CF. Unfortunately at this time I’m stuck using CF MX 6.1 Standard, which is connected to the IIS web sever using the distributed mode configuration. The main site, Partech.com, works fine. Its physical location is inetpub/wwwroot on both web server and the CF application server.
The problem is with the other sites. Their physical location is parallel with the main site. inetpub/wwwroot_parlms, inetpub/wwwroot_gov, and so forth. The start template in all sites is index.cfm.
When I try to browse to one of the other sites, for example parlms.com, I get back the index.cfm of the main site, partech.com. If I reference an htm file I get back the correct index.htm, therefore I conclude that IIS web server is working correctly. The problem is with the CF server or the connection.
I’ve added the site IP’s to the jrun.xml on the CF server.
I’ve checked my jrun.trusted.host entry and it currently has a * (which is ok for my test environment)
I’ve run wsconfig.jar –list command and it tells me there are no connected sites. (That one has me puzzled as I do indeed have valid connection that is working)
I found a tech bulletin [http://kb2.adobe.com/cps/121/121a0b29.html], which states that CFMX can connect to 1 web server, but it doesn’t talk about additional websites on the same server.
Can anyone give me a pointer to some documentation on how to do this? Better yet, tell me how to do this?
thanks, Jim
It now runs just like the day it was new.”