Hello, this is Darren Mallette from Quest Software. I am a Database and Virtualization Solutions Architect. This blog will present the steps to follow to diagnose and resolve an issue that I have been seeing both in my lab and with a couple customers who are running their Foglight servers on Windows (2003).
Foglight uses several ports on the host it resides on - these are defined upon installation and can be reviewed from the Foglight Administration dashboard, in addition to the foglight.config file in $Foglight_Home\config. Depending on the order of startup of Windows services, these ports can be grabbed by other processes. When you connect to your Foglight server, you'll end up with a page not found message:
So now we panic right? Well, the first step is to check that the Foglight Management Server service is running:
That checks out ok, so we'll check with Task Manager to make sure we see the fms.exe process:
It is running, but I notice something interesting. The memory usage is much lower than what I typically see. So I am suspecting an incomplete startup. Time to check the management server logs in $Foglight_home\logs:
I'll search for a couple things in the log. First I'll search for WARN, and sure enough, I find a message. Directly below that, I see that one of the Foglight ports is in use (port 1111). Let's check to see what is using that port. I'll fire up a command window and type "netstat -ano". This will show me the process id in addition to ports in use.
Sure enough, I see port 1111 in use by 2 processes: 5412 and 4548. Well, what are those? Back to Task Manager to find out. If you don't see the PID column, you can go to the View menu, Select Columns, and check the box next to PID (Process Identifier). Then we will sort by that column.
We see the 2 processes above. These are for Performance Analysis for SQL Server. Lucky for me I know the service that controls these processes! For a quick fix, I'll stop the "QAM Launcher 3566" service, restart the Foglight Management Server service, then restart the QAM service. The QAM service will select a couple ports at random and when it first started, 1111 overlapped with Foglight unfortunately. With customers I have seen backup control software, telephony software and custom applications with this behaviour. So now we can try to prevent this from happening in the future. This is the solution that I used in my lab. I wrote a small batch file that starts the services in the order that I want, and I use "ping" to introduce a delay so that the processes have time to get started. I put the batch file as an entry in the registry key "HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\RunOnce" and enabled auto-logon to the Windows server. For details on that, please see this Microsoft KB article and decide if it's right for you or not:
http://support.microsoft.com/kb/324737
This is the batch file that I wrote so you have it as an example:
I test it with a server reboot and the services and their processes all start ok, and I can connect to Foglight.
Thaks for reading, and I look forward to any feedback in the comments. I am especially interested in how you would control the startup order of services (and yes, I did look at dependencies first).