Jun 1

Written by: Foglight R&D Team
6/1/2009 9:33 AM

Foglight Dark Corners - Transition from Foglight 4

Hello foglight.org readers, Geoff Vona here again. A lot of users are moving from Foglight 4 to Foglight 5. This is great news because it will unlock a lot of fantastic new capabilities. Unfortunately, this transition requires a little bit of reorientation.

In this Dark Corners blog, I'll be talking about how Foglight 4 users can find their familiar workflows.

Motivation

The transition from Foglight 4 to Foglight 5 is best described by the Foglight 4 to 5 Transition Guide. This guide is part of the standard Foglight documentation set, and is available on SupportLink here). It covers a lot of the typical transitional issues. In particular, the chapter Finding Your Way Around has good information on how some of the standard Foglight 4 concepts map to Foglight 5.

This blog is intended to be complementary to the guide. In this blog, I'll present a specific slice through transition that will help Foglight 4 users understand what it will be like when they have migrated to Foglight 5. This slice is important because there are some product differences that can cause disorientation.

Why Is Foglight 5 So Different?

Foglight 4 is an agent-oriented monitoring database with a thick-client UI. Data is contained by the agents that collect the data in a simple model that is repeated for all monitored resources. In Foglight 4, Hosts run Clients, Clients run Agents, Agents contain Data and run Rules.

Foglight 5 was designed from the ground up for next-generation monitoring. Monitoring needs have shifted significantly from the agent/server model to much more fluid models. Remote monitoring, mobile monitored resources like virtual images, and unanchored monitored elements like transactions, requests and web services all demand much more flexible monitoring architecturs. In particular, the "agent anchoring" style of Foglight 4 does not lend itself to modern monitored resources like clusters, user requests and loose host->application relationships. This is why Foglight 5 is a model-oriented monitoring server with a thin-client UI.

Knowing that Foglight 5 was going to be vastly different from Foglight 4, we started work on migration at the very beginning of the project. One of the very first models that was created for Foglight 5 was the "Foglight 4" model. This model still exists today. The basic idea was very simple:

The fact that Foglight 4 agents can run largely unmodified against a Foglight 5 server provides that this modelling exercise was a success. However, it has its quirks.

Foglight 4 and 5 Comparison Table

Here's a quick comparison of Foglight 4 and 5 from a Foglight 4 user's perspective.

Foglight 4	Foglight 5	Comments
Thick client UI	Browser-based UI	Moving to a browser-based UI was a main requirement for Foglight 5. We understand that some customers have a personal preference for thick clients, but we believe that this is mostly about perceived performance differences. As Foglight UI performance improvements, and the interactivity improves, this difference will be less important.
Fixed data structure	Flexible data structure	F5 can model F4 data structures, so what is in Foglight 5 is a superset of what was in Foglight 4..
Stores raw data	Stores aggregated data	F5 can be configured to store raw data as well. F5 brings the benefit of managing data for the long term. Foglight 5 can report on data for longer periods of time.
Purges data	Data roll-up/aggregation through retention policies	F5 can manage its size for the long term. Policies can be configured to purge data as in F4, but doesn't work this way by default. Please note that the purge ASPs in F4 agents no longer have any effect in F5.
Limited UI customization	Full dashboarding capabilities for any data	In Foglight 5, you can get access to a number of pre-created visualizations for all the cartridges. In addition, you can get access to any piece of data that has been collected and create a custom dashboard or report.
Good reports	Improved reporting framework, but some reports are missing	This gap will close soon.
Rules run on tables	Rules can run on anything and access any data	The Foglight 5 rule capabilities are very flexible, and can run all of the Foglight 4 behaviours. However, the self-clearing behaviour for rules like LogFilter is missing. This can be worked around using a field solution.
Blackout on rule or agent	Blackout on agent or object	Foglight 5 blackouts are much more flexible and more aligned with modern monitoring requirements. Most users are asking to black out a unit that will undergo maintenance - a "host", or a "server". Blacking out on agents or individual rules is actually more time consuming than needed. Unfortunately, the UI for object blackout in F5 is not complete. This work is on the near term roadmap.
Foglight Operations Console	Agent Status + Agent Blackouts + Agent Hosts + Agents dashboard	All functionality available, roadmap includes plans to tie this all together in one location.

The main shortcomings of note are as follows:

• We need to finish the work of adding domain reports. This work is underway.
• We need to improve self-clearing behaviour on rules that run on top-N data. The workaround for this is easy - delete the old alarms after a certain amount of time. There is already a field solution available for this.
• We need to connect object blackout to a user interface. The command line utility can be used as a temporary workaround
• We need to consolidate agent operations into a single console.

We'll talk about the last two points a little later in this document. First, some background on cartridge conversion and how it affects the user experience in Foglight 5.

Cartridge Conversion Affects User Experience

The Foglight 5 cartridge set contains a mix of converted cartridges and exploitive cartridges. A converted cartridge is created using tooling that migrates the Foglight 4 settings (DCMs) into Foglight 5 equivalents. An exploitive cartridge is created from the ground up to fully exploit Foglight 5 capabilities.

Exploitive cartridges feel quite different from Foglight 4 cartridges when used in the product. The main reason for this is that you can add a lot of new dashboards when creating an exploitive cartridge. In fact, exploitive cartridges often create all-new domain-specific workflows. A good example of this would be the vmExplorer dashboard in the VMWare cartridge, or the JavaEE Requests dashboard in the Java cartridge. Both of these dashboards are excellent starting points for domain administrators. They are optimized for certain users, and work really well for those users.

This is an example of an optimized experience (VMWare):

Unfortunately this new optimized experience is not available for all converted cartridges. Instead, we rely on traditional Foglight 4 workflows to enable users to get value from their monitored data.

This is an example of the user experience for a converted cartridge. First you have to find the view that interests you by choosing the agent:

Then you must choose the view from a view selector. In this case the views are all graphs and tables.

This workflow is common for all Foglight 4 converted agents. This is equivalent to what was in Foglight 4. However, it is clearly not as good as the exploitive dashboard workflow - it doesn't lead the user to specific views as part of a diagnostic workflow. The table below summarizes which Foglight 5 cartridges are converted from Foglight 4, and which have been developed in an exploitive manner:

Converted	Exploitive
OS Cartridges (Windows, AIX, HP-UX, Linux, Solaris)	VMWare
Packaged applications (PeopleSoft, SAP, Siebel)	Custom applications (JavaEE and .NET) &brk;Packaged applications (Oracle e-Business)
Database (Oracle, SQL Server, Sybase, DB2)	Database (Performance Analysis)
Utility Cartridges (WebMonitor, NetMonitor, LogFilter, AppMonitor)	End user (FTR, FxM, FxV)

Some converted cartridges like Oracle have added global dashboards to act as new workflow roots. The OS Cartridge takes this the furthest by adding a common model for all hosts (physical and virtual) plus a common set of visualizations. The bottom line is that even the converted cartridges are getting better with the addition of new dashboards.

The way cartridge conversion works is shown below:

What this means is that the end result of a cartridge conversion is the same views as were available in F4. What's important is knowing how to access these views. In the next section we'll review the workflow for Foglight 4 and 5.

Foglight 4 Workflows

Foglight 4 relies on the Foglight Operations Console (FOC) as the common anchor for all workflows. It is highly optimized for a user that acts as both an administrator and operator.

The FOC shows data organized by agents. The agents are organized by hosts. The hosts, in turn, are organized by network subdomains. This is an intuitive presentation that works for anyone who is primarily interested in the performance of hosts or agents.

Selecting a host or agent will change the alarm set at the bottom.

A number of actions are available by right-clicking on a host or agent. The action set changes depending on whether you are on a host or agent. The actions are a mix of administrative activities (start/stop agents, edit agent properties) and operator activities (find a view).

The state of an agent or host is clearly shown in the FOC. If an agent is down, it turns yellow. If a host is down, it turns yellow. It is clear from the FOC whether a rapsd on a single host is collecting or not.

Improving Foglight 5 to Ease Transition from Foglight 4

Create a Foglight Operations Console Equivalent

Probably the biggest gap in the F4->F5 transition is the lack of a single consolidated view like the FOC. The previous section details how to get the equivalent operation. Ultimately, these operations should be combined onto one Foglight 5 dashboard. The right dashboard is the Agents dashboard. Here is a set of changes for the Agents dashboard that will help:

General Dashboard Changes	Changes Active on a Selected Host	Changes Active on a Selected Agent
Expose the client state so that the dashboard shows when SPID/Agent Manager is down Add the agent and SPID/Agent Manager version #s to the dashboard Display hosts organized by IP address	Add the ability to create and delete agents for a host Add the ability to black out a host - instantly, or on a schedule Add the ability to pull logs for a host. Add the ability to deploy an agent package to a host	Add the ability to black out an agent - instantly, or on a schedule Add the ability to edit the set of rules associated with an agent Add the ability to pull logs for an agent

Add Object Blackout

In the case of blackouts, adding exact equivalence to Foglight 4 doesn't make sense. Foglight 5 is model oriented. Blacking out an agent or individual rule doesn't make sense for all Foglight 5 cartridges. Instead, adding the ability to black out rules on large objects like Hosts is important.

This kind of blackout is already possible using the command line. Having a UI to manage object blackout is an important short-term requirement. This is partly covered by adding the ability to black out a Host or Agent in the Agents dashboard (see previous section).

Complete the Default Report Set

All the reports that were available in Foglight 4 should be available in Foglight 5 out of the box. This work has already started, and is considered critical.

Implement Self-Clearing for Observation-based Rules

For the set of rules that had self-clearing in Foglight 4, there must be an equivalent in Foglight 5. This is less important than the previous three items because it is possible to manage these alarms today. However, the default behaviour should eventually change to make transition easier.

Conclusions

The transition from Foglight 4 to Foglight 5 can be challenging, but there is lots of support for Foglight 4 style workflows and administration. There is a plan in place to fix up the rough spots for Foglight 4 users. The good news is that getting to Foglight 5 opens up a new world of monitoring possibilities that simply weren't available on Foglight 4.

Appendix A: Foglight 4 Workflow Details

Subnetwork Operations:

• New Graph, Table, Report or Rule
• Start or Stop Agents

Host Operations:

• New Agent
• New Graph, Table, Report or Rule
• Start or Stop Agents
• Delete Host
• System Configuration...
• Shows information about the host plus the cartridges installed on that host
• Show Log...
• Pulls the rapsd log for the client running on that host
• Edit Description...
• Allows for the description field to be modified

Agent Operations

• Start or Stop Agents, depending on state
• Update the agent
• Diagnose the agent.
• Edit Rules
• Edit ASPs
• Delete Agent
• Show Log...
• Pulls the log for the agent

Appendix B: Foglight 5 Equivalent Operations

To perform the same activities as the FOC, several Foglight 5 dashboards are required.

Operation	Foglight 5 Equivalent	Pictures
New Graph, Table or Report	Right-hand pane New Dashboard or New Report
Create/Edit Rule	Administration-> Create Rule or Manage Rules	Not as convenient
Start/Stop Agents	Agent Status
Start/Stop Agents	Agents Dashboard
Delete Agent	Agent Status->Hit the Red X	See earlier image
Delete Host	No equivalent
System Configuration	Agents -> Host Monitor	Not as convenient
System Configuration-> Cartridge Information	No equivalent
Show Log...	Agent Status->Show Log button at bottom	See earlier image
Edit Configuration	No equivalent	Marginal benefit
Update the agent	Agent Hosts
Edit ASPs	Agent Status -> Edit Propertes	See earlier image
Access Views	Agents -> Click on agent

 

Tags: