Performance WG BCP Task Force conference call notes
This document is essentially the LHC Tier 2 Best Common Practices draft, along with notes summarizing the discussion from the 9/4/2008 conference call of the "BCP Task Force." The notes from the call are represented as text formatted the way this section appears (red text, red box, shady background).
Participants on the call: Jeff Boote, Carla Hunt, Joe Metzger, Siegrid Rickenbach, Mike Van Norman, ken lindahl.
Section heading numbers have been added, to make it easier to refer to specific parts of the document in subsequent discussion.
Some section headings are marked with a boxed number like 3 . These are an indication of the degree to which the section needs to rewritten:
3 = major rewriting will be required
2 = medium rewrite
1 = minor rewrite
0 = perfect, no rewriting
(Please note that these indications were not determined during the conference call; they are solely ken's determinations and should be interpreted with appropriately sized grains of salt.)

Introduction 3

This documentation is a recommendation from the US PerfSONAR community to the US LHC community that describes how the US LHC Tier-1, Tier-2 and Tier-3 centers can effectively utilize the network measurement infrastructure tools developed by the perfSONAR collaboration to debug, monitor and manage the network paths critical to their center.
The LHC computing model is a complex distributed workflow system that relies on a large number of compute, storage, and network services provided and supported by many different organizations around the globe. Many of the components of this system are new, and have never operated as production infrastructure on this scale before.
The way the LHC community is going to use the global networks is significantly different than most prior large science experiments. It is expected that this new fashion of using and stressing the research and education network infrastructure will probably bump into previously unknown problems or limitations in some parts of the global infrastructure. Simple faults where a single system fails completely until it is repaired are usually easy to diagnose and repair. However, transient faults and subtle partial failures in a system as large and complex as the LHC computing model can be very difficult to track down.
Deploying a perfSONAR network measurement infrastructure in the LHC Tier 2 centers will make the network components of the workflow system more predictable and deterministic. It will make it trivial to determine if the network services are up and functioning correctly, or suffering some impairment. It should reduce the effort required to diagnose complex workflow problems, and it should allow the LHC scientists to focus their time on other parts of the computing model, or the LHC science.

Goals 2

Allow LHC Scientists to easily:
1. Characterize and track network connectivity between their center, and the centers they serve or rely on.
2. Characterize and quantify network performance problems to accelerate diagnosing and fixing them.
3. Differentiate between application and network performance problems.
4. Differentiate between local and remote network problems.
5. Identify, understand and respond effectively to changes in the underlying network.

Use Cases 1

This section seems pretty long; shorten? Move to a different section, e.g. appendix?
There are many use-cases for robust network measurement capabilities. We expect common use cases in the US LHC community to include:

...

The network measurement infrastructure will support the following network measurements. These can be used to characterize the network between points of interest, and in some cases all along the path.

General Diagnostics 1-2

Continuously measure end to end delay

What

• Manage a local star as opposed to a mesh, but neither term is defined in the document configuration of continuous latency measurements from a local measurement point to remote collaborators and store results in an Measurement Archive (MA)
• Publish results via a standardized web service interface

...

Monitor up down status of cross domain circuits

Move this section to later in the document, discuss monitoring layer 3 links first.
What

• Determine the status of a circuit
• Publish status via a web services interface

...

Monitor Link Circuit Capacity Utilization and Errors

Need to specify inclusion of RON-to-campus links (layer 3 and below).
What

• Publish interface utilization & error statistics via a web services interface

...

Diagnostics to look for specific known performance problems 2

What

• Make diagnostic servers available (a la NDT/NPAD)

Both NDT and NPAD use TCP. NPAD is designed for looking at short paths. Jeff: work is underway for a common scheduler, so that NDT and NPAD cannot step on each other.
Why

• T-2 sites will be accessed by end-users. These tools make it possible to diagnose paths not related to LHC T-X traffic patterns, but all the way to the scientist.
• Helps inform the scientist what performance they should expect.

Tools 1-2

There are several tools that can be used to provide the measurements listed above. The first priority in this recommendation is to ensure that diagnostic measurements can take place, but those diagnostics are most useful given historical data for comparison. Therefore, the specific tool recommendations in this list were chosen based on the ability of the tool to work in both an on-demand mode for diagnostics as well as a scheduled mode for on-going monitoring and historical analysis. Additionally, because network performance diagnostics is inherently a multi-domain effort, the perfSONAR framework is used as a medium to share the results of the measurements.
There are several domains within which these tools will be used: RONs, campuses, application communites. A "lookup service" will be needed to find perfSONAR deployments. Development is already underway; should be ready sometime soon.

Delay Measurements

diagnostics:

Two main tools are recommended in this area:
1. ping (ICMP)
Advantages:
No cooperation needed from remote site
Disadvantages:
round-trip metrics only
ICMP echos may be blocked
2. owamp
ken: how useful/important is owamp? some campus folks from CENIC community have expressed disinterest in supporting owamp, in part because of the need for an external clock. Jeff: owamp is the most useful tool for watching delay variation – indicating queuing. NDT is sufficient; an external clock is advantageous but not required.
Advantages:
one-way metrics such as jitter, hops available
direction of performance problems is isolated
Disadvantages:
Remote site must be running an OWAMP daemon (owampd)
Sites need to have reasonably synchronized clocks (NTP)

...

Two tools can be used for on-going monitoring with the decision upon which depending largely on the amount of cooperation between the sites. (It is expected that sites that are serious about monitoring will implement both.)
1. pingER along with the perfSONAR-PS pingER-MA
pingER is used to set up a regular set of ping measurements and creates an archive of them. The pingER-MA is used to share that archive using perfSONAR compliant interfaces. The advantages/disadvantages from the ping diagnostics are relevant.
Many campuses (and perhaps some RONs) have already have smokeping installations, some of them fairly extensive. Guidelines should discuss integration of smokeping into the perfSONAR environment. Jeff: this is already on the list of things to be developed.
2. perfSONARBUOY
perfSONARBUOY is used to setup a regular set of owamp measurments and creates an archive of them. perfSONARBUOY exposes a perfSONAR compliant interface to share that archive. The advantages/disadvatages from the owamp diagnostics are relevant.

Bandwidth Measurements

Here, and perhaps elsewhere in the document, we need to address the IPv4 / IPv6 issues. One approach has been to use v4- and v6-specific names so that the person running the test can be assured of testing over the desired version of IP. (Also, there is an issue with the current bwctl software: the client connects to both endpoints (bwctl servers) to set up the connection. If an endpoint is v4- and v6-capable, v6 is selected, even if the other endpoint is v4-only. But that needs to be fixed independent of these guidelines.)
diagnostics: bwctl
Advantages:

...

The measurement infrastructure contains multiple components that may influence each other making results analysis more difficult. The primary example is that bandwidth tests run on the same computer as continuous latency measurements will affect the latency measurement results. In order to simplify the analysis process, bandwidth measurement points and latency measurement points SHOULD NOT be deployed on the same physical machine.
Measurement points SHOULD be deployed as close to the network administrative boundaries as possible. inside or outside of firewalls? or both? The reason for this is to facilitate diagnosing problems using path decomposition techniques and to make the resulting data as actionable as possible.
If the measurement point is outside the firewall, some sites will not take responsibility for the security of the device, they will want someone else to manage it. This will be a problem: local ownership is really needed, otherwise the system will eventually die – e.g. AMP project.

Bandwidth Measurement Infrastructure

...

Bandwidth Measurements

Schedule

These specifications are probably appropriate for RON-to-RON testing, but may need adjustment for RON-to-campus and campus-to-campus testing, depending on what applications, and what kind of research, the campus (or pair of campuses) is supporting.

• Bandwidth tests MUST be run at least once a day.
• Bandwidth tests SHOULD be run once every 6 hours.
• Bandwidth tests SHOULD transfer data for 60 seconds.
• Bandwidth tests SHOULD NOT not transfer data for more than 120 seconds.
• Bandwidth tests between 2 endpoints SHOULD NOT run more often that once an hour.

...

The implementation guide section of this document describes how to deploy a network measurement infrastructure.
This section probably will need to address differing guidelines for RONs vs campuses. Again, this is likely to be highly influenced by what applications are being supported, what kinds of research are being supported.
Based on what we expect to be available at Tier-1 sites, these tools and configurations would be useful at Tier-2 sites. This should remain fairly high-level, with the expectation that we will create very detailed instructions for the LHC community accepted portions.

...

It is possible to participate in the network measurement infrastructure at different levels. Your organization will get different levels of benefits depending on the level of participation.
EXPAND STRAWMAN BELOW....

Non Participant

Need a better descriptor for this class of participant.
Non-participants do not expend any effort "do not expend any effort" is probably not the right description, and have no control of network measurements made from remote sites into the local infrastructure.
Participating in the measurement infrastructure at this level provides you information about, and some level of control over the measurements made to your local site from remote locations.
You need to do the following to participate at this level:
use the <INSERT NAME HERE> Knoppix CD.
Or, install the following (a, b, c) from (src)

...