HERMES Network anomaly detection · M-Lab

How to read HERMES

HERMES turns raw M-Lab speed tests into anomaly events — moments when a group of users sharing a client AS and city saw their connection get measurably worse (higher latency, lower throughput) versus their own normal baseline. For each event it then tries to answer the hard question: which part of the network is the source? This page explains how it does that and how to read every chart you'll see.

Group one client AS + one client location measuring to one M-Lab server, tracked over time. The unit everything is computed on. (“Client” is the end being measured; “server” is the M-Lab test target. Elsewhere “source” means what an anomaly originates from — the AS, link or metro HERMES holds responsible, which is what the Likely source column reports. That is usually somewhere along the path, but it can be the client's own network: when Detected via reads source, no path hop was implicated and the client's own AS is the thing held responsible.) How coarse “location” is — metro or city — is the detection regime, and it is stamped on every row; see the tag reference below.
Baseline that group's typical RTT / throughput over a trailing window — the "normal" each day is compared against.
Event a day on which the group's measurements drifted far enough from baseline to be flagged as anomalous.
Tomography the inference step that pinpoints the source to a specific AS or a link within / between ASes.
The three-step workflow
  1. Browse & filter — Overview. Set a date range and optional filters (client AS, city, anomaly type, scope, IP version), then scan the events table, the country map, and the events-explained-by-entity bar to see where and what is breaking.
  2. Drill into one event — Event Detail. Click any row in the table. You get the full evidence for that single event: the measurements, the statistical tests that flagged it, and the two tomographies that identify the source.
  3. Zoom back out — Population View. See every event in your current window as one population: how severe, how widespread, how often HERMES could identify a source.
Reading the Overview

The events table is one row per detected event. Key columns:

  • Daily Median vs Baseline (Throughput Mbps, RTT ms) — the day-of value next to the group's normal. A drop in throughput or a jump in RTT is the anomaly.
  • Likely source — the entity the multi-granularity set-cover holds responsible for this group's anomaly (the same one shown in Event Detail), e.g. AS3257 · Chicago. Its granularity and interdomain/intradomain scope show as tags. It is the entity that best explains the day's anomalies across correlated groups — an inference, not a finding of fault, which is why the column no longer says “responsible entity”.

The Tags column, on the right of each row, summarises why it was flagged:

latency day-of RTT rose > 5 ms above baseline, on ≥ 80% of samples.
throughput day-of throughput fell below baseline, on ≥ 80% of samples.
distance a day-of measurement travelled farther than any baseline measurement — a hint the path detoured.
edgenodeASmetroIXP  the granularity of the responsible entity that the set-cover attributed this group to.
path-local  a singleton — this group's anomaly wasn't correlated with any other group, so it was attributed to its own path's finest entity (an edge) rather than a shared culprit. Most events are path-local; edge is reserved for correlated link culprits.
interdomainintradomain  whether the responsible scope crosses an AS boundary or stays inside one.
reroutecongestion  a conclusive temporal-tomography verdict — the path either changed (reroute) or stayed the same but degraded (congestion). Absent when the verdict is indeterminate. Filterable via Temporal verdict.
metro · ipinfo  the detection regime — see below. Note this is a different question from the metro tag above, which is about the responsible entity.
Detection regime — what grain was this run at?

HERMES has to decide which clients count as the same place before it can detect anything. That choice is not fixed for all time: the detector has run under more than one regime, and rows from different regimes sit side by side in the same table. Every row now carries an outlined tag reading <grouping shape> · <geolocation source>, so you can always tell which one produced it.

grouping shape how coarsely clients were bucketed into one group. metro pools a whole metropolitan area (the current regime); city keys on the geolocation database's city string. A coarser shape means more measurements per group — better statistics — but a real problem confined to one city can be diluted by neighbours that were fine.
geolocation source which database placed the client: ipinfo (current) or maxmind (legacy). The two disagree about where plenty of prefixes are, so a group can change shape purely because the provider changed — with no change in the network at all.
unstamped the partition predates the pipeline recording this. Those partitions were city-grouped from MaxMind, but that comes from the pipeline's history, not from a value stored on the row — so the tag says what is actually known rather than back-filling an assumption.
dashed border the anomaly could not be resolved back to a tested group, so its group label is best-effort. Hover the tag for the specifics.

Why this matters when comparing dates. A jump in event counts across a regime change is not necessarily a change in the internet. Re-grouping clients changes both the number of groups and how many measurements each one has, which moves the detection thresholds underneath you. Before reading a trend across the boundary, check that the tag says the same thing on both sides.

The Culprit granularity and Scope filters in the toolbar let you zoom straight into a class of cases — e.g. only metro-wide outages, or only interdomain edges.

Events by country shades each country by event count. Events explained by entity ranks the ASes / segments that tomography flagged as the source most often across the window. Both honour the filters above.

Reading Event Detail — card by card
Measurement flow

A Sankey of how this group's measurements flow through the network. Red marks the sub-group that carried the events.

RTT, throughput, distance & loss over time

Each metric over time for the group. The orange band is the period of interest (the event day); the horizontal line is the baseline. A ★ star marks tests that also carry a reverse traceroute (only ~25% do). Click any point to load that exact measurement into the cards below.

Detection statistics — day-of vs baseline

The actual tests that flag an event. A row passes (pass) or not ():

  • Anomalous-sample ratio ≥ 80% — most samples moved the wrong way, not just the median.
  • Median RTT shift > +5 ms / throughput below baseline — the central tendency moved meaningfully.
  • Welch t-test p < 0.05 and Mann-Whitney p < 0.05 — the day-of vs baseline difference is statistically significant (parametric and rank-based).

The bold Anomaly verdict row is the same rule that listed the event in the Overview.

Correlation tomography — responsible entity

A single mixed-granularity set-cover attributes each anomalous group to exactly one responsible entity, picking the right level for the evidence — one of:

  • edge a link between two ⟨AS, metro⟩ nodes · node one ⟨AS, metro⟩ · AS a whole network · metro a whole city · IXP an exchange.

How it chooses (shown in the card so you can audit it):

  • Eligibility — an entity counts only if a material share of its own paths are anomalous (purity ≥ 10%) and it's statistically significant (Fisher's-exact p, odds ratio).
  • Coverage-first — among eligible entities it picks the one explaining the most still-unexplained anomalies, then repeats on what's left.
  • Zoom out only when broad — it reports a coarser entity (a whole AS or metro) only when that entity is still high-purity on the remaining anomalies and spans ≥ 2 distinct sub-entities; otherwise it keeps the finer one. So a metro is flagged only when several of its ASes are affected, an AS only when it's broadly bad. The card's drill trail shows when a coarser candidate was considered and rejected.
  • Path-local — a small singleton tail (groups not pooled with anyone) is attributed on their own path at the finest level. Lower confidence than a correlated entity.

The funnel shows the co-affected groups ( = yours) converging on the entity; purity = anomalous ÷ (anomalous + healthy) paths through it. Click any group in the funnel to drill into that event.

Temporal tomography — usual vs day-of path

Did the event come from the route changing or the same route degrading? The card leads with a one-line summary verdict:

  • Reroute — the path changed; the summary names the links traffic diverted onto and stopped using, with the before → after share of paths.
  • Congestion in place — the path stayed the same but got slower, pointing at load rather than a routing change.
  • Path stable / Inconclusive — no material change, or not enough path data to classify.

Below the summary, the What changed table lists every link the path gained or dropped (biggest swing first); the full forward/reverse path diagrams are tucked into a collapsible section.

How “usual” and “day of” % are calculated

Both are a prevalence: of the measurements in that window, what fraction traversed this link. Two windows are compared, and they are split by health, not purely by time — this is the part that most often surprises people:

  • usual = the group's healthy measurements — every measurement not flagged anomalous, including ones taken on the event day itself.
  • day of = the group's anomalous measurements whose window starts on or after the event day.

A measurement is anomalous when its RTT is ≥ 5 ms above the group's baseline median with an anomaly ratio ≥ 0.8, or its throughput is below baseline on the same ratio test. Each path is reduced to its AS-metro hops, and every consecutive pair of hops is one link. So usual 90% → day of 10% means nine in ten healthy paths crossed that link but only one in ten anomalous ones did.

The two percentages are independent and do not sum to 100. Each is divided by its own window's measurement count, and those counts are per direction — the forward figures are over measurements that have a forward path, the reverse figures over those with a reverse path. That matters because reverse traceroute exists for only ~25% of tests; the panel headers state which count each percentage is against.

How “diverted onto” is calculated

A link is labelled diverted onto when its share grew from the usual to the day-of window: either it was absent from the usual path entirely, or its share rose by at least 15 percentage points. abandoned is the mirror image — absent on the day, or down by 15 points or more. Anything in between is unchanged and is left out of the table. The Δ column shows the swing in percentage points, which is the quantity being thresholded — not a ratio.

The 15-point threshold decides what this table highlights. The reroute-vs-congestion verdict above it is computed separately in the pipeline, over the same prevalences but with its own thresholds, so an edge can appear here without being the one the verdict names.

What “agrees with the responsible entity” means

Two independent methods pointing at the same place. Correlation tomography finds the responsible entity by looking across groups — which AS, link or metro is shared by many anomalous paths and few healthy ones. Temporal tomography looks within this one group over time and names a single segment:

  • for a reroute, the link that gained the most share on the changed direction;
  • for congestion in place, the link whose share barely moved but whose per-hop RTT rose the most.

The ✓ agrees with the responsible entity chip appears when that segment is one of the entities the correlation set-cover independently blamed. It is a corroboration signal, not an extra test: its absence is weak evidence, since the two methods can legitimately name different parts of the same path. Its presence is the strongest single indication that the attribution is right.

Logical & geographic paths

The forward and reverse paths drawn two ways: logical (AS / hop sequence, with reached vs not-reached hops) and geographic (the same hops on a world map). Use these to sanity-check that the source segment sits where the path actually goes.

Reading Population View

Every event in the window as one population. The summary cards:

  • Events — count in the window. Throughput drop / RTT increase — median severity, with p95 for the tail.
  • IPs / group — median distinct users behind a group (a proxy for blast radius).
  • Inter-domain — share of classified events whose source crosses an AS boundary.
  • Source identified — share of events where tomography actually identified a source (vs. inconclusive).

Below: a per-day event-count trend and distributions (throughput drop, RTT delta, unique IPs on a log scale, and inter- vs intra-domain split) across all events.

Glossary
AS / ASN
Autonomous System — a network operator (e.g. an ISP) identified by a number. Paths cross one or more ASes.
Intradomain vs Interdomain
A responsible edge whose two ⟨AS, metro⟩ ends are in the same AS vs. ones that span two different ASes.
RTT
Round-trip time in ms — the latency metric. Higher is worse.
Throughput
Download speed in Mbps. Lower is worse.
Baseline vs day-of
The group's trailing-window normal vs. the value measured on the event day.
Reverse traceroute
The path back from the M-Lab server, available for only ~25% of tests; lets HERMES attribute the source to the return direction, not just the forward one.
Granularity
The level of the responsible entity: edge (link), node (⟨AS, metro⟩), whole AS, metro, or IXP. The set-cover picks the coarsest level that's still broad and high-purity.
Purity
Of all paths crossing an entity, the share that are anomalous — anomalous ÷ (anomalous + healthy). An entity is eligible only at ≥ 10%.
Demoted-from / drill trail
A coarser candidate that was considered but rejected in favour of the finer entity actually reported (because it wasn't broad enough).
Attribution method
correlation (the set-cover pooled this group with others) or path-local (a singleton, attributed on its own path).
Interpreting responsibly
  • Reverse-path evidence exists for only ~25% of tests — absence of a reverse finding isn't absence of a reverse problem.
  • Tomography narrows correlation toward cause, but a weak or path-local verdict is a lead, not a proof.
  • Geolocation can be wrong. If a city or country looks off, use the ⚑ flag control on the measurement to submit a correction.
  • The inference can be wrong, and telling us is how it improves. Every row in the Overview table carries a control next to its . Use it when the responsible entity, the inter-/intra-domain scope, the reroute-vs-congestion verdict, or the event itself looks wrong. The flag is recorded per row, together with exactly what the dashboard showed you at the time, so a later algorithm change can be replayed against it.
  • A flag needs a reason — that field is required on purpose. "This is wrong" cannot be acted on; "the day-of path never crosses this link, so the correlation is with a co-located group" can. Say what evidence contradicts the inference, and what you think the answer should have been if you know.