Home› Blog› AWS Daily Intelligence #83 - Catalog-scale anomaly…
AWS Daily Intelligence AWS

AWS Daily Intelligence #83 - Catalog-scale anomaly detection, and the anomaly it learns to call normal

Verified against current vendor documentation on 21 November 2026. Pricing, limits and API behaviour were checked against the official docs on that date. Cloud services change fast — if you are reading this much later, treat the specifics as a starting point and re-check the linked sources.

The scaling story is the real one: define checks once for a catalog and every table in it gets them, including tables that do not exist yet. The part to think about before switching it on is what happens to an anomaly nobody looks at, because the documented answer is that it becomes normal.

“When an anomaly is detected, it is considered normal for subsequent runs. The machine learning algorithm will consider this anomalous value as input unless it is explicitly excluded.”

Executive summary

“AWS Glue Data Catalog introduces data quality monitoring for catalogs and databases, powered by AWS Glue Data Quality, which profiles and evaluates your tables at catalog scale from a single configuration”. The scoping is the feature: “You define data quality checks once for a catalog, a database, or a single table, and AWS Glue Data Quality evaluates every table in scope, including tables added later, on the cadence you choose”.

Triggers are flexible and one option is new: “evaluations run on a schedule or whenever table metadata changes, including every Apache Iceberg commit”. Results go “through the existing AWS Glue Data Quality APIs” or, opt-in, to Iceberg tables or S3. It is available “in all AWS Commercial, AWS GovCloud (US), and AWS China regions”.

The anomaly detection is the headline — “After three runs on a table, AWS Glue Data Quality flags anomalous statistics that drift outside expected bounds, surfacing changes that no rule covers” — and it is the part whose documented behaviour changes how you should deploy this. It needs feedback it will not get at catalog scale, its default mode does not match the trigger being promoted, and it is billed per statistic here while being free in the other place you can run it.

What changed

Previously a DQDL ruleset was attached to a table or run inside an ETL job. The unit of configuration is now the catalog or the database, and membership is dynamic — tables added later inherit the checks with no action.

That removes the failure mode where a new table lands in a governed database and simply has no quality checks because nobody remembered. In a lake where tables arrive from pipelines rather than from people, that is most of them.

You can supply “built-in profiling options that AWS Glue provides for common baseline statistics” or your own DQDL ruleset “with analyzers, rules, or both”. Rules assert expectations; analyzers gather statistics without asserting anything. Useful detail if you use both: “AWS Glue Data Quality will gather statistics only once, even if you have both Rule and Analyzer for the same columns”.

Architecture

Diagram: AWS Glue Data Catalog introduces data quality monitoring for catalogs and databases, powered by AWS Glue Data Quality, profiling and evaluating tables at catalog scale from a single configuration, where checks defined once for a catalog, database or table are evaluated against every table in scope including tables added later. Built-in profiling or a custom DQDL ruleset with analyzers, rules or both can be used, and evaluations run once at creation or on triggers, on a schedule or whenever table metadata changes including every Apache Iceberg commit. After three runs on a table Glue flags anomalous statistics that drift outside expected bounds, matching the documented minimum of three data points to detect anomalies. The feedback loop is the key constraint: when an anomaly is detected it is considered normal for subsequent runs and the machine learning algorithm considers that anomalous value as input unless explicitly excluded, so retraining by acknowledging or rejecting anomalies is critical, and at catalog scale across thousands of tables that per-table triage does not happen, meaning a genuine regression is absorbed into the baseline within a few runs. Compounding it, when anomalies are generated data quality scores are not impacted, so an anomaly does not fail anything by itself. Mode selection matters: Linear is the default and models trends and seasonality and suits evaluations on a regular schedule, while Fixed treats all data points as equally spaced and suits irregular or unpredictable intervals, so the headline trigger firing on every Iceberg commit produces the irregular case that the default mode is documented as unsuited to. Cost carries an asymmetry: there is no additional charge for detecting anomalies in AWS Glue ETL jobs, but in the Data Catalog every statistic is charged 1 DPU for the time it takes to detect anomalies, on average 10 to 20 seconds per statistic, at 0.44 dollars per DPU-hour billed per second with a one-minute minimum, so roughly 18 dollars for ten thousand statistics across a thousand tables. Statistics storage is free but limited to 100,000 statistics per account and retained for a maximum of two years, so ten passes of ten thousand statistics reaches the cap. Glue gathers statistics only once when a rule and an analyzer cover the same columns, and the time component is crucial because limits differ for the same values over one-minute, hourly and daily intervals. Available in all AWS Commercial, GovCloud US and China regions.

The feedback loop, and why catalog scale breaks it

Anomaly detection here is not a threshold. It learns: “AWS Glue Data Quality requires a minimum of three data points to detect anomalies”, then predicts a range and flags values outside it. Fine so far.

What matters is what happens next. “Retraining the anomaly detection model is critical to detect the right anomalies. When anomalies are detected, AWS Glue Data Quality includes the anomaly in the model as a normal value.” The documentation is explicit that correcting this is a human action — acknowledging or rejecting the anomaly through Glue Studio or the APIs.

Per table, with an owner watching, that is a reasonable design. Applied to a catalog it inverts. A row count that halves because an upstream extract silently failed gets flagged once. If nobody rejects it, the model takes the halved value as a legitimate observation, and within a run or two the new normal is the broken normal. The detector quietly stops being able to see the problem it detected.

And it will not have blocked anything on the way past: “When anomalies are generated, data quality scores are not impacted.” An anomaly is an observation, not a failure. Nothing downstream fails because of one unless you build the thing that makes it fail.

So the honest reading is that this launch scales the generation of anomalies without scaling the triage of them. Which is fine — as long as you deploy it knowing that is what you bought, and put the queue somewhere a person actually works.

The default mode does not match the headline trigger

There are two modes. “Linear (default) models trends and seasonality in your data over time”, and the documentation says to use it when evaluations run on a regular schedule. “Fixed treats all data points as equally spaced regardless of the actual time intervals between evaluation runs”, for irregular or unpredictable intervals. “If you do not specify a mode, anomaly detection uses the LINEAR mode by default.”

Now put that beside the trigger the announcement leads with: evaluations that run “whenever table metadata changes, including every Apache Iceberg commit”. Commits are not a schedule. A table might get forty in an hour during a backfill and none for two days.

That is the textbook case for FIXED, and the default is LINEAR. Worse, the interval is load-bearing for the model: “The limits will differ for the same values over a one-minute interval, an hourly interval, or a daily interval.” A trend-and-seasonality model fed irregularly spaced points will produce bounds that reflect commit cadence as much as data behaviour.

How you trigger evaluationsMode to setWhy
Nightly or hourly scheduleLINEAR (default)Regular spacing; trends and seasonality are real signal
On every Iceberg commitFIXED, explicitlyIrregular intervals; documented as the Fixed case
On metadata change, low-churn tablesFIXEDLong unpredictable gaps between points
Mixed estateSeparate configurationsMode is set per run, so one cadence per configuration

The mode is set with ObservationMode in AdditionalRunOptions, alongside the ObservationScope: ALL that enables anomaly detection for Catalog tables at all. Both are opt-in fields, which means the quiet path — create a configuration, accept the defaults — gives you the schedule-shaped model regardless of how you actually trigger it.

Cost is per statistic, and free somewhere else

“Every statistic is charged 1 DPU for the time it takes to detect anomalies.” The pricing page puts a duration on it: “You will incur 1 DPU per statistic for the time it takes to detect anomalies. On average, it takes between 10 -20 seconds to detect anomaly for 1 statistic”, against “$0.44 per DPU-hour, billed per second with a 1-minute minimum”.

At the 15-second midpoint that is roughly $0.0018 per statistic. A thousand tables carrying ten statistics each is 10,000 statistics, or about $18.33 per full catalog pass for the anomaly detection alone, before the evaluation tasks that produced the statistics. Modest per pass; the variable is how often your trigger fires, and the Iceberg-commit trigger is the one that can fire a lot.

The asymmetry is the thing worth knowing: “There is no additional charge for detecting anomalies in AWS Glue ETL jobs.” The same capability, on the same statistics, is free inside an ETL job and billed per statistic in the Data Catalog. If your tables are already written by Glue ETL jobs, running detection there instead is a legitimate cost decision rather than a workaround.

The account-level ceiling nobody mentions

“There is no cost to store the statistics. However, there is a limit of 100,000 statistics per account. These statistics will be stored for maximum of two years.”

Catalog-scale configuration meets an account-scale cap. At 10,000 statistics per pass that is 10 passes before the limit, on the reading that the cap counts stored statistics cumulatively. The documentation does not say whether 100,000 counts distinct statistic series or individual stored data points, and the difference is enormous — so this is a limit to measure against your own catalog early rather than a number to plan around confidently.

Either way it is an account-level resource that a single catalog-wide configuration can consume on behalf of every other user of that account. Worth knowing before enabling it broadly in a shared account.

Business value

The dynamic-membership property is what justifies this. Data quality coverage decays because tables are created faster than checks are attached to them, and every catalog has a long tail of tables nobody ever configured. Scoping checks to a database and having new tables inherit them fixes the decay rather than the symptom.

Analyzers without rules is underrated here. For a catalog you do not understand well enough to write assertions about, gathering baseline statistics on everything and letting drift surface is genuinely more useful than a handful of carefully authored rules on the five tables somebody had time for.

The Iceberg-commit trigger is the most interesting operational capability, because it moves evaluation from a schedule to the actual event that changes the data. Just set the mode to match.

Security considerations

Statistics are data about data, and some of them are close to being data. Column-level statistics such as minimums, maximums and distinct counts can disclose information about a sensitive column to anyone who can read the quality results — a minimum and maximum on a salary or date-of-birth column is a meaningful leak even without a single row.

The statistics are stored in the Glue service and can be encrypted with customer-managed KMS keys. If you opt in to writing results to Iceberg tables or S3, that output is a new copy of this metadata in a location with its own access policy, and it should be treated as being as sensitive as the most sensitive column it profiles.

A catalog-wide configuration is also a broad grant by construction: the role running evaluations needs read access to every table in scope. Scoping to a database rather than a whole catalog keeps that blast radius proportionate.

Cost considerations

Two charges: the evaluation tasks that produce statistics, at “$0.44 per DPU-hour, billed per second with a 1-minute minimum”, and anomaly detection at 1 DPU per statistic. The one-minute minimum matters at catalog scale, because a very fast evaluation on a small table still bills a minute of its DPUs.

The cost driver is trigger frequency multiplied by tables in scope multiplied by statistics per table. A catalog-wide configuration on an Iceberg lake with per-commit triggers multiplies all three at once, which is the combination to model before enabling rather than after.

Run anomaly detection in Glue ETL where the tables are already produced by Glue ETL, since there it carries no additional charge.

Operational considerations

Decide who reads the anomalies before you generate them. Results surface through the Glue Data Quality APIs, so the queue needs a consumer — a ticket, a channel, an owner per database. Without one, the feature is generating observations that will be absorbed back into the baseline unreviewed.

New tables are unmonitored at first: three data points are required before anything can be flagged, so a table created today is not protected by anomaly detection today. Rules, unlike anomalies, work on the first run.

Do not change the evaluation cadence casually once a model has learned, since the bounds depend on the interval between points.

Tradeoffs

ApproachWhat you getWhat it costs
Catalog-wide profiling with analyzers Coverage of every table including future ones, with no authoring Per-statistic cost, account statistic cap, and an anomaly queue nobody owns yet
Database-scoped DQDL rulesets Assertions that actually fail, on a proportionate blast radius Authoring effort, and rules go stale as the business changes
Anomaly detection in Glue ETL instead The same detection at no additional charge Only covers tables written by Glue ETL jobs
Per-commit triggers on Iceberg tables Evaluation tied to the event that changed the data Unpredictable cost, and FIXED mode must be set explicitly

Implementation guidance

Start with one database rather than a catalog, with built-in profiling and a schedule. Measure the statistics produced per table and the DPU time consumed over a week, then extrapolate before widening scope.

Set ObservationScope: ALL to get anomaly detection at all, and set ObservationMode deliberately — FIXED for metadata-change and per-commit triggers, LINEAR only where evaluations genuinely run on a schedule.

Wire the results API to something a person reads, and make acknowledging or rejecting an anomaly part of that workflow. That single step is what keeps the model from learning your outages as normal.

Best practices

  • Set the observation mode explicitly; the default assumes a regular schedule.
  • Give every anomaly a reviewer, because an unreviewed anomaly is absorbed as normal.
  • Use rules, not anomalies, for anything that must actually fail — anomalies do not affect the score.
  • Scope to a database rather than a catalog, to bound both the IAM grant and the cost.
  • Treat statistics output as sensitive metadata; encrypt it and restrict the opt-in S3 or Iceberg destination.
  • Measure against the 100,000-statistic account limit early, since the documentation does not define what it counts.

Who should adopt this and when

Now: teams with a governed database whose table count grows faster than their check coverage. Built-in profiling on a database scope, on a schedule, is a clear improvement over nothing and needs no DQDL authoring.

Now, with a reviewer named: anyone turning on anomaly detection. The feature's accuracy is a function of someone triaging its output.

Model the cost first: Iceberg-heavy lakes considering per-commit triggers across a catalog. That is the configuration where all three cost multipliers move together.

Key takeaways

The scaling is genuine and well designed: “AWS Glue Data Quality evaluates every table in scope, including tables added later, on the cadence you choose”, which attacks the real reason quality coverage decays.

The sentence to plan around: “When an anomaly is detected, it is considered normal for subsequent runs.” Generating anomalies across a catalog without triaging them produces a detector that learns to ignore your failures, and anomalies do not affect the quality score in the meantime.

Two concrete defaults to override: the mode, because LINEAR assumes a schedule and the headline trigger is per-commit; and the venue, because detection is billed per statistic in the Catalog and carries “no additional charge for detecting anomalies in AWS Glue ETL jobs”.

Official AWS references

Comments

How was your experience?
Your feedback helps improve this site.
PoorExcellent