AWS Architecture Series #31 — Iceberg on S3: Why Table Formats Arrived

Business Challenge

The previous post ended on a table with four schemas and no arbiter, and that is the honest summary of schema-on-read: the catalog describes files, the description is applied at query time, and when the description and the files disagree the reader decides. Table formats are what happened when a decade of that argument was lost. Apache Iceberg does not add a feature to that model — it moves the table's definition out of the catalog and into the data, so that the list of files belonging to a table is a written, versioned artefact rather than whatever a LIST against a prefix returned this morning.

That change lands cleanly. Athena describes it in one sentence: each Iceberg table maintains a versioned manifest of the S3 objects that it contains. From that single property everything the format is known for follows. A commit is atomic because it swaps one pointer. A delete is real because the row's absence is recorded rather than inferred. Time travel exists because the old manifest was never overwritten. Schema evolution means something because columns have identity independent of their name and position.

The part that does not land cleanly is what those manifests cost to keep. A Hive table on S3 has no maintenance: objects arrive, the crawler notices, and nothing accumulates that a reader must traverse. An Iceberg table accumulates snapshots, manifests, metadata files and — on AWS in particular — delete files, all of which a query engine reads before it reads any data. Left alone, an Iceberg table gets slower and more expensive in a way a Hive table does not.

So the format solves a correctness problem and creates a scheduling problem. AWS answers the scheduling problem three separate times: manually through Athena's OPTIMIZE and VACUUM, automatically through AWS Glue Data Catalog table optimizers, and automatically-and-not-yours through S3 Tables. The three have different defaults, expressed in different units, with different things they refuse to do. Choosing between them is the actual architecture decision, and it is made after the interesting part — picking the format — is already over and everyone has moved on.

Architecture

The design question is not "should we use Iceberg" — for a mutable analytic table on S3 that is close to settled. It is "which component owns this table's file list, and what is it allowed to delete".

What actually changed: the file list stopped being a directory listing

In the Hive model a table is a prefix plus a set of partitions in the catalog, and the files in a partition are whatever is under its location. The catalog does not know how many objects exist, which is why the scale numbers do not line up: Athena will query a Glue table with 10 million partitions, but it cannot read more than 1 million partitions in a single scan. That is a factor of ten between what the catalog is willing to hold and what a query can traverse, and the gap is bridged by hoping the predicates prune well.

Iceberg replaces the enumeration with a tree. A pointer in the catalog names the current metadata file; the metadata file names a snapshot; the snapshot names a manifest list; the manifest list names manifests; the manifests name data files, each with per-column bounds. A query prunes at the manifest level using those bounds, so the work is proportional to what the predicate matches rather than to how the data is laid out. This is why hidden partitioning is not a marketing phrase: the partition transform is recorded in the table, so a query filtering on a timestamp gets partition pruning without the analyst knowing a partition column exists, and the transform can be changed later without rewriting the query or the old data.

Two consequences that are easy to miss. First, the catalog's role shrinks to holding one pointer and swapping it atomically — which is why Athena is explicit that it supports AWS Glue optimistic locking only, and warns that modifying an Iceberg table through any other lock implementation "will cause potential data loss and break transactions". The atomicity everything else rests on is a property of that one swap. Second, the metadata is now a read cost. Every query walks the tree before it touches data, and the tree grows with every commit until something prunes it.

Row-level change is real, and the write mode is not yours to choose

UPDATE, DELETE and MERGE INTO in Athena follow the Iceberg v2 positional delete specification and enforce snapshot isolation. Athena only creates and operates on v2 tables, and supports Iceberg 1.4.2 with Parquet, ORC and Avro data files. That is a genuine capability and it is the reason most teams arrive here: erasure requests, late-arriving corrections and CDC upserts stop being a full-partition rewrite.

What is worth knowing before designing around it is that Athena implements exactly one of the two Iceberg write modes. It always uses merge-on-read. The documentation is unusually direct about the consequence:

Read that as an operator. A table property set deliberately, reviewed in a pull request, and present in the table's metadata is inert, and nothing announces it. The effect is not cosmetic: merge-on-read defers the cost of a delete to every subsequent read, and the deferred cost only comes back down when compaction merges the delete files into data files. So on Athena, compaction is not a storage-cost optimisation that can be postponed — it is the second half of every DELETE that has been run.

Which makes the compaction thresholds worth reading as policy rather than as tuning knobs. optimize_rewrite_delete_file_threshold defaults to 2: a data file with fewer than two associated delete files is not rewritten, deliberately, to let delete files accumulate and make the eventual rewrite worth its cost. optimize_rewrite_data_file_threshold defaults to 5, and both are capped below 50. Raising them buys cheaper compaction and slower reads, in that order.

-- Compaction is charged by data scanned, and only partition columns
-- may appear in the predicate. Scope it, or the table is rewritten.
OPTIMIZE sales.orders REWRITE DATA USING BIN_PACK
  WHERE event_date >= DATE '2026-08-01';

-- Snapshot expiry and orphan removal. Set the retention first --
-- the default is five days, and this is what ends time travel.
ALTER TABLE sales.orders SET TBLPROPERTIES (
  'vacuum_max_snapshot_age_seconds' = '2592000',   -- 30 days
  'vacuum_min_snapshots_to_keep'    = '10'
);
VACUUM sales.orders;

Three owners of maintenance, three sets of defaults

The same three jobs — compact files, expire snapshots, remove unreferenced files — are offered by three AWS surfaces, and picking one is picking a set of defaults that will probably never be revisited.

The row worth staring at is snapshot retention. Athena's default is 432,000 seconds and S3 Tables' default is 120 hours. Those are the same number — five days — written in two units, on two surfaces, in two consoles. It is a sensible default for a storage bill and a poor one for the reason most people say they want a table format. If the recovery story is "we can roll the table back", the untouched version of that story lasts until Friday.

Time travel is a retention policy wearing a feature's clothes

The syntax is deliberately unremarkable: FOR TIMESTAMP AS OF for a point in time, FOR VERSION AS OF with a bigint snapshot ID for a specific commit. Both replaced the FOR SYSTEM_TIME AS OF and FOR SYSTEM_VERSION AS OF forms from Athena engine version 2, which is worth knowing when inheriting a query that will not parse.

-- What the table said before this morning's load
SELECT * FROM sales.orders FOR TIMESTAMP AS OF (current_timestamp - interval '1' day);

-- Diff a snapshot against the current state
SELECT c.order_id
FROM sales.orders c
FULL JOIN sales.orders FOR VERSION AS OF 949530903748831860 p
  ON c.order_id = p.order_id
WHERE c.order_id IS NULL OR p.order_id IS NULL;

The warning attached to VACUUM is the whole story: run a snapshot expiration and you can no longer time travel to expired snapshots. There is no separate archive, no recycle bin and no soft delete. Retention is the feature, and the two automated owners both ship it at five days.

One asymmetry that catches people: vacuum_min_snapshots_to_keep takes precedence over the age. If the minimum remaining snapshots are older than vacuum_max_snapshot_age_seconds, they are kept and the age is ignored. So a table that has not been written to in a month still has its last snapshot, and a table written to every minute has five days. The retention that results depends on the write rate, which is not how anybody reads that setting.

Every failure mode here is a deletion or a stop

This is the part that repays reading the limitations pages rather than the launch posts. None of the following raises an error at the time it matters.

• Two tables, one prefix. When multiple Glue Data Catalog tables share the same S3 location and have optimizers enabled, one table's snapshot retention or orphan file deletion may delete files still referenced by the other. This is not a corner case in practice — a table and its _backup, or two catalog entries pointed at one curated path, gets there quickly.
• An S3 lifecycle rule over the table path. AWS states it plainly: lifecycle expiration rules applying to Iceberg table storage locations can delete manifest and data files still referenced by active snapshots. The standard "expire everything older than 90 days" rule, applied at the bucket level for cost hygiene, is a data-loss mechanism on an Iceberg prefix.
• Iceberg-native retention on an S3 table. Setting history.expire.max-snapshot-age-ms or history.expire.min-snapshots-to-keep as a table property does not configure S3 Tables snapshot management — it makes it fail for the entire table, regardless of the value. Any user-defined tag or branch does the same. The table keeps working, nothing expires, metadata grows, and the only signal is GetTableMaintenanceJobStatus.
• Four failures and compaction stops. Glue automatically suspends a compaction optimizer after four consecutive failures, to avoid burning compute. Correct behaviour, and it means an optimizer enabled six months ago is not evidence that compaction ran last night.
• The million-file ceiling. Glue snapshot retention and orphan file deletion delete at most 1,000,000 files per run. Anything eligible beyond that stays in table storage as orphan files — so the job reports success while the backlog it was meant to clear persists.
• Orphan deletion has an epoch. The Glue orphan file deletion optimizer only removes files created after the optimizer's own creation date. Files created before or on that date are never deleted. Enabling it to clean up years of accumulated debris does precisely nothing to the debris.
• VACUUM without delete permission. If the query execution role lacks s3:DeleteObject, the VACUUM query succeeds and no files are removed. A green query history and a growing bill.

Where S3 Tables changes the shape of the problem

S3 Tables is a bucket type whose subresource is a table, and the argument for it is not only that maintenance becomes somebody else's job. It is a request-rate argument. A general purpose bucket gives at least 3,500 PUT, COPY, POST or DELETE requests, or 5,500 GET and HEAD requests, per second per partitioned prefix, with no limit on the number of prefixes — a fine model when the key layout spreads naturally, and a poor one for an Iceberg table whose commits concentrate on a metadata path. AWS states S3 Tables deliver up to 10x higher transactions per second than Iceberg tables in general purpose buckets, and that is the number to weigh, rather than the maintenance automation.

The defaults are opinionated in a useful direction. Compaction targets 512 MB, configurable between 64 MB and 512 MB, with a strategy of auto that applies sort compaction to a table with a defined sort order and binpack to one without. S3 Tables applies the Parquet row-group default of 128 MB, so a compacted file at the default target holds four row groups — the unit a scan actually skips. Iceberg statistics and Parquet column indexes are on by default here and opt-in everywhere else, through use_iceberg_statistics and use_iceberg_parquet_column_index, which is a real difference in query plans and not a footnote.

The constraint to design against is the quota shape: 10 table buckets per Region per account, each holding up to 10,000 namespaces and 10,000 tables, all adjustable only by contacting Support. Ten is a small number if the instinct is one bucket per team or per environment. Namespaces are the intended axis of separation, and they are cheap; buckets are not.

The parts that are still sharp

Two limits worth surfacing before they are found during an audit. Athena does not support DDL on Iceberg tables registered with Lake Formation, and Lake Formation cannot manage permissions for VACUUM, MERGE, UPDATE or OPTIMIZE on Iceberg tables at all — it governs read access. If the governance model assumes Lake Formation mediates everything, maintenance operations sit outside it and need IAM.

And a data-fidelity one: Iceberg supports microsecond precision on timestamps; Athena supports milliseconds, on both reads and writes, and retains only milliseconds for time columns rewritten during manual compaction. For most analytics that is invisible. For event ordering, idempotency keyed on an event timestamp, or anything reconciling against a source system that records microseconds, it is a silent truncation performed by a maintenance job.

Why This Architecture Holds Up

The previous post argued that schema-on-read fails by changing answers, and that the failure is invisible in every job metric. Table formats fix that class of failure convincingly: the schema is in the data, column identity survives a rename, a commit either happened or did not, and the state of the table on any past day is a query rather than a restore. If the problem is correctness, the argument is over.

What the format does is trade a correctness risk for an operational one, and operational risks are only better if they are actually operated. An Iceberg table with no maintenance owner does not return wrong answers. It returns right answers, slowly, at rising cost, with a history that quietly stopped being retained — and the first symptom is a bill or a failed recovery, both of which arrive long after the decision that caused them.

That is why the monitoring has to change with the format. On a Hive table the useful alarms were about drift: table count per database, partition count per table. On an Iceberg table they are about maintenance liveness. The highest-value signal is the maintenance job's own status, because every automated owner has a documented way of stopping while the table carries on serving: a Glue optimizer auto-suspends after four failures, and S3 Tables snapshot management fails for a whole table if somebody adds a branch or sets an Iceberg retention property. Alarm on job status and on metadata file count per table, and the two failures that produce no error become the two seen first.

The second-order point is about who is allowed to delete. A Hive table on S3 tolerated a bucket-wide lifecycle rule, a shared prefix and a tidy-up script, because nothing but the objects themselves mattered. An Iceberg table's correctness depends on files that look unreferenced and are not, so every one of those habits becomes a way to lose data. The rule that follows is blunt: one table, one prefix, one deleter.

Key Architecture Decisions

Closing Thought

Table formats arrived because the alternative had run out of room. Schema-on-read made a lake cheap to fill and impossible to guarantee, and every mechanism built on top of it — crawler policies, partition inheritance, format-specific access rules — was reconciliation after the fact. Iceberg's answer is not cleverer reconciliation. It is to write the table down.

What that buys is a real transaction boundary, and what it costs is that a table is now a living data structure with a maintenance schedule. The AWS surfaces around it are good and getting better — managed compaction, S3 Tables' automated maintenance, and the request rates that come with a purpose-built bucket type are genuine improvements over rolling it yourself. But every one of them defaults to five days of history, and every one of them has a documented way of stopping without saying so.

Which turns the interesting question away from the format and onto ownership. The format has made the table honest. Whether the estate stays healthy depends on something much duller: knowing, per table, which component is allowed to delete a file, and having an alarm on whether it still is.

Official AWS Reference

• Query Apache Iceberg tables, Amazon Athena
• Create Iceberg tables, Amazon Athena
• Update Iceberg table data, Amazon Athena
• Perform time travel and version travel queries, Amazon Athena
• Optimize Iceberg tables, Amazon Athena
• OPTIMIZE, Amazon Athena
• VACUUM, Amazon Athena
• Service Quotas, Amazon Athena
• Compaction optimization, AWS Glue
• Deleting orphan files, AWS Glue
• Table optimizers, considerations and limitations, AWS Glue
• Maintenance for tables, Amazon S3 Tables
• S3 Tables Regions, endpoints and service quotas, Amazon S3
• Best practices design patterns: optimizing Amazon S3 performance
• Amazon S3 Tables