Glue 6.0 Is Cheaper, and Its Headline Feature Makes Tables Athena Cannot Read

Executive summary

AWS Glue 6.0 is generally available with lower pricing than previous Glue versions and Apache Iceberg v3 support, described by AWS as the most complete v3 implementation on any fully serverless managed Spark service.

Both halves of that headline need a second look, and the migration guide supplies it.

• Iceberg v3 tables cannot be read by Athena. AWS documents the exact error: Cannot read unsupported version 3. Its own guidance is to use v2 if you need cross-engine compatibility.
• The v3 types do not work with DynamicFrames — Glue's own abstraction — only with Spark DataFrames. Visual ETL does not support them either.
• This is a Spark 4 migration, not a version bump. EMRFS removed, AWS SDK for Java v1 removed, Scala 2.12 to 2.13 with a binary break, and ANSI mode on by default.

The price reduction is real and welcome. It is also the least consequential thing in this release.

What changed

Glue 6.0 is a substantial runtime refresh. The version table from the migration guide is the clearest summary of what a job actually lands on:

Alongside the runtime, Glue 6.0 adds Spark Declarative Pipelines, Spark Connect for interactive sessions, Arrow-native Python UDFs, a customer-managed Python virtual environment via --python-virtual-env, and a real-time mode for stateless streaming.

The Iceberg v3 additions are the VARIANT type with variant shredding, nanosecond-precision timestamps, and the Geometry and Geography geospatial types.

Architecture

There is no topology to redraw for a runtime release — a Glue job lands on a stack of pinned versions, and an upgrade replaces the stack. What makes this one worth reading architecturally is that one component in that stack writes a property into your data which outlives the job that wrote it.

Four layers move together here, and they fail in different places at different times. The language and engine layer breaks at build time, loudly, when a custom JAR will not load. The filesystem and SDK layer — S3A as the only S3 filesystem, AWS SDK for Java v2 only — breaks at job start, when configuration or an import no longer resolves. The semantics layer, ANSI mode on by default, breaks at runtime, on a row that previously overflowed quietly to NULL. All three are contained by the job: they are found by running it.

The fourth is not contained by anything. The table format layer is the only one whose choice is written into storage, because format-version is a property of the table rather than of the job. A job writing v3 leaves behind a table that some other engine now has to be able to read, and Athena cannot. So the boundary that decides this upgrade is not the one between Glue 5.1 and 6.0 — it is the one between what a Glue job can write and what everything downstream can read.

That asymmetry is the reason to take the runtime decision and the format decision separately. Moving the runtime is reversible by setting GlueVersion back. Writing v3 tables is not reversible by changing a job setting, because the artefact left behind is the table.

The trap: Athena cannot read v3

This is the sentence to carry away, and it is in AWS's own known limitations:

Iceberg tables created with 'format-version'='3' cannot be read by Athena SQL (error: Cannot read unsupported version 3). Use Iceberg v2 for cross-engine compatibility with Athena.

The point of an open table format is that more than one engine can read the table. A Glue job writing v3 produces a table that AWS's own serverless query service rejects — and Athena is, for a great many organisations, the thing analysts actually use.

Two further limits narrow it again:

• The new v3 data types work only with Spark DataFrames. They do not work with DynamicFrames, which is Glue's native abstraction and the one most existing Glue code is written against.
• Visual ETL in Glue Studio does not support them. AWS's recommendation there is to migrate those jobs to SageMaker Unified Studio.

So the headline capability is available to Spark DataFrame code, writing tables that a major downstream consumer cannot read. That is not a reason to avoid Glue 6.0 — it is a reason to keep format-version at 2 until every engine in your estate can read 3.

The breaking changes the headline does not mention

Moving to Glue 6.0 is a Spark 4 migration. The guide lists these as breaking:

• EMRFS has been removed. S3A is the only S3 filesystem. Jobs using s3:// paths use S3A automatically, but EMRFS-specific configuration such as fs.s3.consistent.* must be removed.
• AWS SDK for Java v1 has been removed. Only v2 (2.44.6) is present. Anything importing com.amazonaws.services.* must move to software.amazon.awssdk.services.*. Python jobs using boto3 are unaffected.
• Scala 2.12 to 2.13 is a binary break. Custom JARs compiled against 2.12 fail with NoSuchMethodError or ClassNotFoundException and must be recompiled. JavaConversions is gone in favour of CollectionConverters.
• ANSI mode is on by default in Spark 4.1. Integer overflow, invalid casts and out-of-bounds array access now throw instead of returning NULL. spark.sql.ansi.enabled=false restores the old behaviour.
• SQLContext is removed — use SparkSession.
• getResolvedOptions no longer abbreviates arguments (allow_abbrev=False by default). Partial argument names that used to resolve now fail.

The ANSI change deserves particular attention because it is the one that alters results rather than failing the build. A pipeline that has been quietly writing NULL on overflow for years will start throwing, which is better behaviour and a worse Tuesday.

Two features that are not new

Coverage of Iceberg v3 usually leads with deletion vectors and row lineage. Neither arrives here. The migration guide is explicit that both “are already supported by AWS Glue ETL since AWS Glue 5.1” — deletion vectors as merge-on-read using Roaring Bitmaps in Puffin files, and row lineage through first-row-id metadata.

If those were the reason to upgrade, you may already have them.

Business value

The value splits cleanly in two, and only one half is available to everybody.

Stated plainly: the cost and runtime benefits are broadly available, and the headline feature is narrowly available. A team that upgrades for the price and keeps its tables at v2 collects most of the value of this release while taking the least of its risk.

Cost considerations

AWS states 30% lower pricing than previous Glue versions. The public pricing page lists a single ETL and interactive sessions rate of $0.44 per DPU-hour, billed by the second, and does not break that rate out by Glue version — so confirm the effective rate for your Region and version before modelling the saving, rather than assuming it lands automatically.

AWS's own worked example on that page is a 15-minute job on 6 DPUs: 6 × 0.25 × $0.44 = $0.66. That is the shape of the arithmetic to redo with whatever rate applies to you.

The larger cost effect is indirect. Spark 4.1, Arrow 18.3 and Arrow-native Python UDFs change how long a job runs, and Glue bills by the second on DPU-hours — so runtime improvements show up on the bill directly. Measure a representative job on both versions rather than reading the rate card alone.

Security considerations

One limitation is a governance issue rather than a functionality one: fine-grained access control is not supported with VARIANT columns. VARIANT exists to hold semi-structured data, which is exactly the kind of column likely to contain fields you would want to restrict. Adopting VARIANT on a table governed by FGAC means losing that control on those columns.

Iceberg native table encryption keys are also unsupported, and Iceberg multi-argument transforms are not available.

Operational considerations

Upgrading is a job configuration change — GlueVersion set to 6.0 via CreateJob or UpdateJob, and AWS notes no API changes are required. The work is not in the switch; it is in the dependencies.

Two things to check before flipping anything: every --extra-jars entry needs recompiling for Scala 2.13 and Spark 4.1, and --additional-python-modules still works but is now deprecated in favour of --python-virtual-env. Python code needs a pass for 3.13, which removes several long-deprecated standard library modules.

For estates with many jobs, AWS points at Generative AI upgrades for Apache Spark, which handles Glue 2.0 and later.

Tradeoffs

Implementation guidance

1. Leave format-version at 2 unless nothing downstream reads the table with Athena. This is the single decision that matters most in this release.
2. Inventory custom JARs and recompile against Scala 2.13.17 before changing any job's version.
3. Grep for com.amazonaws in Scala and Java jobs. Those imports do not exist in 6.0.
4. Remove EMRFS configuration such as fs.s3.consistent.*, and confirm S3A can resolve the right Region through your VPC endpoint.
5. Run a representative job with ANSI mode on in a non-production account before migrating. This is where silent behaviour changes live.
6. Benchmark one real job on 5.1 and 6.0 and compare DPU-seconds, not just the rate. Billing is per second, so runtime is half the cost story.
7. If you use DynamicFrames or Visual ETL, plan on the v3 types being unavailable to you regardless of runtime version.

Best practices

• Leave format-version at 2 unless nothing downstream uses Athena. It is the one choice in this release that writes itself into storage and cannot be undone by changing a job setting.
• Separate the runtime upgrade from the format decision. Take the engine and the price now; take v3 when the engines that read your tables agree on it.
• Inventory and recompile before flipping GlueVersion. Custom JARs against the new Scala, and a grep for com.amazonaws across Scala and Java jobs.
• Rehearse ANSI mode in a non-production account. It is the only breaking change that appears as a thrown exception on real rows rather than at build or start time.
• Benchmark DPU-seconds, not the rate. Billing is per second, so a shorter run is as much of the saving as a cheaper one — compare one real job on both versions.
• Move from --additional-python-modules to --python-virtual-env. The old flag still works and is deprecated, so migrate while nothing is forcing the issue.
• Do not upgrade for deletion vectors or row lineage. Both have shipped since Glue 5.1; if they are your reason, you may already have them.

Who should adopt, who should wait

Adopt now: teams on Spark DataFrames with few or no custom JARs, who want the runtime and the lower rate and can keep tables at v2. The upgrade is a configuration change for them.

Plan carefully: anyone with compiled Scala dependencies, AWS SDK v1 usage, or SQL that relies on NULL-on-overflow. None of that is hard, but all of it is work that must precede the switch.

Wait: teams whose analysts query Glue-written Iceberg tables through Athena and who want v3 features. The format and the query engine have not met yet, and using v3 today means choosing which of the two matters more.

Key takeaways

• Glue 6.0 is GA in all Regions where Glue operates, with Spark 4.1.1, Scala 2.13.17, Python 3.13 and Iceberg 1.11.0.
• Iceberg v3 tables cannot be read by Athena SQL — Cannot read unsupported version 3. AWS itself recommends v2 for cross-engine compatibility.
• The v3 data types work only with Spark DataFrames, not DynamicFrames, and not in Visual ETL.
• Fine-grained access control does not work with VARIANT columns, which is a governance gap on exactly the column type meant for semi-structured data.
• Breaking changes: EMRFS removed, AWS SDK for Java v1 removed, Scala 2.13 binary break, ANSI mode on by default, SQLContext removed.
• Deletion vectors and row lineage are not new — both have shipped since Glue 5.1.
• ETL is $0.44 per DPU-hour billed by the second on the published rate card, which is not version-differentiated; confirm your effective rate rather than assuming the 30% applies to your bill as printed.
• Because billing is per second, a faster runtime is a price cut too — benchmark rather than reading the rate alone.

Official AWS references

• Migrating AWS Glue for Spark jobs to AWS Glue version 6.0 — version table, breaking changes and known limitations
• AWS Glue 6.0 now available with 30% lower price and full Apache Iceberg v3 support — launch post
• AWS Glue pricing — DPU-hour rate and billing granularity
• AWS Glue 6.0 delivers 30% price reduction and Iceberg v3 support — announcement