The index begins when you create it, and ends before your logs do

Business Challenge

#112 ended on the cost of asking a question of your logs: a Logs Insights query is billed on data scanned, under the “DataScanned-Bytes” usage type against “StartQuery”. Field indexes are the documented lever on that number — “This reduces the scan volume of your queries that use field indexes, making it possible to return results faster” — and the feature does what it says.

It is also the most time-dependent primitive in CloudWatch Logs, and that is not obvious from the configuration. An index policy looks like a property of a log group, in the way that retention is. It is not. It is a property of the events that arrived while it existed.

1The index starts at creation and expires in 30 days; retention does neither

Two sentences, next to each other in the documentation, decide most of how this feature behaves:

“CloudWatch Logs indexes only the log events ingested after an index policy is created. It doesn't index log events ingested before the policy was created.”

“After you create a field index, each matching log event remains indexed for 30 days from the log event's ingestion time.”

Put those together with a log group whose retention is a year, and the log group has three zones: events too old to have been indexed, events indexed and still within their thirty days, and events that were indexed and have since fallen out. One field, one policy, three behaviours, decided entirely by when each event arrived.

The operational consequence is the one worth internalising: adding an index during an incident does nothing for that incident. The events you are investigating are already ingested. The index you just created begins with the next one.

Fix

Treat index policies as capacity planning, not as an incident tool — they go in whenever the log group is created, alongside retention, and the fields chosen are “fields that you often need to query for”. AWS also names high-cardinality fields as good candidates, because “a query using these field indexes will complete faster because it limits the log events that are being matched to the target value”. If you are querying beyond thirty days, assume you are paying the unindexed price and size the query accordingly.

2filterIndex changes the answer, not just the cost

This is the finding that matters most, and it is easy to read past because filterIndex is introduced as an optimisation. Its actual contract:

“Use filterIndex to return indexed data only, by forcing a query to scan only log groups that are indexed on a field that you specify in the query.”

And then, describing a query across five log groups where four are indexed and one is not: “For each indexed log group, it attempts to scan only log events that have the indexed field, and it also returns only results from after the field index was created.”

Two exclusions, both silent. The fifth log group contributes nothing. Events from before the index existed contribute nothing. The query succeeds, returns rows, and does not tell you that a log group was skipped or that a time range was out of reach. Replace filterIndex with filter and the same question gets a different answer: “the query will attempt to scan not only the log events that contain the value in the indexed log groups, but will also scan the fifth log group that isn't indexed, and it will scan every log event in that fifth log group.”

So the two commands are not a fast path and a slow path to one result. They answer different questions: what does the index know about this value, and what is in these log groups. For a dashboard that is a cost decision. For a security investigation asking whether an address appears anywhere, it is the difference between a complete answer and a cheap one.

Fix

Pick the command from the question, and write the reason next to the saved query. filterIndex for high-volume exploration across “as many as 10,000 log groups”, where speed is the point and a miss is tolerable. filter for any question whose value depends on completeness — incident forensics, audit evidence, "did this ever happen". If you need both, run filterIndex first to find the shape and filter to confirm it.

3Four ways the index is quietly not used, and none of them errors

An index that does not apply is not a failure. The query runs, returns the right rows, and scans everything — so the only symptom is the bill.

Case. “Indexed fields are case-sensitive. For example, a field index of RequestId won't match a log event containing requestId.” One capital letter between an indexed query and a full scan.

Format. “Fields indexes are supported only for the structured log formats of JSON and service logs.” An application writing plain-text lines gets nothing from this feature at all, no matter what the policy says.

Query shape. Indexes are “for efficient equality-based searches”, and the documented forms are requestId = {{value}} or requestId IN [{{value}}, {{value}}, ...]. A substring match or a range is not an equality search.

Value novelty. The subtlest one: the skip applies to events “that CloudWatch Logs has detected a value for that field in the past”, and within indexed log groups the query “skips log events that have an IPaddress field but not observed 198.51.100.0 as the value for that field”. The index is value-aware, which is what makes it fast — and is worth knowing when reasoning about what a filterIndex query could have missed.

Fix

Copy the field name out of a real log event rather than typing it, and index the field as your application actually writes it. Confirm the log group is JSON before writing a policy for it. And keep the @ rule in mind for custom fields: “To create an index of a custom field in your log groups that starts with @, you must specify the field with an extra @ at the beginning of the field name” — so @userId is indexed as @@userId, while generated fields like @logStream take no extra @.

Architecture

Diagram: A CloudWatch Logs field index enables efficient equality-based searches, and when a query uses one it attempts to skip processing log events known not to include the indexed field, which reduces the scan volume that Logs Insights queries are billed on under the DataScanned-Bytes usage type. The index is a property of ingestion rather than of a log group: CloudWatch Logs indexes only the log events ingested after an index policy is created and does not index events ingested before, and each matching event remains indexed for 30 days from its ingestion time, while log group retention is independent and may be much longer. An index created during an incident therefore does nothing for that incident. The filterIndex command returns indexed data only, forcing a query to scan only log groups indexed on the specified field, and for each indexed log group it attempts to scan only events that have the indexed field and also returns only results from after the field index was created, so two exclusions are applied silently. Using filter instead scans the un-indexed log groups too, every event in them, which costs more and is complete. Four conditions make an index silently inapplicable with no error: indexed fields are case-sensitive so RequestId does not match requestId, field indexes are supported only for the structured log formats of JSON and service logs, the index serves equality searches of the form field equals value or field IN a list, and the skip applies to values CloudWatch Logs has detected for that field in the past. Quotas are 20 fields per policy, 100 characters per field name, and as many as 40 account-level index policies of which 20 may use log group name prefix selection criteria and 20 may use data source based selection criteria, with no two prefix policies allowed to overlap and only one policy per data source name and type combination. A log-group-level policy overrides account-level policies that match the log group as a whole, while account-level policies matching at the log event level for a data source name and type apply in addition. A filterIndex query can scan as many as 10,000 log groups chosen by as many as five log group name prefixes. Default field indexes exist for all Standard log class log groups and for several AWS data source types, and they do not count towards the quota. A field index policy created in a monitoring account is not used for log groups in linked source accounts, because a policy applies only in the account where it is created.

Three zones in one log group

When the event was ingested	Indexed?	What a filterIndex query does with it
Before the policy existed	No — “It doesn't index log events ingested before the policy was created”	Excluded from results
After the policy, within 30 days of ingestion	Yes	Scanned, with non-matching events skipped
After the policy, more than 30 days ago	No longer — the index lasts “30 days from the log event's ingestion time”	Outside what the index can accelerate

Retention is not in that table, and that is the point. A log group retaining a year of data has at most thirty days of it indexed at any moment. The index is a moving window over the newest arrivals, not an attribute of the group.

filterIndex against filter, on the same question

	`filterIndex`	`filter`
Log groups with no index on the field	Skipped entirely	Scanned, every event
Events from before the index existed	Not returned	Returned
Scan volume	Reduced, sometimes enormously	Whatever the time range holds
Completeness	Only what the index covers	Everything in range
Tells you what it skipped	No	Nothing to skip

The last row is the architectural problem. #112's hazard was a query that returned a number lower than the truth because the data had not arrived. This is the same shape with rows instead of a count: a result set smaller than the truth, with no state, no warning and no flag.

Scope and precedence, which decide what is actually in force

Policies exist at two levels and the interaction is specific: “If you create log-group level index policy, that log group does not use account-level policies that match at the log group level.” So a log-group policy is a replacement, not an addition — add one field locally and you lose the twenty the account policy was providing.

With one exception that is additive: “Account-level policies which match at the log event level (such as, for a given data source name and type combination) will apply in addition to policies which match the log group as a whole.”

And the account-level namespace has structural rules: “no two of them can use the same or overlapping log group name prefixes”, only one policy per data source name and type, and the catch-all forecloses the others — “If you have an account-level index policy that has no name prefixes and applies to all log groups, then no other account-level index policy with log group name prefix filters can be created.”

The quotas worth knowing before you design the policy set

Limit	Value
Fields per index policy	“As many as 20 fields can be included in the policy”
Field name length	“as many as 100 characters”
Account-level policies	“as many as 40” — 20 by log group name prefix, 20 by data source
Log groups a `filterIndex` query can reach	“as many as 10,000 log groups”, via “as many as five log group name prefixes”
Default field indexes	Free — “not counted towards your field index quota”

The defaults are more generous than most teams realise. Every Standard-class log group already has indexes on @logStream, @aws.region, @aws.account, traceId, severityText and others; CloudTrail data and management logs come with eventSource, eventName, awsRegion, errorCode, readOnly, requestId and more; VPC flow logs arrive with action, logStatus, region, flowDirection and type. Before writing a policy, check whether the field you want is already free.

The cross-account gap

“If you create a field index policy in a monitoring account, that policy is not used for log groups in linked source accounts. A field index policy applies only in the account where it is created.”

Which inverts the usual cross-account observability intuition. The monitoring account is where you run the wide queries, and it is the one place a policy written there has no effect on the data you are querying. Indexes have to be created in each source account, by whoever owns it — a distribution problem rather than a configuration one.

Why This Architecture Holds Up

Forward-only indexing is the honest implementation

Indexing historical data would mean reading all of it — the exact cost the index exists to avoid, paid at policy-creation time instead of query time, on data you may never query. Forward-only means the index costs nothing to create and nothing to backfill. The right response is not to wish it were retroactive but to create policies early, which costs nothing either.

A separate command for the lossy behaviour is the correct choice

filterIndex could have been an optimisation inside filter, silently applying the index when available. That would have made every query's completeness depend on configuration nobody querying could see. Making it a distinct keyword means the lossy version is the one you have to ask for by name — and the documentation states the exclusions plainly. The default is the complete answer.

Free defaults aimed at the logs people actually query

Default indexes on CloudTrail's eventName and errorCode, on VPC flow logs' action and logStatus, on WAF's action — these are the fields in nearly every investigation of those log types, indexed without anyone configuring anything and without consuming quota. That is a well-chosen default set rather than a token one.

Key Architecture Decisions

Decision	Choose this	Because
When to create index policies	With the log group, next to retention	Only events “ingested after an index policy is created” are indexed — creating one later cannot reach back
Queries that must be complete	`filter`, never `filterIndex`	`filterIndex` skips un-indexed log groups and “returns only results from after the field index was created”
Wide exploration across an estate	`filterIndex` with prefixes	It reaches “as many as 10,000 log groups” while scanning a fraction of them
Queries older than 30 days	Budget them as unindexed	The index lasts “30 days from the log event's ingestion time”, however long retention runs
Log-group-level policies	Use sparingly, and restate the account fields	A log-group policy replaces account-level policies matching the log group as a whole
Account policy layout	Decide prefix-based or catch-all up front	A catch-all policy forecloses every prefix-based one, and prefixes may not overlap
Which fields to index	Check the defaults first	Defaults cover the usual CloudTrail, VPC flow log and WAF fields and “are not counted towards your field index quota”
Cross-account observability	Create policies in each source account	A monitoring-account policy “is not used for log groups in linked source accounts”

The audit worth running

Find every saved query and dashboard using filterIndex. For each one, decide whether an incomplete answer is acceptable. Any that back an audit, a security investigation or a compliance statement should be using filter, and the cost difference is the price of the guarantee.

Compare your index policies' creation dates against the questions you ask. A policy created last week cannot serve a query over last month, and nothing in the query result says so.

Check case, character for character, against a real log event. RequestId against requestId is a silent full scan that looks exactly like a working index.

List the log groups you query most and confirm they are JSON. Field indexes are “supported only for the structured log formats of JSON and service logs”, so a policy on a plain-text log group is configuration that cannot do anything.

In a monitoring account, verify where your policies live. The ones that matter for cross-account queries are the ones in the source accounts.

Closing Thought

This run of posts keeps arriving at the same structural problem from new directions. #110: an alarm state that means I cannot tell, firing nothing. #111: a signal produced by the watcher, so its silence has two causes. #112: a query that ran before the logs landed and returned a number lower than the truth.

Field indexes complete the set, and filterIndex is the sharpest version yet, because the incompleteness is something you asked for. The command does exactly what it documents: returns indexed data only. It skips the log group with no policy, and the week before the policy existed, and it returns rows that look like an answer. Nothing is broken. The question quietly changed from where does this value appear to where does the index know this value appears, and those differ by precisely the configuration and history you were not thinking about when you typed the query.

So the rule for this primitive is about which question you are asking. Indexes are a cost control and a speed control, and they are excellent at both. They are not a completeness guarantee, and the keyword that uses them is the one that gives completeness up. Create the policies early, because creation time is the only time that matters; then choose filter or filterIndex by what the answer is for, and write the reason down where the next person will read it.

Next in this series

Operational Excellence — CloudWatch Logs log classes: what the Infrequent Access class gives up, which features stop working when a log group is not in the Standard class, and why the choice is made at log group creation and is hard to reverse.

AWS Architecture Series #113 — The index begins when you create it, and ends before your logs do