Home› Blog› Week 1: From Ticket to EC2 in 6 Minutes: Enterpris…
AWS Weekly Lab AWS Terraform

From ticket to EC2 in six minutes

A developer needs a VM, opens a ticket, and waits for an engineer to run the same handful of API calls as last week. This replaces the engineer with a catalogue item, and the instance arrives tagged, governed and reachable without a key.

AWS Platform Engineering Lab · Week 1 of 52

Every large enterprise has this problem: a developer needs a VM. They open a ticket. It goes to a queue. Someone reads it three days later, asks clarifying questions, manually clicks through the AWS console, and emails back a hostname. The developer has moved on.

This week I built the automated version of that story. A developer fills out a ServiceNow catalog form, clicks Submit, and 6 minutes later a production-grade EC2 instance is running behind an ALB — provisioned by Terraform, tracked in state, tagged with the ticket number, and monitored in CloudWatch. No human in the loop.


Lab Overview

Someone needs an EC2 instance. What stands between them and having one?

The problem. Usually a ticket, a queue and an engineer who runs the same handful of API calls they ran last week. The work takes minutes; the waiting takes days, and the engineer is the bottleneck by design rather than by accident.

The objective. A ServiceNow catalogue item that produces a real, tagged, governed EC2 instance with no one in the middle, and a ticket that closes itself with the instance details.

The outcome. Ticket to running instance in about six minutes. One thing does not work: the ServiceNow developer instance will not accept the write-back, which is a limitation of the free tier rather than of the pipeline, and the post says so.

Cloud Services and Concepts

ServicePurpose hereWhat it is
Amazon EC2The thing being provisionedVirtual machines you rent by the second. You choose the size and the operating system, and you are responsible for what runs on them.
AWS Step FunctionsRuns provisioning as declared stepsRuns a workflow as an explicit state machine, so each step, its retry behaviour and its failure path are declared rather than buried in code.
AWS LambdaReceives the request and calls the AWS APIsRuns a piece of code on demand with no server to manage. You are billed per request and per millisecond it runs.
Amazon API GatewayThe endpoint ServiceNow posts toA managed front door that accepts HTTP requests and forwards them to a backend.
AWS Systems ManagerReaches the instance without SSHManages instances through an agent already on them, so there is no inbound port to open and no key to distribute.
TerraformDeclares all 74 resourcesInfrastructure as code: you describe the resources you want and Terraform works out what to create, change or destroy to get there.

The idea the whole series starts from is that self-service is not a UI problem. The catalogue item is the easy part; what makes it safe is that the instance arrives already tagged, already governed and already reachable without a key, because the pipeline had no way to produce one that was not.

Architecture and Workflow

Architecture

aws AWS Cloud ServiceNow catalog request 1 Region: us-east-1 API Gateway signed POST 2 webhook receiver validates 3 Step Functions 4 states 4 EC2 instance Terraform-built The workflow pauses on waitForTaskToken while Terraform runs. GitHub Actions calls back with the result, and only then does the ticket close -- so the ticket reflects the build, not the request. GitHub Actions runs terraform apply calls back with the token

What Gets Deployed (74 Resources)

LayerResources
NetworkingVPC, 2 public subnets, 2 private subnets, IGW, NAT Gateway, Route Tables, VPC Flow Logs, VPC Endpoints (SSM, SSMMessages, EC2Messages, S3, CloudWatch Logs)
ComputeLaunch Template, Auto Scaling Group (min 1 / max 4), EC2 in private subnet
Load BalancingALB (public), Target Group, HTTP Listener
Security3 Security Groups, IAM Role + Instance Profile, OIDC Provider
Automation3 Lambda Functions, Step Functions State Machine, API Gateway REST API
ObservabilityCloudWatch Dashboard, 4 Metric Alarms, Log Groups, SNS Topic + Email
Configuration5 SSM Parameters (ServiceNow creds, GitHub token, webhook secret)

Prerequisites

RequirementWhyHow to check
A ServiceNow instanceThe catalogue item is the front door. A free developer instance works for everything here except the write-backThe catalogue item appears in the portal
A VPC with subnets the instance can launch intoProvisioning assumes somewhere to put itaws ec2 describe-subnets returns them
An instance profile allowing Systems ManagerWithout it the instance launches and is then unreachable, because this build opens no SSH port at allThe instance appears in Fleet Manager
Terraform, and credentials that can create 74 resourcesThe whole stack is declared; nothing is clickedterraform plan completes

Implementation

Step 1 — Setting Up the ServiceNow Catalog Item

The first piece is the user-facing form. In ServiceNow Catalog Builder, I created a new catalog item called "AWS EC2 Instance Request" under Technical Catalog → Infrastructure.

Four questions drive the entire provisioning workflow — everything Terraform needs comes from these fields:

⚠️ Critical: The Name field on each question (not the Label) must match what the Business Rule sends. A mismatch silently falls through to default values — your t3.micro request becomes t3.medium and you'll spend an hour debugging.

Step 2 — Submitting the First Request

After publishing the catalog item under Technical Catalog → Infrastructure, it appeared in the portal. First submission looked like this:

The ServiceNow catalogue item, submitted
The ServiceNow catalogue item, submitted: the only thing a requester touches

How to Implement This Yourself

Everything below references the GitHub repo: katta698 / week-01-enterprise-ec2-provisioning. Fork it, follow these steps, and you'll have the full pipeline running in your own AWS account.

Prerequisites

  • AWS account with admin access
  • GitHub account + fork of the project repo
  • ServiceNow Personal Developer Instance — free at developer.servicenow.com
  • Terraform 1.10+ installed locally
  • AWS CLI configured (aws configure)

Phase 1 — AWS Bootstrap (One-time)

# Create state bucket (choose a unique name)
aws s3api create-bucket \
  --bucket YOUR-terraform-state-bucket \
  --region us-east-1
aws s3api put-bucket-versioning \
  --bucket YOUR-terraform-state-bucket \
  --versioning-configuration Status=Enabled
aws s3api put-bucket-encryption \
  --bucket YOUR-terraform-state-bucket \
  --server-side-encryption-configuration \
  '{"Rules":[{"ApplyServerSideEncryptionByDefault":{"SSEAlgorithm":"AES256"}}]}'

Phase 2 — Configure Variables

Edit terraform/environments/dev/terraform.tfvars (gitignored — never commit):

project              = "selfservice-ec2"
environment          = "dev"
aws_region           = "us-east-1"
artifact_bucket_name = "YOUR-terraform-state-bucket"
tf_state_bucket      = "YOUR-terraform-state-bucket"
github_org           = "YOUR-github-username"
github_repo          = "week-01-enterprise-ec2-provisioning"
create_github_oidc_provider = true
alert_emails         = ["your@email.com"]
servicenow_instance_url = "https://devXXXXXX.service-now.com"
servicenow_username     = "admin"
servicenow_password     = "YOUR-SN-PASSWORD"
github_token            = "ghp_xxxxxxxxxxxx"  # needs 'repo' scope
webhook_secret          = "your-random-secret"

Phase 3 — Deploy Infrastructure

cd terraform/environments/dev
terraform init \
  -backend-config="bucket=YOUR-terraform-state-bucket" \
  -backend-config="region=us-east-1"
terraform plan   # preview 49 resources
terraform apply  # ~6 minutes
# Save this output — needed for ServiceNow
terraform output api_gateway_url

Phase 4 — Set GitHub Actions Secrets

Repo → Settings → Secrets and variables → Actions → New repository secret:

Secret NameValue
AWS_ROLE_ARNFrom terraform output github_actions_role_arn
TF_STATE_BUCKETYOUR-terraform-state-bucket
AWS_REGIONus-east-1

Phase 5 — Configure ServiceNow

5a. REST Message (outbound endpoint): System Web Services → Outbound → REST Messages → New. Set Endpoint to your api_gateway_url. Add POST method using the script in servicenow/business_rule_script.js.

5b. Catalog Item: Service Catalog → Catalog Builder → New item named "AWS EC2 Instance Request" under Technical Catalog → Infrastructure. Add 4 questions with these exact Name values:

LabelName (must match exactly)Type
Instance Typeinstance_typeDropdown: t3.micro, t3.small, t3.medium, t3.large
EnvironmentenvironmentDropdown: dev, staging, prod
Desired Capacitydesired_capacitySingle line text
Cost Centercost_centerSingle line text

5c. Business Rule: System Definition → Business Rules → New. Table: sc_req_item. When: After. ✅ Insert only — uncheck Update. Paste script from servicenow/business_rule_script.js.

⚠️ Only check Insert, not Update. If Update is checked, the rule fires on every RITM state change → 5+ parallel GitHub Actions runs per request.

Phase 6 — End-to-End Test

  1. ServiceNow → Service Catalog → search "AWS EC2 Instance Request"
  2. Fill in: Instance Type = t3.micro, Environment = dev, Capacity = 1
  3. Submit → watch GitHub Actions start within 30 seconds
  4. Step Functions execution runs ~6 minutes through all 4 states
  5. EC2 Console → new instance selfservice-ec2-dev-app appears ✅
ServiceNow submissions
The request as ServiceNow records it, before anything reaches AWS
GitHub Actions Terraform Apply log, succeeded
The apply that built the stack, run by GitHub Actions: 49 resources changed, no errors
Step Functions execution succeeded
The Step Functions execution: each provisioning step declared, and each one green
EC2 console showing two running self-service instances
The EC2 console showing the self-service instance in place

Phase 7 — Teardown

terraform destroy  # destroys all the resources created
# S3 state bucket and OIDC provider are preserved — reusable across all weekly projects
terraform destroy output
Teardown: the stack destroyed, with nothing left running

All code available on GitHub. Questions? Drop a comment below.

Validation and Evidence

This is the first post in the series and its evidence is thinner than later weeks. The rows below are what the post actually recorded; where nothing was captured, the table says so rather than guessing.

TestExpectedActualStatus
A request can be submittedThe catalogue item accepts a request and raises a ticketSubmitted from the ServiceNow portalPass
The request reaches AWSThe workflow starts and completesStep Functions execution completedPass
An instance is createdA running, tagged EC2 instanceThe instance is present and running in the consolePass
End-to-end timeMinutes rather than daysAbout six minutes from submission to a running instancePass
The ticket is updatedThe requester is told, in the ticket, what they gotFails. The ServiceNow developer instance refuses the write-back; the pipeline makes the call and the instance rejects itFail — free-tier limitation
TeardownEverything removedDestroy completedPass
No SSH path existsAccess only through Systems ManagerNot recorded at the time. No inbound SSH rule was created, but no access attempt was made to prove itNot tested

Security and Cost

Security Patterns Baked In

No SSH keys anywhere. EC2 instances use SSM Session Manager (AmazonSSMManagedInstanceCore). Connect with aws ssm start-session --target i-xxx. No key pair, no port 22, no bastion host.

IMDSv2 enforced. Launch template sets http_tokens = "required". Blocks SSRF attacks from reaching the metadata endpoint.

Private subnets. EC2 lives in private subnets, unreachable from the internet. Only accessible through the ALB.

OIDC for GitHub Actions. No AWS_ACCESS_KEY_ID in GitHub Secrets. GitHub exchanges a JWT for temporary STS credentials scoped to your specific repo.

Secrets in SSM Parameter Store. ServiceNow credentials and GitHub token stored as SecureString (KMS-encrypted). Never in env vars, never in code, never committed to git.


Cost Breakdown

ResourceMonthly Cost (est.)
t3.micro EC2 (1 instance, always on)~$8.50
NAT Gateway~$32 + data transfer charges ← biggest cost
ALB~$18
Lambda / Step Functions / API GWFree tier
CloudWatch~$3
Total~$65/month
💡 Cost tip: For a dev lab you use occasionally, run terraform destroy between sessions. The state stays in S3 and you can rebuild everything in 6 minutes. You'll pay cents instead of $65/month.

Troubleshooting and Lessons

This is where a post usually jumps to "and then it worked". Four things had to be fixed first, and three of them were a name that did not match another name.

Problem. ServiceNow fired the webhook and API Gateway returned 502.
Root cause. Terraform created the function with STATE_MACHINE_ARN and the handler read STEP_FUNCTION_ARN. The KeyError crashed the function, and a crashed function is a 502 at the gateway.
Resolution. Align the name in the handler with the one Terraform sets.
Lesson. An environment variable is an interface between two files that never see each other. 502 is what that mismatch looks like from outside.

Problem. Every function failed with Runtime.HandlerNotFound.
Root cause. Terraform declared the handler as handler.lambda_handler; all three files defined def handler.
Resolution. Renamed the functions to lambda_handler.
Lesson. The same mismatch as the first bug in a different place. The runtime cannot tell you a function is missing until it is already running.

Problem. The execution failed at CheckValidation: the path $.validation.Payload.valid was not found.
Root cause. The validation step reused the status updater, which returns no valid field, so the Choice state was reading a field that had never existed.
Resolution. Replaced the task with a Pass state for the lab.
Lesson. Reusing a function because it is the right shape is not the same as it returning the right thing.

Problem. The deployment trigger got 404 from the GitHub API.
Root cause. The token was scoped to public_repo and the repository is private, so it was invisible to that token and the failure presented as a missing repository rather than a refused one.
Resolution. A token with repo scope. On Git Bash, set MSYS_NO_PATHCONV=1 when writing the parameter, or the path is rewritten to C:/selfservice-ec2/... before the CLI sees it.
Lesson. A not-found from an API you are authenticated to is worth reading as a scope problem before a missing resource.

Cleanup and Key Takeaways

The waitForTaskToken pattern in Step Functions is underused. Most people either poll (wasteful) or fire-and-forget (lose visibility). The callback pattern gives you synchronous visibility into a long-running async process with zero idle cost.

version = "$Latest" in ASG launch templates is a silent footgun. Terraform doesn't see LT version changes as an ASG change. Always use aws_launch_template.app.latest_version to trigger instance refresh automatically.

IMDSv2 + private subnets + SSM is the modern EC2 security baseline. If any deployment in your org still has a key pair and port 22 open, that's technical debt worth addressing.

Design every external call as best-effort. Your infrastructure pipeline should never be blocked by a downstream ITSM system. Graceful degradation is a feature, not a workaround.


Comments

How was your experience?
Your feedback helps improve this site.
PoorExcellent