Generate AWS cost reports automatically using the Cost-Explorer API
Unlocking Cost Savings and Financial Insights through FinOps
Introduction
Maintaining cost control is crucial for running an AWS business. To streamline this process, we have implemented a solution that automatically generates cost reports and sends them to your designated email addresses. This solution leverages the Cost Explorer API, Lambda functions, EventBridge scheduled events and the Simple Email Service (SES).
By utilizing the Cost Explorer API, one can programmatically retrieve cost and usage data from your AWS account. Lambda functions are employed to automate the process, allowing for regular and scheduled cost report generation. These functions fetch the required cost data using the Cost Explorer API and generate comprehensive reports based on predefined parameters.
To deliver the generated reports, we utilize the Simple Email Service (SES) provided by AWS. With SES, we can send these reports directly to your specified email addresses, ensuring that you and your team have easy access to the latest cost information. This enables you to monitor your AWS billing conveniently without manually navigating the billing console.
The cost report is generated daily, providing you with up-to-date information regarding your AWS expenses. Additionally, on Fridays, a weekly report is generated, offering a comprehensive overview of the entire week’s costs.
Here is a visual representation of how the email will appear when it is sent to your designated email addresses.
By simply reviewing the cost report, we can identify opportunities for immediate cost reduction.
Architecture Diagram
The architecture diagram for the solution is presented below.
Procedure
Step-1: SES verified Identity
To enable email sending through AWS SES, we need to add and verify an email identity within the SES console. This verified identity will serve as the sender for the emails we generate using the solution. You can refer to this document to create a verified identity – https://docs.aws.amazon.com/ses/latest/dg/creating-identities.html
Step-2: Lambda function
The provided Lambda code is designed to be triggered by an EventBridge rule. It includes logic to determine the current day and performs specific actions accordingly. On all days it generates a daily cost report based on the last 24 hours of usage data. Additionally, if the day is Friday, it also generates a weekly cost report aggregating data from the past 7 days, starting from the current date. This ensures that both daily and weekly cost reports are generated on Fridays, providing a comprehensive overview of costs for that particular week.
Make sure you add the necessary permission for the Lambda function.
Also, add the datetime Python library and Pandas as layers for the Lambda function.
You can make use of AWSSDKPandas-Python39 which is provided by AWS
For, the datetime library you need to download it locally and zip it and upload it as a layer. The folder structure is as follows –
Datetime.zip/layer/python/lib/python3.9/site-packages/*
The python code is added below.
import json
import boto3
import datetime
from botocore.exceptions import ClientError
import pandas as pd
def lambda_handler(event, context):
billing_client = boto3.client('ce')
# getting dates (yyyy-MM-dd) and converting to string
today = datetime.date.today()
yesterday = today - datetime.timedelta(days = 1)
str_today = str(today)
str_yesterday = str(yesterday)
# get total cost for the previous day
response_total = billing_client.get_cost_and_usage(
TimePeriod={
'Start': str_esterday,
'End': str_today,
#'Start': "2023-05-16",
# 'End': "2023-05-17"
},
Granularity='DAILY',
Metrics=[ 'UnblendedCost',]
)
total_cost = response_total["ResultsByTime"][0]['Total']['UnblendedCost']['Amount']
print(total_cost)
total_cost=float(total_cost)
total_cost=round(total_cost, 3)
total_cost = '$' + str(total_cost)
# print the total cost
print('Total cost for yesterday: ' + total_cost)
# get detailed billing for individual resources
response_detail = billing_client.get_cost_and_usage(
TimePeriod={
'Start': str_yesterday,
'End': str_today,
},
Granularity='DAILY',
Metrics=['UnblendedCost'],
GroupBy=[
{
'Type': 'DIMENSION',
'Key': 'SERVICE'
},
{
'Type': 'DIMENSION',
'Key': 'USAGE_TYPE'
}
]
)
resources = {'Service':[],'Usage Type':[],'Cost':[]}
for result in response_detail['ResultsByTime'][0]['Groups']:
group_key = result['Keys']
service = group_key[0]
usage_type = group_key[1]
cost = result['Metrics']['UnblendedCost']['Amount']
cost=float(cost)
cost=round(cost, 3)
if cost > 0:
cost = '$' + str(cost)
resources['Service'].append(service)
resources['Usage Type'].append(usage_type)
resources['Cost'].append(cost)
df = pd.DataFrame(resources)
html_table = df.to_html(index=False)
print(resources)
message = 'Cost of AWS training account for yesterday was'
html = """
<html>
<head>
<style>
body {{
font-family: Arial, sans-serif;
color: white;
background-color: black;
}}
h2 {{
color: white;
font-size: 25px;
text-align: center;
}}
h1 {{
color: #333333;
font-size: 40px;
text-align: center;
background-color: yellow;
}}
p {{
color: white;
font-size: 30px;
line-height: 1.5;
margin-bottom: 20px;
text-align: center;
}}
p1 {{
font-size: 10px;
text-align: center;
margin-left: auto;
margin-right: auto;
}}
</style>
</head>
<body>
<p> Training Account report for the day {} </p>
<h2> {} </h2>
<h1> <strong> <em> {} </em></strong> </h1>
<p1>{}</p1>
</body>
</html>
""".format(str_yesterday,message,total_cost,html_table)
ses_client = boto3.client('ses', region_name='us-east-1')
message = {
'Subject': {'Data': 'AWS training account cost report'},
'Body': {'Html': {'Data': html}}
}
response = ses_client.send_email(
Source="<SES_verified_identity>",
Destination={'ToAddresses': [ "email-1","email-2"]},
Message=message
)
if today.weekday() == 4:
print('week')
week = today - datetime.timedelta(days = 7)
str_week = str(week)
response_total = billing_client.get_cost_and_usage(
TimePeriod={
'Start': str_week,
'End': str_today },
Granularity='MONTHLY',
Metrics=[ 'UnblendedCost',]
)
print(response_total)
length=len(response_total["ResultsByTime"])
print(length)
if (length==2):
total_cost_1 = response_total["ResultsByTime"][0]['Total']['UnblendedCost']['Amount']
total_cost_2 = response_total["ResultsByTime"][1]['Total']['UnblendedCost']['Amount']
total_cost_1=float(total_cost_1)
total_cost_2=float(total_cost_2)
total_cost = total_cost_1+total_cost_2
total_cost=round(total_cost, 3)
total_cost = '$' + str(total_cost)
# print the total cost
print('Total cost for the week: ' + total_cost)
# get detailed billing for individual resources
response_detail = billing_client.get_cost_and_usage(
TimePeriod={
'Start': str_week,
'End': str_today
},
Granularity='MONTHLY',
Metrics=['UnblendedCost'],
GroupBy=[
{
'Type': 'DIMENSION',
'Key': 'SERVICE'
},
{
'Type': 'DIMENSION',
'Key': 'USAGE_TYPE'
}
]
)
resources = {'Service':[],'Usage Type':[],'Cost':[]}
resources_1 = {'Service':[],'Usage Type':[],'Cost':[]}
for result in response_detail['ResultsByTime'][0]['Groups']:
group_key = result['Keys']
service = group_key[0]
usage_type = group_key[1]
cost = result['Metrics']['UnblendedCost']['Amount']
cost=float(cost)
cost=round(cost, 3)
if cost > 0:
cost = '$' + str(cost)
resources['Service'].append(service)
resources['Usage Type'].append(usage_type)
resources['Cost'].append(cost)
for result in response_detail['ResultsByTime'][1]['Groups']:
group_key = result['Keys']
service = group_key[0]
usage_type = group_key[1]
cost = result['Metrics']['UnblendedCost']['Amount']
cost=float(cost)
cost=round(cost, 3)
if cost > 0:
cost = '$' + str(cost)
resources_1['Service'].append(service)
resources_1['Usage Type'].append(usage_type)
resources_1['Cost'].append(cost)
for key, value in resources_1.items():
if key in resources:
resources[key] += value
else:
resources[key] = value
else:
total_cost = response_total["ResultsByTime"][0]['Total']['UnblendedCost']['Amount']
total_cost=float(total_cost)
total_cost=round(total_cost, 3)
total_cost = '$' + str(total_cost)
# print the total cost
print('Total cost for the week: ' + total_cost)
# get detailed billing for individual resources
response_detail = billing_client.get_cost_and_usage(
TimePeriod={
'Start': str_week,
'End': str_today
},
Granularity='MONTHLY',
Metrics=['UnblendedCost'],
GroupBy=[
{
'Type': 'DIMENSION',
'Key': 'SERVICE'
},
{
'Type': 'DIMENSION',
'Key': 'USAGE_TYPE'
}
]
)
resources = {'Service':[],'Usage Type':[],'Cost':[]}
for result in response_detail['ResultsByTime'][0]['Groups']:
group_key = result['Keys']
service = group_key[0]
usage_type = group_key[1]
cost = result['Metrics']['UnblendedCost']['Amount']
cost=float(cost)
cost=round(cost, 3)
if cost > 0:
cost = '$' + str(cost)
resources['Service'].append(service)
resources['Usage Type'].append(usage_type)
resources['Cost'].append(cost)
print(type(resources))
df = pd.DataFrame(resources)
html_table = df.to_html(index=False)
print(resources)
message = 'Cost of AWS training account for the was'
html = """
<html>
<head>
<style>
body {{
font-family: Arial, sans-serif;
color: white;
background-color: black;
}}
h2 {{
color: white;
font-size: 25px;
text-align: center;
}}
h1 {{
color: #333333;
font-size: 40px;
text-align: center;
background-color: yellow;
}}
p {{
color: white;
font-size: 30px;
line-height: 1.5;
margin-bottom: 20px;
text-align: center;
}}
p1 {{
font-size: 10px;
text-align: center;
margin-left: auto;
margin-right: auto;
}}
</style>
</head>
<body>
<p> Training Account report for the week {} and {} </p>
<h2> {} </h2>
<h1> <strong> <em> {} </em></strong> </h1>
<p1>{}</p1>
</body>
</html>
""".format(str_week,str_today,message,total_cost,html_table)
ses_client = boto3.client('ses', region_name='us-east-1')
message = {
'Subject': {'Data': 'AWS training account cost report'},
'Body': {'Html': {'Data': html}}
}
response = ses_client.send_email(
Source='<SES-verified-identity>',
Destination={'ToAddresses': [ 'email-1','email-2']},
Message=message
)
print(response)
Step-3: EventBridge rule
To schedule the execution of the solution, we will create an EventBridge event rule. In the rule configuration, we will select the rule type as “schedule” and set a cron expression to specify the desired execution time. In this case, the cron expression used is “030 5 ? *”, which corresponds to 11:00 AM IST (Indian Standard Time). Additionally, we will set the Lambda function as the target for this event rule, ensuring that it gets triggered at the specified time for generating the cost report.
That’s It!
Congratulations on completing the setup! You will receive cost report emails automatically, ensuring you stay informed about your AWS expenses. This convenient and regular monitoring allows you to promptly identify any cost spikes and take swift action to address them. You can easily optimize your resource usage by eliminating unnecessary resources, thereby reducing expenses effectively. With this level of visibility and control, you can make well-informed decisions to manage your AWS budget efficiently and achieve significant cost savings.
Authors
For further information or any inquiries, please feel free to reach out to us via:
~ Linkedin – sahith palika | LinkedIn , Jaya Sree Gundasu | LinkedIn
~ Email ID – sahithpalika25@gmail.com, gjayasree1919@gmail.com