By combining AWS Systems Manager (SSM) with AWS Key Management Services (KMS), Amazon CloudWatch, and Amazon OpenSearch Service, you can encrypt and securely store user session logs and explore log data. increase. These tools are easy to integrate and offer powerful analytical capabilities.
This series walks you through configuring AWS SSM Session Manager so that you can securely store session logs in Amazon CloudWatch, generate relevant metrics from the logs, and explore the logs for specific events. . Part 1 creates a Customer Managed Key (CMK) in Amazon KMS and sets the permissions required to use that key. It also explains how to create a secure log group in Amazon CloudWatch to securely send session logs to Amazon CloudWatch. In Part 2, you will query your logs in Amazon CloudWatch, create a custom metric and metric namespace, and finally configure log delivery to Amazon OpenSearch Service for advanced analysis.
Background
Session Manager is a feature of AWS Systems Manager that provides a one-click interactive browser to start Amazon Elastic Compute Cloud (Amazon EC2) instances, on-premises instances, and virtual machines (VMs). You can manage it using a base shell connection or using the AWS Command Line Interface (CLI). AWS SSM provides secure instance management without opening inbound ports, maintaining bastion hosts, or managing SSH keys. Traditionally, system administrators have used underlying operating system features to record SSH session activity locally to a log file and export this file as needed for investigation. AWS SSM Session Manager can automate the process of recording session logs as well as forwarding and storing session logs in Amazon CloudWatch's encrypted log store. Amazon CloudWatch enables you to extract metrics from your collected logs and share the information only with personnel with appropriate permission levels.
Solution Overview
This solution uses an AWS Identity and Access Manager (IAM) role to set the appropriate permission level to access the console logs. AWS KMS is used to create CMKs that encrypt console logs. This solution also sets up a secure log group in Amazon CloudWatch to store Session Manager console logs.
A few important notes about this solution:
Assumptions
The assumptions for this solution are:
- You must already be using AWS SSM. If not, use Quick Setup to create the necessary IAM roles and configure your service.
- EC2 instances or on-premises servers that you access through Session Manager must have SSM Agent installed and properly configured. Many Amazon Machine Images (AMIs) come with this agent pre-installed. See the link for more information on this setting.
- The IAM role used in the instance profile must include permissions provided by the AmazonSSMManagedInstanceCore managed policy, or equivalent.
- Predetermine a name for the new CloudWatch Logs log group. Then make sure the log group name is unique. This blog refers to it as “ssm-secure-consolelog-group”.
Once you've met the prerequisites, let's move on to the next step.
Step 1: Create a KMS Key and Policy
First, you need to create a new CMK within Amazon KMS. To obfuscate sensitive data, such as masking keyboard strokes during password reset operations, session logs should be protected using a CMK. Creating a KMS key will cost your account a small fee. Learn more about Amazon KMS pricing here.
It is a best practice to use KMS encryption for session log data.
From the KMS console, select "Customer Managed Key" and press "Create Key".
Next, specify a "symmetric" key and specify "KMS" for the key material origin. You can also use the Cloud Hardware Security Module (HSM) or import your own key material. If so, you can proceed accordingly. Select "Single Region Key" for Regionality.
Enter a name for the new CMK. Here we use “session-manager-logs”. Any other name is fine. Click Next.
During the CMK creation process, you choose which users or roles are allowed to manage this key. This is a very sensitive action and should be restricted to as small a group as possible. We recommend that you add only the most privileged users in your organization.
You must also define a user for this key. This should also be as restrictive as possible, but should include a group that allows reading session log data and his IAM role to use in the instance profile. For example, if you are using the custom role 'EC2RoleSSMCWSecureLogs' as the IAM role for your EC2 instance, this role must have Key Usage permissions and should be selected in the list.
Select Next, then select Finish to complete the CMK creation. You should see a message that the key was created with the alias "session-manager-logs" (or whatever name you provided).
Next, you must manually edit the key policy to give the CloudWatch Logs service access to this key. Select the alias or key ID hyperlink to view the details of the key you just created. Make a note of the Amazon Resource Name (ARN) of your newly created key. This value is required when creating a log group. In the Key Policies section, select the Switch to Policy View button.
Select the Edit button on the displayed screen to manually edit the key policy.
Paste the following policy statement as a new permission into your policy, replacing , with the appropriate value.
{"Sid": "Allow access for CloudWatch secure log group","Effect": "Allow","Principal": { "Service": "logs..amazonaws.com"},"Action ": [ "kms:Encrypt*", "kms:Decrypt*", "kms:ReEncrypt*", "kms:GenerateDataKey*", "kms:Describe*"],"Resource": "*","Condition" : { "ArnEquals": {"kms:EncryptionContext:aws:logs:arn": "arn:aws:logs:::log-group:" }}}Select Save to save your policy changes. Make a note of the key's ARN. You have now created a customer-managed symmetric key and granted it permissions for the specified action to be used by the log group named 'ssm-secure-consolelog-group'.
Step 2: Create a CloudWatch Logs Log Group
The next step is to create an encrypted log group with the same name you specified in the CMK's policy section.
In the CloudWatch Logs console, select Create log group.
In the Log Group Details section, enter the same name that you specified in the CMK's policy (here, 'ssm-secure-consolelog-group'). Select your preferred retention period and enter the new KMS key ARN that you wrote down earlier. Select the Create button to finish creating the log group.
After you create an encrypted log group, configure your Amazon EC2 instance to send Session Manager console access logs to that log group.
Step 3: Update the EC2 Instance Profile
To allow EC2 instances to send Session Manger logs to CloudWatch, configure EC2 to allow communication with the CloudWatch Logs log group you created. I need to update the IAM role used by the instance. The AmazonSSMManagedInstanceCore managed policy does not include this permission, so add it manually before using Session Manager.
You must set up a role with the AmazonSSMManagedInstanceCore managed policy attached and use it as the IAM role for your EC2 instance. In the following example, the IAM role "EC2RoleSSMCWSecureLogs" is used in EC2.
Click the IAM role hyperlink to open Amazon IAM. The following policy should be added to an existing role ("EC2RoleSSMCWSecureLogs" in this example). This allows the local SSM Agent to use the new logging destination. Again, replace , and .
{ "Version": "2012-10-17", "Statement": [{"Sid": "VisualEditor0","Effect": "Allow","Action": "logs:DescribeLogGroups" ,"Resource": "arn:aws:logs:::log-group:*"},{"Sid": "VisualEditor1","Effect": "Allow","Action": [ "logs:CreateLogStream", "logs:CreateLogGroup", "logs:PutLogEvents"],"Resource": [ "arn:aws:logs::::log-group::log-stream:*", "arn:aws:logs::::log- group:"]} ]}To add this inline policy, select the Add Inline Policy link. The policy editor will open.
In the policy editor, select the JSON tab and copy and paste the policy statement. Here, replace , with the correct value. Make sure there are no errors and select the Review Policy button.
On the Review Policy page, specify a name for the inline policy statement and select the Create Policy button.
The policy is now created as an inline policy within the role.
Step 4: Enable Logging to Amazon CloudWatch Logs
Now that your KMS key, key access policy, CloudWatch log group, and access to your CloudWatch log group on EC2 are properly configured, you can log to CloudWatch. You can enable delivery of session logs to To do so, go to the AWS SSM console and select Settings under Session Manager. You can see that KMS encryption and CloudWatch logging are disabled by default. This must be enabled for the solution to work. Select the Edit button to continue.
First, enable KMS encryption by selecting the checkbox and selecting the KMS key with the name you created earlier from the KMS Key dropdown.
Scroll down to the CloudWatch logging section. Now enable CloudWatch logging and select the previously created log group.
"Save" your changes to complete the process. That's all for setting up the solution. Now, to check if the solution is working, start a session using Session Manager and enter some commands at the session prompt. By opening CloudWatch, selecting Log Groups from the Logs menu, and selecting the log group you created, you can see log files containing information about the session, the commands entered, and the responses the user saw during the session. increase.
Troubleshooting
If no logs are uploaded to CloudWatch Logs, in most cases you can diagnose it from the Amazon SSM Agent logs. Check the EC2 instance (or on-premises server) logs to see if the error is related to key permissions. Below is an example of misconfigured key permission mappings.
grep ERROR /var/log/amazon/ssm/amazon-ssm-agent.log |grep ssm-session-worker | tail 2021-03-29 23:09:29 ERROR [ssm-session-worker ] [me@me.local] [DataBackend] [pluginName=Standard_Stream] error when calling AWS APIs. error details - AccessDeniedException: User: arn:aws:sts::xxxxxxxxxxxx:assumed-role/managed-ec2-ssm-role/ i-024259ed24a2cbaeb is not authorized to perform: logs:DescribeLogGroups on resource: arn:aws:logs:us-east-1::log-group::log-stream:2021-03-29 23:09:29 ERROR [ssm -session-worker] [me@me.local-] [DataBackend] [pluginName=Standard_Stream] Validation failed, err: We couldn't start the session because encryption is not set up on the selected CloudWatch Logs log group. Either encrypt the log group or choose an option to enable logging without encryption.Summary
In Part 1 of this series, we stored AWS SSM Session Manager logs in Amazon CloudWatch. provided an overview of the process of searching for You created an encryption key using Amazon KMS and used that key to encrypt Session Manager logs sent to Amazon CloudWatch. Part 2 shows how to use the CloudWatch console to search your logs and set up metric filters based on filter patterns in your new metric namespace. It also configures sending log data to Amazon OpenSearch Service for further analysis.
About the Author
Rich McDonough
Rich McDonough is a Senior WW CloudOps Specialist Solutions Architect at AWS based in Toronto. He primarily helps customers grow their AWS usage safely and securely, in Cloud Operations, and guides customers in observability practices and service adoption. Prior to joining AWS in 2018, he helped customers migrate to the cloud.
Manoj Subhadevan
Manoj Subhadevan is the Engineering Lead and Software Development Manager for the Amazon Mobile Shopping Foundation. His team is responsible for the release engineering of the Amazon mobile shopping app, using AWS technology to scale. In my free time, I plan to listen to music and travel around the world.
SA Ishibashi was in charge of the translation. Read the original here.