Cover image
Back-end
11 minute read

SSH Logging and Session Management Using AWS SSM

A comprehensive tutorial on how to log SSH activity—minus sensitive input, like passwords—occurring in Linux AWS EC2 instances to either CloudWatch Logs or S3 buckets.

Setting up custom tools or scripts to keep an SSH log on Linux can be tedious and error-prone.

Engineers can use rootsh, screen, or another utility to log user activity but if the user permissions are not set correctly, a skilled user can erase audit logs to cover their tracks. Another option would be to set up logging at the kernel level, but the expertise needed for that isn’t so common.

Thankfully, there’s a way to log user activity without writing even a single Linux command! We’ll need these services:

Let’s see how to set up each of them.

EC2 and IAM

Launching an EC2 instance is normally fairly easy, but there’s one key task that must be done during launch: We need to attach an IAM role to our instance, otherwise we won’t be able to achieve the expected results detailed at the end of this article.

The IAM role we associate with our EC2 instance must have the built-in AmazonSSMManagedInstanceCore policy, plus this policy (attached as inline or customer-managed):

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "logs:CreateLogStream",
                "logs:DescribeLogStreams",
                "logs:DescribeLogGroups",
                "logs:PutLogEvents"
            ],
            "Effect": "Allow",
            "Resource": "*"
        }
    ]
}

Aside from the IAM role, we generally use this EC2 instance configuration:

  • The OS is Amazon Linux 2, because by default it comes with AWS Systems Manager Agent (SSM Agent) installed. (So do Ubuntu distributions, which are also an option.)
  • The Instance Type is t3.micro (but any type will do).
  • The default Security Group that AWS assigns to the VPC will work, as we don’t need an SSH port open for this exercise.

We can start to set up KMS without waiting for the EC2 instance to boot up.

KMS

We need a KMS key if we want all session logs delivered to CloudWatch to be stored encrypted. Let’s head over to the KMS service and create a key.

A screenshot of AWS with breadcrumbs "KMS," "Customer managed keys," and "Create key," currently at Step 1: "Configure key." The "Key type" can be set to "Symmetric" (selected) or "Asymmetric." Under "Advanced options," the setting "Key material origin" can be "KMS" (selected), "External," or "Custom key store (CloudHSM)."
Step 1: Choosing a symmetric key type.

A screenshot of AWS with the same breadcrumbs, now at Step 2: "Add labels." An Alias field is set to "cwlg," an optional Description field is left blank, and an optional Tags field has no tags added.
Step 2: Naming our key.

A screenshot of AWS with the same breadcrumbs, now at Step 3: "Define key administrative permissions." The first field, "Key administrators," has a blank search box with 10 rows of results (page 1 of 3) with columns Name, Path, and Type. Only the first row (with respective column values "admin," "/," and "User") has its corresponding checkbox checked. The other field, "Key deletion," has a single option, "Allow key administrators to delete this key," which has its checkbox checked as well.
Step 3 (optional): Assigning an administrator.

We recommend assigning an administrator so that the key can be managed by users other than the AWS account root user, but if others won’t need access, we can skip Step 3.

Here we chose the IAM user “admin” as a key administrator, but we’re free to choose any user or role. We can also opt to disable key deletion permission for the administrator.

A screenshot of AWS with the same breadcrumbs, now at Step 4: "Define key usage permissions." The first field, "This account," has the same search results as in Step 3, but none of them are checked. The other field, "Other AWS accounts," has nothing added to it.
Step 4: Skipping the user assignment page.

We won’t be assigning any users to this key because we want this key to be used only by the CloudWatch Logs service for SSH log encryption and decryption operations.

A screenshot of AWS with the same breadcrumbs, now at Step 5: "Review." The first field, "Key configuration," lists the "Key type" as "Symmetric," the "Key spec" as "SYMMETRIC_DEFAULT," the "Key usage" as "Encrypt and decrypt," and the "Origin" as "AWS_KMS." The next field, "Alias and description," lists one Alias, "cwlg," with no Description. The next field, "Tags," shows no data. The last field, "Key policy," includes a textbox prefilled with a policy in JSON format.
Step 5: Review our configuration and swap the default key policy.

On the Review page, we will need to change the KMS-generated key policy because it doesn’t include permissions for CloudWatch Logs to use the key. We’ll replace it with this policy:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "Enable IAM User Permissions",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::ACCOUNT_ID:root"
            },
            "Action": "kms:*",
            "Resource": "*"
        },
        {
            "Sid": "Allow access for Key Administrators",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::ACCOUNT_ID:user/USERNAME"
            },
            "Action": [
                "kms:Create*",
                "kms:Describe*",
                "kms:Enable*",
                "kms:List*",
                "kms:Put*",
                "kms:Update*",
                "kms:Revoke*",
                "kms:Disable*",
                "kms:Get*",
                "kms:Delete*",
                "kms:TagResource",
                "kms:UntagResource",
                "kms:ScheduleKeyDeletion",
                "kms:CancelKeyDeletion"
            ],
            "Resource": "*"
        },
        {
            "Sid": "Allow access to CloudWatch Log",
            "Effect": "Allow",
            "Principal": {
                "Service": "logs.REGION.amazonaws.com"
            },
            "Action": [
                "kms:Encrypt",
                "kms:Decrypt",
                "kms:ReEncrypt*",
                "kms:GenerateDataKey*",
                "kms:DescribeKey"
            ],
            "Resource": "*",
            "Condition": {
                "ArnLike": {
                    "kms:EncryptionContext:aws:logs:arn": "arn:aws:logs:REGION:ACCOUNT_ID:*"
                }
            }
        }
    ]
}

Make sure to replace all the placeholders:

  • ACCOUNT_ID becomes our AWS account ID.
  • USERNAME becomes the administrator username selected in Step 3. If we opted out of that step, here we remove the second statement block (the one with "Sid": "Allow access for Key Administrators").
  • REGION becomes the regional identifier code we are deploying services to (e.g., us-west-1).

With that, KMS is ready to go.

CloudWatch Log Group

Next we need a CloudWatch Log group (and/or an S3 bucket—see below) where SSM Agent can send SSH session logs. Let’s create it.

A screenshot of AWS with breadcrumbs "CloudWatch," "CloudWatch Logs," "Log groups," and "Create log group." There is no multistep sidebar. The first field, "Log group details," has three sub-fields: "Log group name" (set to "ssm-session-demo"), "Retention setting" (set to "Never expire" from a dropdown), and "KMS key ARN - optional" (set to a truncated value beginning with "arn:aws:kms"). The second field, "Tags," has no tags.
Creating a “CloudWatch Logs” log group.

Note the KMS key ARN field: AWS provides us with the value needed here after the key is created in Step 5 of the KMS section.

(If we didn’t associate the correct policy to our KMS key earlier, allowing the CloudWatch Logs service to use the key, we’ll receive an error related to the KMS key at this point.)

Storing SSH Logs in an S3 Bucket

For storing activity logs with S3 instead of—or in addition to—CloudWatch Logs, we need to add these permissions to our EC2 instance profile as a separate policy (or manually combine them with other permissions we may need associated):

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:PutObject"
            ],
            "Resource": "arn:aws:s3:::BUCKET_NAME/*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetEncryptionConfiguration"
            ],
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": "kms:GenerateDataKey",
            "Resource": "*"
        }
    ]
}

In the above snippet, be sure to replace the BUCKET_NAME placeholder.

Now that we’ve set up somewhere to store the logs—CloudWatch Logs, an S3 bucket, or both—we’re ready to tap into SSH sessions.

AWS Systems Manager Session Manager

This is the final key step, where we configure secure access to our Linux machine for SSH session monitoring and logging. We’ll start at the Session Manager dashboard.

A screenshot of the AWS Session Manager dashboard with sections, "How it works," "Why use Session Manager?," "Getting started," "More resources," and in the upper-right corner, "Start a session." The latter section has an orange "Start Session" button and a white "Configure Preferences" button.
Step 1: Getting started with the dashboard.

On the dashboard, click on the white “Configure Preferences” button in the upper-right “Start a session” box to enable session logging.

On the Preferences page, we will find multiple sections that we could explore, but our focus will be on streaming SSH session logs to CloudWatch or S3 to allow us to quickly see what is happening within our Linux machine.

A screenshot of an AWS section entitled "CloudWatch logging." Its first setting, also called "CloudWatch logging," has a checkbox labeled Enable that is checked. The next setting, "Choose your preferred logging option," has "Stream session logs (Recommended)" selected instead of "Upload session logs." The next setting, "Enforce encryption," has a checkbox labeled "Allow only encrypted CloudWatch log groups" that is checked. The final setting, "CloudWatch log group," has "Choose a log group name from the list" selected instead of "Enter a log group in the text box." Beneath it is a list, "CloudWatch log groups," with "ssm-session-demo" selected. It has corresponding columns "Encryption" (set to "Encrypted"), "Expire events after" (set to "Never expire"), "Metric filters" (set to 0), "Stored bytes" (set to 0), and a "Creation time" timestamp.
Step 2a: Enabling CloudWatch logging.

Just after the “CloudWatch logging” section, there’s an “S3 logging” section where we can select the bucket.

A screenshot of an AWS section entitled "S3 logging." Its first setting, "Send session logs to S3," has a checkbox labeled "Enable" that is checked. Its next setting, "Enforce encryption," has a checkbox labeled "Allow only encrypted S3 buckets" that is checked. Its next setting, "Choose S3 bucket," has "Choose a bucket name from the list" selected instead of "Enter a bucket name in the textbox." Beneath that, "ssm-session-demo" is selected from a drop-down list. The last field, "S3 key prefix - optional," is blank.
Step 2b: Enabling S3 logging.

Once SSH logging is configured, we can SSH into our Linux machine and execute some commands to see if the activity is getting captured or not.

So, let’s start a session. On the same page, we will find a “Sessions” tab where we can start a session. Clicking the “Start session” button will give us a list of EC2 machines on which we can initiate a session:

A screenshot of AWS with breadcrumbs AWS Systems Manager, Session Manager, and Start a session. A "Target instances" search box has no query filled in and only one result, with "Instance name" set to "SSM Demo."
Step 3: Selecting our EC2 instance to start an SSH session with.

If we don’t see our EC2 Linux instance in the list, we should check whether it’s in a running state and has the IAM permissions associated with it that we described earlier.

Handling an SSM Agent “Doesn’t Support Streaming Logs” Error

In case we receive an error saying the SSM Agent version installed on our EC2 machine does not support streaming CloudWatch logs, not to worry. There’s a painless way to fix this.

A screenshot of an AWS error message with white text on a red background. A circled X is next to the message, "The SSM Agent version installed on this instance doesn't support streaming logs to CloudWatch. Either update the SSM Agent to the latest version, or disable the streaming logs option in your preferences."
A potential “outdated SSM agent version” error.

To update the SSM Agent, we need to navigate to Run Command in the left panel of the AWS Systems Manager service.

A screenshot of the AWS Systems Manager Run Command dashboard with sections, "How it works," "Features and Benefits," "Use Cases and Blogposts," "Documentation," and in the upper-right corner, "Manage your instances." That section contains only an orange "Run a Command" button.
Step 1: Starting with the “Run Command” dashboard.

Once we’re there, we can click on the orange “Run a Command” button, leading to a new page where we can fill in some parameters.

A screenshot of AWS with breadcrumbs AWS Systems Manager, Run Command, and Run a command. A search box labeled "Command document" lists 10 rows (page 4 of more than 5), with one named "AWS-UpdateSSMAgent" selected. It has "Amazon" in its "Owner" column and "Windows, Linux" in its "Platform types" column. A field at the bottom, "Document version," has "1 (Default)" selected from a drop-down list.
Step 2: Selecting a command document.

We’ll start by selecting AWS-UpdateSSMAgent from the list.

A screenshot of an AWS section entitled "Targets." The first field, also called "Targets," has options "Specify instance tags," "Choose instances manually" (selected), and "Choose a resource group." At the bottom is an "Instances" search box with no query, with its only result, "SSM Demo," checked. The corresponding instance ID in the row is copied to a box just above "Instances" with an X.
Step 3: Selecting the EC2 instance that has an SSM agent in need of an update.

Once that’s selected, we’ll scroll down until we see the “Targets” section. There we need to select the EC2 instance on which we want to update the SSM Agent, then hit the “Run” button at the end. This will send us to a page where we can monitor the progress of the update.

A screenshot of AWS with breadcrumbs "AWS Systems Manager," "Run Command," and "Command ID" (followed by a GUID). The first section, "Command status," shows success indicators, as does the only row of the next section, "Targets and outputs," which lists the single instance from earlier. There are also two unexpanded sections at the bottom, "Command description" and "Command parameters."
Step 4: Monitoring our update progress.

The agent update shouldn’t take more than five minutes. Once that’s done, we should be able to create a session back in the Session Manager.

At this point, we should have an SSH session started.

A screenshot of AWS with a session ID and instance ID above a terminal. The prompt is "sh-4.2$" and the commands "whoami" and "pwd" have been entered, with the outputs "ssm-user" and "/usr/bin" respectively.
An SSH session using the SSM agent via AWS Systems Manager Session Manager.

After executing a few commands, let’s navigate to the CloudWatch Logs log group (or our S3 bucket, not shown) and confirm that the activity is being recorded.

A screenshot of AWS with breadcrumbs "CloudWatch," "CloudWatch Logs," "Log groups," "ssm-session-demo," and the session ID of the previous step. The only section is a search box, "Log events," with rows that each have a timestamp and a JSON-formatted message. One of them is expanded to reveal its JSON pretty-printed and with a white button to the right labeled "Copy."
SSH log events being recorded in CloudWatch Logs.

We now have a setup, with at-rest encryption enabled, recording every command fired in our Linux machine.

In fact, every command may be more than we want: Any secrets provided or generated during the session will be recorded in CloudWatch or S3 and can be seen by anyone who has the required permissions. To prevent that, we can use stty -echo; read passwd; stty echo; for each secret we need to provide during the session.

A Great SSM/SSH AWS Logging Solution With Minor Caveats

Session Manager is a useful tool to gain remote access to our virtual machines in AWS without having to open port 22. In fact, we can’t generate SSH logs this way if we use port forwarding or a direct SSH connection, as the Session Manager documentation notes.

Nonetheless, combining Session Manager with session logging is a robust solution for controlling and monitoring activity within VMs. Moreover, we’re not charged for using the Session Manager service. We only pay for our EC2 instance and CloudWatch Logs or an S3 bucket for storing logs.

For readers who prefer video tutorials, I’ve covered a very similar procedure a bit more thoroughly on YouTube.


Further Reading on the Toptal Engineering Blog:

Understanding the basics

SSH (Secure Shell) is used for accessing other machines securely even over an unsecured network, as the communication between client and server is encrypted. It’s also used for transferring files to or from a server via the SFTP protocol, which uses SSH under the hood. SSH listens by default on port 22.

There are multiple ways to find active SSH sessions, including the commands w, who, and "ss | grep ssh". With AWS Systems Manager they can be viewed in the Session Manager console or via "aws ssm describe-sessions --state Active" in a terminal.

AWS CloudWatch Logs is a fully managed log aggregation service. You can query, monitor, filter, and trigger condition-based alarms, e.g., when more than 10% of HTTP requests are receiving a 5xx error code. Moreover, CloudWatch Logs Insights presents log data in helpful bar, line, and stacked area charts.

AWS IAM is responsible for authenticating and authorizing requests made to AWS resources. It supports both RBAC and ABAC authorization models. In RBAC, permissions are granted based on job roles. In ABAC, permissions are granted based on an attribute, e.g., assigning permissions to a principal based on tags.

Amazon EC2 (Elastic Compute Cloud) lets developers launch virtual machines in the cloud. Highly scalable in nature, EC2 allows for instances to go from 1vCPU to 20 and vice versa in just a couple of minutes. Instances can use Intel, AMD, or ARM processors and run workloads on Linux, Windows, or macOS.

EC2 instances are virtual machines in the cloud, used to run workloads without needing management of the underlying hardware, yet still providing full control of computational resources. EC2 offers a wide range of processors, OSs, and storage for running anything from a small web app to a huge SAP HANA database.