Good security is no longer optional for software developers. It never really was, but today we face the explosion of supply chain attacks. These are sped along, of course, by coding agents that produce a flood of code contributions to open source projects, often too much code to reasonably review, and make it much easier for bad actors to write malicious code. So I don’t just have to keep someone from getting my system password. I have to prepare for the possibility that some malicious code will get onto my system through the back door–through one of the thousands of open source packages I use and update every day. That means I have to prepare for the possibility that some process is on my system and running as me. How in the world do I protect secrets like authentication credentials against that?

Getting my secrets somewhere safe

For my ssh and pgp keys, I opted to store them on a physical YubiKey and require my physical touch for every use of the credential. I can’t sign a git commit or connect over ssh to our servers without reaching over and touching the key each time. That means even if a malicious process gets onto my machine, running as me, it still can’t use those credentials without me seeing my YubiKey flash and touching it. That gives me a fighting chance to notice that some malicious code is trying to call out with my data, and stop the request.

But what about the tokens used by shell commands like gh or aws? I built a small utility called simplesec that stores my secrets with PGP encryption and retrieves them on demand. My PGP agent is configured to require a physical touch of my YubiKey each time a secret is retrieved. That’s a good first step, but I have to somehow feed those retrieved secrets into each gh or aws command. How?

This question sent me down a rabbit hole looking at various methods of injecting secrets and the security holes each of them leaves. In some ways it was a depressing search. It’s often impossible to get a secret value into a command without exposing it somewhere. But there are definitely better and worseoptions.

What about shell substitution?

My first naive thought was “Just use shell substitution! Easy!” So I could do something like this:

curl -H "Authorization: Bearer $(simplesec get github/token)" https://api.github.com/user

or

somecli --password "$(simplesec get db/pass)"

But I quickly realized that this still exposed the actual secret value in a lot of places.

This post is my attempt to get a full overview of how malware might view those retrieved secrets. Some of these routes expose a secret to any user on my machine (other UID), other routes expose it only to processes running under my UID (same UID), and still others can only be accessed by the root user. Some of these points of exposure can be closed or mitigated, and there are alternative ways to get secrets into commands. But to begin with I want to take shell substitution as a case study and ask, “If I were to inject secrets this way, what are all the ways a malicious process could see that sensitive value?”

NOTE: I’m focusing here on Linux and MacOS systems because…I have no idea what goes on under Windows. It’s not part of my world.

The holes

Shell history (other UID)

If a process with my UID inspects the bash history or zsh history they can see the command that retrieved the secret, not the secret itself. But this still makes it easier for malware to discover our secret store mechanism and try to exploit it.

Process argv (same UID)

If any user running under my UID runs a standard ps command, the resolved value of the substituted secret is visible in the default output as long as the command is running. On Linux this can also be done by polling /proc/<pid>/cmdline and on MacOS by using proc_pidinfo. This makes it trivially easy for malware running as me to harvest the secret. It used to be easy for other users to see the same resolved values, but on modern Linux and MacOS (properly configured) other users generally can’t.

Shell memory during expansion (same UID)

This exposure is extremely brief (often milliseconds) but the resolved secret does sit in the memory heap of the shell running the command very briefly. On Linux, process_vm_readv could watch the shell process from outside and theoretically catch the resolved secret during the brief window the shell is resolving it. This is much more difficult on MacOS, so there a bad actor would have to use software running inside the same shell process–like a compromised shell plugin or rc file (see the section on wrapped processes below).

Command process memory while running (same UID)

The more realistic scenario on Linux is for malware to inspect the running command process itself using the same process_vm_readv command and search through the heap for the resolved secret. ptrace or a debugger like gdb or lldb can also be attached to a running process and inspect its memory, but this is harder to do without detection since both require stopping the process. MacOS, on the other hand, makes this kind of inspection difficult, often impossible without triggering permission prompts.

xtrace shell output to stderr (same UID / CI )

If a command is run in a shell with debug mode (the -x flag) turned on, this prints shell xtrace output to stderr–including the resolved value of substituted secrets. That output can also be directed to a file that could be read by anyone with access. But it requires that the command be run in such a shell. For malware to turn this mode on, it would have to be already running inside the shell process–again likely via a compromised shell plugin or rc file.

The more common vector for this vulnerability is CI jobs. It’s not uncommon for a CI task to run a shell command with -x to facilitate seeing exactly what is being run. A compromised CI job (like a Github action) might turn this on deliberately. In that case the resolved secret value would often be included in the CI task logs, and so be exposed in plain text to anyone with permission to read them. This can be prevented on Github by careful masking, as we’ll see below.

Crash reports/core dumps (root / same UID)

Resolved substitution values can also appear in files written after an application crashes abnormally (e.g., due to a segfault) or in some cases when it is killed. On MacOS, Crash Reporter writes a .ips diagnostic report under something like ~/Library/Logs/DiagnosticReports/, readable by any same-UID process. On modern Linux a full core dump is often written to a file owned by root somewhere like /var/lib/systemd/coredump/, readable via coredumpctl or journalctl. Generally the full core dump is only accessible to the root user and requires sudo, although it is possible for sections of the dumped memory to appear in journal output readable by same-UID processes. Still, at this point modern Linux (using systemd-coredump) is generally more secure.

Audit/EDR logs (root)

On systems with Audit or EDR logging configured, this can also contain the resolved value of substituted secrets. Audit logs can record every process that runs on a system, including the arguments passed to them (if they log execve). This is done by auditd or kernel hooks on Linux, MacOS uses OpenBSM. EDR security software like CrowdStrike, SentinelOne, etc. records the same process information and usually sends it to a central server. But in both cases, these logs are usually only viewable by someone with root permission, audit group members (on Linux), and any EDR/SIEM agent process along with the people withaccess to its dashboard. A process that simply runs with my UID can’t view them.

CI logs (anyone)

The picture is somewhat different for CI logging. We already looked at the risk of catching secrets when processes are run with -x and logged. Those are recorded because there’s a log kept of stdout and stderr for each step in, e.g., a Github workflow. For public Github repositories (like our Knowledge Commons ones), these logs are readable by anyone on the internet. This is a major source of confidential data leaks in general.

One solution is just to ensure that your workflows don’t run commands with the
-x flag. Another solution is to avoid command substitution for Github workflows and instead use Github’s own secrets management (secrets.*), which automatically masks secret values in the step logs. (This is actually the preferred way to do it.) If it is necessary to use some kind of command substitution (e.g., to fetch a secret at runtime from an external source) these secrets can also be masked using ::add-mask:: in the workflow as long as that mask is invoked before the secret appears in the log output.

This is distinct from Github’s org/enterprise audit log. That keeps a record of who runs what, but it does not record any execution arguments (argv) or command results (stderr, stdout). If a workflow runs on a self-hosted Github runner, there’s the possibility that the runner host is also doing audit/EDR logging. But this is not an issue on Github-hosted runners.

Of course, if some malware is running in your running CI process (e.g., it was pulled in as part of a dependency package) then all the same means of access are open to it that exist on a local system or server.

Wrapped executables on PATH

Maybe the most insidious kind of infiltration will wrap an executable app with its own runner code. When I think I’m calling the gh CLI tool, I’m actually calling a malware application that has hijacked its name. That fake gh application then runs the real gh for me and I don’t even realize it’s there. This is done by modifying the system’s PATH, the list of folders the operating system checks to find executable app files. When I call gh, the shell reads PATH from left to right and checks each folder for a file with the name gh. The first matching file it finds is the one that the shell executes. So if a bad actor modifies PATH to add a folder at the beginning, it can place its own files in that folder and give them the names of whatever apps they want to wrap.

There are three kinds of targets here:

(a) A secrets management tool like simplesec can be wrapped, in which case the runner code can intercept the tool’s output like this:

exec /real/simplesec "$@" | tee /tmp/out

It will capture every secret the tool retrieves when the substitution calls it.

(b) The app that’s receiving the secret can be wrapped. In this case when we resolve the secret as an argument, we hand it directly to the malicious wrapper code before it ever reaches the actual command we’re trying to run. The weakness of this approach is that an attacker has to decide which processes to wrap with a runner. So it either has to know its target ahead of time (maybe targeting common commands like curl, gh, aws, etc.) or it needs to somehow discover likely targets on my machine.

(c) The shell application itself can be wrapped (bash, zsh, etc.), in which case the runner can attach listeners like debuggers, log the xtrace output, etc.

The PATH is most often modified by rewriting shell startup files like .bashrc or .zshrc. This can also be done by a malicious shell plugin installed by me. Since PATH is just an environment variable, malware running under my UID can also simply inject their own env variable when it executes a command:

PATH="/tmp/evil:$PATH" ./my-build-script.sh

In fact, this technique can be used when the malware modifies any process that can accept an injected env variable: git hooks, npm postinstall scripts, CI steps, dockerfiles, etc. If malware under my UID can modify the execution of any process to inject an env variable, it can wrap that executable. All of this requires, again, that the malware is targeting a specific application to wrap, and that it’s able to write its own malicious file in a new folder.

The sobering reality of command substitution

The hard truth is that there are a lot of ways secrets can be observed if I inject them using shell substitution. Now, there are some ways to mitigate these risks. And most of them are only available to malware running under my UID. The problem is, of course, that if I bring in a compromised package in my supply chain, it will likely be executed as me. Still, under regular OS configuration, any user on my system can read resolved substitute values in argv by running ps and observing the process list. Or by reading process files in /proc/<pid>/cmdline on Linux.

I should emphasize this does not make secrets injection useless!! It is still meaningfully harder to discover the resolved values than it is to read them from plain text sitting around in an .env file. And in the next post in this series I’ll collect some mitigation strategies. I’ll also cover some alternative methods of injecting secrets that give us a bit more protection. The quest for system security continues!

Basic Anatomy of a Network Request

I recently had to debug a tangled networking mess, and in the debugging process I had to revisit how networks work. I wanted to let a colleague piggy-back on my VPN connection, using a machine on my tailscale private network as an “exit node,” funnelling all of his requests out through my machine’s connection to the external VPN. It’s a clever setup. (Much thanks due to Martin Eve for the suggestion.) But I took a winding path through routing tables, tcpdumps, and firewall rules to get it working. To keep it all straight, I had to picture the network connections visually, and these diagrams were the result. They offer a visual overview of how a network request works on a *nix system, whether that’s my MacBook or a Linux server on AWS.

Application initiates a connection

We start with an application on one machine that wants to talk to some other machine. The local app can’t send any data until a connection has been established, a channel set up to carry the messages between two machines. So my application calls a function like connect() (in whatever language I’m using) to initiate a connection. My app hands that function three pieces of information: the hostname of the machine we want to talk to, the port it should connect to on that machine, and the protocol (set of conventions) to use. (For
most internet traffic, we say it should use the TCP protocol.)

Note that connect() might be called something else, but most languages have some function that abstracts away the process of setting up an open network connection. In Python it might be socket.create_connection(), in Javascript it might be net.Socket.connect(), in Go net.Dial(). Note, too, that these are lower-level functions that most developers likely don’t use directly. In Python we’re most likely using a method like requests.get() or in Javascript fetch(). These higher-level functions completely hide the creation of the network connection and give the impression that we can just start sending data whenever we feel like it! But under the hood, all of thosee methods start by orchestrating a new open connection. So how?

Connect creates a socket

Our theoretical connect() method is handed the hostname of the machine we want to address, the port on that system, and the protocol. The operating system then uses this to create a socket. The socket is my end of the communication channel we’re setting up to the other machine. It’s like a tin can that as kids we might have connected, via a piece of string, to a matching tin can on the other end.

In order the create this socket, we first need to know the IP address of the machine on the other end. We (usually) don’t start a connection with a raw IP address like 198.51.100.1. We start with a hostname like “google.com” (or even “localhost” if we’re talking to ourselves!). So we ask our operating system’s DNS resolver to find a DNS server that will translate the hostname (“google.com”) into an IPv4 address like 198.51.100.1 or an IPv6 address like 2001:db8:abcd::42.

With that target IP in hand, the connect() method then asks the OS kernel to assign an ad hoc (“ephemeral”) port number (like 54321) that will represent our end of the connection.

The kernel can then create a socket object that knows the IP address and port number to use at both ends of the connection. (Remember, we told connect() what port to connect to on the other end.)

Socket used to initiate TCP/IP connection

From there, the socket can begin the actual communication process with the destination machine. If we’ve created a socket to use the TCP protocol, it assembles the initial packet of information that we’ll send to begin the “handshake” that establishes a machine-to-machine connection. The packet is at its core a blob of binary information. That’s the payload. It will be decoded at the other end as a JSON string or CSV file or image segment or whatever. That binary payload is then wrapped in a series of headers, each of which has conventional fields and flags. If our packet is built using the TCP convention, it will have a TCP header that tells the receiving socket what it needs to know to pass the payload along to the application–how big it is, its checksum (to make sure it wasn’t corrupted), whether it’s part of a larger multi-packet payload and where it falls in the sequence, what receiving port to use, etc. When we’re establishing the initial connection, though, we send an empty payload and include the “SYN” flag. That’s the first step of the TCP “handshake” the two machines will perform to confirm that both ends of the communication channel are set up.

Around that TCP header, we wrap the payload in another IP header. This is like the outer envelope of an old fashioned letter with the street address written on it. To traverse any network, the internet included, a message needs to hop from machine to machine. In some cases this will just be from one machine on my local area network directly to another computer. But usually some of the machines in between include my local router, a router on the other end, and a chain of gateway machines in between. Each machine in this chain looks at the destination IP address in the outer IP header to know what direction it should pass the message on for the next hop, until it finally arrives at the machine whose address is that IP.

Routing table decides on an interface

We’re getting ahead of ourselves, though. We now have an assembled packet with TCP and IP headers. But before we can get any message out, we have to know what device on our local machine to use to launch the packet on its first hop along its path. We need to identify an “interface”, like our machine’s ethernet or wifi connection. Our operating system holds routing tables with rules for which interfaces to use for different ranges of IP address. So the OS looks up the correct interface and sends the packet out along its way.

The reply establishes the usable connection

Although we’ve sent a packet out, remember that we still haven’t sent any actual payload. Our packet’s TCP header just included the “SYN” header. That’s like an empty envelope with just a slip of paper inside saying “Are you still at this address? Do letters reach you here?” The receiving machine sends back another emtpy packet, back through the same interface, socket, and port that we used before. But this time with the TCP header “SYN-ACK”. “Yup. I’m here. I can receive letters at this address.” And finally our machine sends a follow up packet with the header “ACK”. “Great. Keep an eye on your mailbox because I’m about to send a letter.” This exchange of empty packets–“SYN”, “SYN-ACK”, “ACK”–establishes a live connection. The connect() method is complete and it returns a confirmation to the application: “You’ve got a connection ready to go.”

Sending and receiving application data

Only now can the application call another method like send() or write() (again, using whatever language) to send data along the established channel to its destination and receive a reply back. The TCP handshake does not have to be repeated for the life of the connection. The application can re-use the socket to send and receive more messages through the same port until it’s finished or the connection is broken from the other end. We’re talking!

To be continued…

Of course, we haven’t said anything yet about security. It feels like 80% of our time as software engineers these days is taken up just with keeping jerks (and worse) out of our data. How do we open the connections we want without opening the door to bad actors? The next installment will cover firewalls, TLS handshakes–the things that keep our connections safe, and can give us migraines!