Moonbeam Security Recommendations
Community Contributions

Moonbeam Security Recommendations

By February 14, 2022No Comments

Moonbeam Collator ADVANCED Security Recommendations
by TrueStaking

Assumptions:
1. You have followed the moonbeam setup guide found here: https://docs.moonbeam.network/node-operators/networks/run-a-node/systemd/
2. You are using Ubuntu 18.0 or newer
3. You are using systemd and running on bare metal server

Note: this guide is about the security of your Linux server. For key management practices see the Moonbeam presentation for collators: www.youtube.com/watch?v=jPWAegmgJBg

 

Linux Server Security Basics

We have to crawl before we walk, right? So don’t skip the basics!

You can find many good lists of generic best practices for Linux System Administrators to follow. You can find short lists such as this Cardano forum entry, or long and very detailed lists with tools for remediation such as the CIS security benchmarks.

 

Essential Basics:

Minimize attack surfaces — Don’t run any services you don’t absolutely NEED on your collator (Seriously, your collator should be a dedicated server devoted solely to the task of collating for Moonbeam — you should literally have only SSHD and the moonbeam service accepting remote connections on your collator)

Avoid weak remote access — run SSHD on a non-standard port. Disable password based authentication and require SSH keys for authentication. Many people think “why do I need to run SSHD on a non-standard port? Because if you don’t, then every internet scanning database out there will have your hostname/IP address and SSH version — and IF there is ever a vulnerability discovered in SSH, then you are on the target list and the scripts are pointing at you. Staying off those lists will avoid the first wave of scripted exploitation and give us time to hear about the exploit and remediate it.
Just edit /etc/ssh/sshd_config and replace “#Port 22” with Port X — where X is any number greater than 1024, not 2222, and not found in /etc/services . To enable key authentication and disable password authentication see: https://www.linuxbabe.com/linux-server/setup-passwordless-ssh-login

Minimize privilege escalation opportunities — Keep tight controls on user and group permissions, and force all administrative activity to use the SUDO mechanism

Control Incoming Network Connections — Use a hostbased firewall (UFW, IPtables, or NFtables) and tightly control inbound connections. Why use a firewall? Because it is easier than monitoring for listening sockets so if we ever accidentally install some software that opens a port, or some malware lands on our box — the listening port will be inaccessible without changing the firewall first.

Address Emerging Vulnerabilities — Keep your server fully patched and updated

Make Exploitation Difficult — Run moonbeam_service from systemd as an un-privileged user, please DO NOT skip this step in the build document!

With the basics completed — on to the good stuff!

 

Advanced Preventive Control: AppArmor

What it is:
AppArmor is a kernel level mechanism to assign rights to a running process and restrict what files/directories the process can read/write/update. Using AppArmor addresses two security concerns:
1) the service could be exploited remotely as it is exposed to external connections
2) the source code could be manipulated

(Of course, we don’t believe either is probable, but the Solarwinds debacle of 2020 and the plethora of monthly patches for remote exploits in commercial and open source software dictates that we take prudent precautions.)

AppArmor is installed and loaded by default in modern Ubuntu. However, we will want to install the optional AppArmor utils package.

 

Step 1: install apparmor utilities

 

sudo apt install apparmor-utils

 

Step 2: create moonbeam profile
First, copy moonbeam-profile -> /etc/apparmor.d/abstractions/moonbeam

 

wget https://github.com/truestaking/moonbeam-security-recommendations/blob/main/moonbeam-profile.txt 
sudo cat moonbeam-profile.txt >/etc/apparmor.d/abstractions/moonbeam

Next, we will create an AppArmor profile for our moonbeam service. Profiles are simply text files stored in /etc/apparmor.d/

If we followed the node build instructions, we should have the executable located in /var/lib/moonbeam-data/moonbeam — so create /etc/apparmor.d/var.lib.moonbeam-data.moonbeam (See how the “/” in the normal path are replaced with “.” in the profile name?)

Insert the following text into /etc/apparmor.d/var.lib/moonbeam-data.moonbeam:

 

#include <tunables/global>

/var/lib/moonriver-data/moonbeam flags=(complain) {
  #include <abstractions/nameservice>
  #include <abstractions/moonbeam>
  
  /proc/*/cgroup r,
  /proc/*/mountinfo r,
  /proc/sys/kernel/random/uuid r,
  /sys/devices/** r,
  /sys/fs/** r,
  owner /proc/*/maps r,
  owner /proc/*/task/** rw,
  owner /var/lib/moonbeam-data/** rwk,
}

NOTE: we specified “complain” mode*, and thus the system won’t block any access but will allow and log all access outside the defined access.

(Adjust the paths as needed for your setup in both the filename and the content of the file.)

 

Step 3 – invoke the profile

 

sudo apparmor_parser -r /etc/apparmor.d/var.lib.moonbeam-data.moonbeam

 

Step 4 – restart the service

 

sudo systemctl restart moonbeam.service

Let it run for 15 minutes.

Now we update the profile if needed with: “sudo aa-logprof ” (“A” for allow, and “S” for save.)

Bam! Now you are running moonbeam under AppArmor — it can only read/write were you have allowed on the system and thus you have seriously reduced any impact in the event the service is compromised.

 

Advanced Detective Control: Monitoring AppArmor with auditd

Using AppArmor highly restricts where the moonbeam_service account can read/write on the system. Highly restricted is good — but doesn’t completely close the door to the bad guys. Assuming the attacker does compromise the moonbeam executable. What do they do next? It is very rare to remain entirely memory resident, almost always they reach out to pull down tools to establish command and control, perform reconnaissance and plan their next move.

Anticipating these tactics, we will:
a. Detect if moonbeam_service tries to read/write outside the allowed profile by monitoring our logs for AppArmor alerts
b. Enable auditd and thus detect if they write and then execute any tools in /var/lib/moonbeam-data

 

Step 1: install auditd

 

sudo apt install auditd

 

Step 2: update audit.rules

 

sudo echo "auditctl -w /var/lib/moonriver-data/ -p x -k NOEXE" >> /etc/audit/rules.d/audit.rules

 

Step 3: edit audit.conf

 

edit /etc/audit/auditd.conf and set the following two variables as below:

 

max_log_file = 4
num_logs = 2

NOTE: This will ensure we only ever have 2 files total, each no more than 4 megabytes and since we only have 1 audit rule engaged — this is is super light and tight with no noticeable system impact

 

Step 4: start auditd

 

sudo systemctl start auditd

check all is well with

 

sudo cat /var/log/audit/audit.log

we should see some startup entries
we can also do “sudo auditctl -l” and we should see our audit rule is engaged

Now restart moonbeam with “sudo systemctl restart moonbeam.service”

 

Step 5: ensure our events are logged

 

sudo cat /var/log/audit/audit.log

this should produce some events
if not, then do “sudo systemctl is-active auditd”. If not active, check “sudo journalctl -f -u auditd” and do a little troubleshooting

 

Step 6: a quick framework for alerts

 

  1. Detecting forbidden filesystem access

 

if sudo cat /var/log/audit/audit.log | grep type=AVC | grep operation=\"open\" | grep comm=\"moonbeam\"; then echo "RED ALERT"; else echo "GREEN"; fi

NOTE: if we had some hits, then
a. improve our profile by running aa-logprof as above with “A” for accept and “S” for save…
b. clear the logs with: “sudo systemctl stop auditd; rm /var/log/audit/audit* ; systemctl start auditd”
c. restart moonbeam with “sudo systemctl restart moonbeam.service”

Rinse, lather, repeat — until the logs remain clear after restarting the service.

 

  1. Detecting unexpected launch of executable file within allowed directory

 

if sudo cat /var/log/audit/audit.log | grep SYSCALL | grep NOEXE ; then echo "RED ALERT"; else echo "GREEN"; fi

We should have a “RED ALERT” but we can see that is from the “comm=moonbeam” — which is always going to be the case because the moonbeam_service has to launch itself, right?

So let’s filter it out by adding and additional filter of grep -v comm=\”moonbeam\”

 

if sudo cat /var/log/audit/audit.log | grep SYSCALL | grep NOEXE | grep -v 'comm="moonbeam"'; then echo "RED ALERT"; else echo "GREEN"; fi

If all went well, there are no red alerts!

Now, all you have to do is put these two commands into a script, run it from a timer service file or crontab — and call your alert function when you get a RED ALERT.

Configuring an alerting mechanism is left to the reader — you could use either email, sms, both — or some other choice. (IF there is enough interest, we’ll consider putting these security alerts into the Moonbeam Collator Community Monitoring). If you are starting from scratch on this, here a few quick pointers to get you started:

Email —

 

  • install Postfix as outbound sender only ( https://www.atlantic.net/vps-hosting/how-to-configure-postfix-as-a-send-only-smtp-server-on-ubuntu-20-04/ )
  • configure Postfix to use your email service (assuming you have a hosted solution such as gmail — see https://www.linode.com/docs/guides/configure-postfix-to-send-mail-using-gmail-and-google-workspace-on-debian-or-ubuntu/ )
  • Success here is when you can run the following command and get an email:
    mail -s “subject line here” -r root@yourcollator.com you@youraddress.com <<< “Collator Security Warning”

SMS —

 

  • Go to textbelt.com, “create your own key” and for $5 you get 200 text — enough to last you a very looooong time…
  • create a warn.sh script containing the following:

#!/bin/bashcurl
-X POST https://textbelt.com/text –data-urlencode phone=’YOUR_PHONE_HERE’ –data-urlencode message=’$1′ -d key=YOUR_API_KEY_HERE

 

  • Save it, and make it executable with chmod +x warn.sh
  • Then you call it with warn.sh “warning message”

We trust the community finds this document useful and that it results in a more secure collator set for moonbeam/moonriver/moonbase. If nothing else, we hope it sparks collaboration and conversation that will produce positive outcomes.

–Daniel@truestaking.com

@perltk (telegram)
Daniel | TrueStaking#7508 (Discord)