Endlessh: SSH Honeypot Analysis
This lab we will use Kali Linux, tcpdump, netcat, NMap, Wireshark, and Hydra. We will be analyzing traffic being sent to a server running endlessh, a honeypot service to thwart or delay automated scans and (possibly) bruteforce SSH attacks.
Purpose:
1) Assess what endlessh is successful at doing
2) Confirming what are its limitation in thwarting attacks
3) What an adversary would face
4) What we could implement in substitute or in addition to further dissuade threats to our network.
Skeeto's Github repo of Endlessh
Part 1: Set up our victim server
Part 2: Set up Honeypot service
Part 3: Attacking and Analysis
Part 4: Closing Notes
Set up our victim server
1) Cloned our working RHEL baseline VM template for testing
2) Edit IP configuration, and hostname using nmtui
3) Edit sshd config file to host OpenSSH on port 2222/tcp
4) Add port to firewall to allow access from other machines
5) Reload firewall
6) Restart SSH service
If you get any errors restarting sshd!
sestatus && getenforce
b) Install semanage if it is missing
sudo dnf install -y policycoreutils-python-utilsc) Check if SELinux is only allowing binding to TCP 22.
sudo semanage port -l | grep ssh_port_t
d) Change SELinux to allow SSHd to bind to port 2222
e) Restart ssh again, you should receive no errors
f) Check ss for new listening ssh port
7) Connect to machine via new TCP port.
ssh user@IPaddress -p 2222
Note: After installing the honeypot service I fell into my own trap multiple times by not specifying the new port. This is important to note that your trap may catch authorized users or IPs and if you are setting auditing around your honey pot (which you should) it is important to create filters to discern from adversaries and your own users. A simple filter using a whitelist of IPs would separate events with source IP addresses containing internal IPs, your own external IP, and authorized external IPs.
Set up Honeypot service
1) Install git
sudo dnf install git
2) Clone endlesssh
git clone https://github.com/skeeto/endlessh.git
3) Install make and gcc
sudo dnf install make gcc
4) Install endlessh using make command
sudo make install
a) If glibc is older than 2.17 dependant on your version of RHEL or RHEL like system:
make LDLIBS=-lrt
5) Couple of ways to check if endlessh is installed
man endlessh
endlessh -V
NOTE: You are not yet running endlessh as root, when attempting to run with sudo privileges you will get errors or failure if you are running this in a restricted OS which RHEL and RHEL like system likely are. Instead of permanently adding the default location of where endlessh is installed (/usr/local/bin) to our path we are going to directly call it from its location using the sudo command. This is to abide by security best practices and avoid allowing sudo access to the directory due to privilege escalation mitigations set by our OS.
6) Check if endless is actually working. I created a tmux split pane to run endlessh and also check ss at the same time.
7) Finally install tcpdump so that we can monitor and even capture traffic being sent to this machine. We will see how endlessh responds to SSH clients, and adversaries attempting to scan and attack our default SSH port.
sudo dnf install tcpdump
Attacking and Analysis
Attacker IP address:192.168.0.240
Honeypot IP Address: 192.168.0.250
Lets create three tmux window, one to run endlessh and another to monitor and/or capture TCP/IP traffic being sent or delivered to our attacker's machine., and the third to control our Kali Linux instance which we will use to deploy attacks.
Pane 1: SSH terminal of Attacker's machine
Pane 2: SSH terminal of Honeypot capturing network data
Pane 3: SSH terminal of Honeypot running honeypot service
Lets first try a simple SSH client without specifying our actual SSH port. This will attempt to connect to our default SSH port that is running endlessh. We are using a username that does not exist on the system but it does not matter if the client is using a real or non-existent username. 
We can see our ssh client blinking, or not doing anything after entering the command. The endlessh should be sending endless streams of strings as its service banner making the SSH client wait to fully receive before continuing. We stop the attempt to connect via SSH and end the packet capture and the running of endlessh to take a closer look at to what occurred.
Lets take a look at our PCAP capture in Wireshark. Lets break down what happened.
1) The initial TCP handshake confirming they can talk to each other. We see the initial SYN packet sent from our client machine, to SYN/ACK response from our server, then finally the ACK reply from the client confirming and establishing the TCP communication.
2) Our client sends its service banner to the server stating what it is using to talk to it to establish application "language". Our server acknowledges our client message then the fun begins
3) This is where we see our honeypot in action. Immediately after the client service banner is acknowledged, it is time for our service to begin its shenanigans. The client waits for the server to return its banner and our server responds with random strings. The biggest question I had were, "Why is the client allowing the server to continue to send it random strings and not realize it is being sent a fake version banner?"
With no easy answer my research online led me to the official RFC documentation detailed in RFC 4253.
RFC 4253, "The Secure Shell (SSH) Transport Layer Protocol", Page 4
Rule 4.2 Protocol Version Exchange states,
"When the connection has been established, both sides MUST send an identification string. This identification string MUST be SSH-protoversion-softwareversion SP comments CR LF ...",
"... Key exchange will begin immediately after sending this identifier.".
This means developers building SSH clients or scanners are using this specific rule to build their software to identify when banner version occurs and ends, and the developer of Endlessh is weaponizing this standards to defeat or at least delay scanners. It was only after finding RFC 4253 document that I also discovered that the developer states this in their own blog, here. Sometimes it is best to just ask or find the answer directly from the source, work smarter not harder.
Back to our lab, lets use Wireshark's "Follow TCP stream" to look at only the payload being communicated from our client and servers. Here we can see each TCP reponse's payload in this stream.
.
From our attacker's side we perform a simple nmap scan targeting the default SSH port, Unfortunatley our endlessh service did not
recognize any activity on. This means a simple nmap scan does not perform a TCP handshake (or at least a full handshake) and likely means it only just
performs a ping scan of the socket address and did not attempt to complete a handshake to retrieve a banner. If we want to see this captured we can use the -sV argument to instruct nmap to perform the handshake to retrieve version banner. Although I did perform this it caused erratic behavior. I believe nmap has either a short timeout by default since it was opening new ports or sockets and completing new handshakes establishing different communications with our honeypot even when instructed to only try a set number of times. This can be a whole article by itself and likely the answer is there is a default timeout on nmap that is shorter than our own honeypot, and likely to combat this we can set a shorter delay on our honeypot to avoid the reset of our connections. 
Lets try capturing the banner using netcat, next! This will definitely complete a handshake and even return in our terminal the TCP payload which would contain the version string. Using netcat we can see the randomized strings the honeypot is returning.
Finally lets see if Hydra does anything to combat this. Typically in this scenario an attacker has performed a ping scan and was informed that there is something responding on that port and has not run into capturing its service banner yet. Here the attacker likely has a username that is used in the organization and will run a brute force attack using a dictionary of passwords. We can see that endlessh does perform the handshake and then timeouts not too long after. This should raise flags to an attacker that something is not quite right. One would likely think that they are being deceived that the port is actually being used, or that their IP might be filtered after a few attempts to brute force. I would think that hydra would report that it was at least able to try a few passwords before showing the timeout error. We can try this in the future by implementing fail2ban to see how an attacker would be allowed to send a few password brute forcing attempts then be banned.
Finally our last question, "What we could implement in substitute or in addition to further dissuade threats to our network?". This tool is great at stopping or delaying threats that require the TCP handshake. Someone could build a more realistic SSH honeypot by implementing key exchange and further disillusioning scanning or attacking software to believe it is successfully scanning a real SSH server or attempting to brute force a legitimate service. The next step would liekly be storing legitimate responses to return to an adversary to make them believe a specific vulnerable version of an SSH server is running on our machine. Another interesting idea is to create a honeypot that does this with multiple services like LDAP, SMTP, Telnet, HTTP, and more and having it randomly go offline for random services. THis can lead an attacker to believe to be overwhelming the service and causing unintentional Denial of Service which some adversaries may avoid due to the perception that their actions will get noticed quickly.
Closing Notes
Thank you for reading this article. I truly appreciate your interest in how I approached testing this honeypot service. If you have any comments, feedback, or questions about this lab, feel free to share them here. Thank you to Christopher Wellons a.k.a. "skeeto" who created, developed, and published Endlessh for the public domain under the Unlicensed license.
Comments
Post a Comment