Machine Learning Security Evasion Competition 2.0

For CSCE 482, Texas A&M’s Engineering Capstone class for Computer Science & Engineering Majors, our team reached out to Dr. Marcus Botacin, a professor I had taken previously for CSCE 413 and CSCE 439, both of which have been pivotal for my cybersecurity education. Our team wanted to fulfill a project idea that Dr. Botacin had mentioned the previous semester- a one-and-done open source platform for hosting a Machine Learning Security Evasion Competition (MLSEC).

The GitHub repository can be found here: https://github.com/Botacin-s-Lab/mlsec-comp-platform

What is the Machine Learning Security Evasion Competition?

The MLSEC was previously hosted in 2022. Each team has the opportunity to create an attack and a defense.

Attacks

An attack is where a team takes a competition-provided malware sample and makes it evasive, i.e., hard to detect by antivirus software. The competition provides these samples to set a baseline for behavior- if the evasive sample behaves too differently from the provided sample, then the evasive techniques have possibly rendered the malware useless and are thus disqualified. You can zero all the bits in a sample, and it will assuredly be evasive, but it won’t work.

Defenses

A defense is where a team uses machine learning to classify attacks (binaries made evasive by attackers). These antiviruses are packed into Docker containers, which are then booted up by competition hosts, fed attacks, and scored. The defense containers must meet certain standards of accuracy and performance (95% accuracy, < 1 GB RAM, 5s classification) to be able to enter the competition. Competition hosts provide a defense evaluation set (a set of goodware and malware) that is (ideally) randomly fed into defense containers. If the defense container cannot boot, classify samples in time, classify samples under the allotted amount of computer, or classify samples accurately, it may not be entered into the competition. A defense container that calls everything malware would be powerful; however, the defense validation set ensures against this.

Hosting a MLSEC-style Competition

The rise of AI-assisted code generation has given a big advantage to attackers. Creating malware variants is much easier now, so we must prepare defenders with the skills and technology to combat this paradigm shift. Sadly, hosting competitions like MLSEC is time-consuming, logistically complex, and expensive. Our capstone, MLSEC 2.0, hopes combat these factors by making a free platform that manages all of these for you.

What is MLSEC 2.0?

MLSEC 2.0 is our take on a competition platform for MLSEC. It is a series of Docker containers (for modularization) that host a platform (website) for the competition. It automatically ingests security artifacts (malware attack samples and defense dockers), performs verification on said submitted samples, and pits attacks against defenses. The scores are then automatically collected and displayed on a live leaderboard on the website. Since our platform is accepting real Windows PE binary malware as well as Docker containers, our system has many features in place to contain and isolate user-submitted data.

I could keep rambling because I think our work is really cool, but I’ll instead point you to the aforementioned GitHub link to check out the docs yourself.

Roles in MLSEC 2.0

It was an absolute pleasure to work with my team on this project. As a quick overview of our roles:

• Aaron Thompson was the head of the Gateway (a Docker container that managed network traffic for security) and worked with me on the parallelism of our platform.
• Graham Dungan (me) designed the architecture for parallelism, system grading, and worked with Karl on the API and Maxim on the sandbox and frontend.
• Karl Farrar was the head of API and authentication and worked on testing, documentation, and team organization.
• Maxim Mouget was the head of frontend and sandbox (don’t let these two things fool you- sandbox was a feat of its own).

This is the most elaborate project I’ve worked on yet, and I’m glad to have had such a responsive, knowledgeable, and tenacious team.

The video below summaries the outcome of this project.