Strategic Detection Engineering: Enhancing Cloud Security for Government Applications
Writer: Jeremyah Joel Koesnadi; Co-Writer: Nur Adhi Nugroho, M. Saiful Islam
At GovTech Edu, our security team successfully overcame intricate digital challenges with resilience. Despite working within a budgetary constraint, we embraced the opportunity to enhance the security of aging systems undergoing digital transformation. While these systems were integral, they presented challenges regarding visibility and documentation, particularly compared to our in-house products. Recognizing this as an opportunity for improvement, we proactively addressed our security concerns within the given budgetary parameters.
Our approach was detection engineering. Using insights from our offensive security specialist, we developed a strategy focused on recognizing attack patterns. Our aim was clear yet challenging: to quickly spot any attempts to breach our systems. This approach allowed us to monitor all endpoints effectively, even those with limited visibility.
The success of our strategy was evident in real-world results. We found and fixed multiple high-risk vulnerabilities, even in less visible areas. This proactive method improved our defenses and highlighted the importance of staying ahead of potential threats.
With detection engineering, we transformed our security measures at GovTech Edu. This approach gave us a strong defense against known and unforeseen threats.
Detection Engineering: A Deep Dive
At its core, detection engineering pivots on a foundational principle: proactive identification. Instead of merely reacting to security breaches after they occur, the focus shifts to spotting and mitigating potential threats at the earliest possible stage. This proactive stance is particularly valuable when working with older, legacy systems, as it compensates for their inherent limitations.
Key Elements of Detection Engineering:
- Pattern Recognition: By analyzing historical data and drawing from known cyberattack methodologies and previous findings, systems can be trained to recognize signs of an impending attack. It’s akin to recognizing the storm clouds before the rain begins.
- Real-time Monitoring: Constant surveillance of all endpoints ensures that any deviation from the norm, however minute, is instantly flagged. This ensures swift action can be taken before any significant damage is inflicted.
- Behavioral Analysis: Beyond mere patterns, sophisticated detection systems can analyze behavior. For instance, an unusually high volume of data transfer from a specific endpoint might indicate a data breach attempt.
- Integration with Existing Systems: Especially crucial for legacy systems, detection tools can be integrated with existing infrastructure, ensuring that even older platforms benefit from modern security enhancements.
Detection Engineering: Addressing Key Threats
In the realm of cybersecurity, certain vulnerabilities frequently surface as prominent threats. Detection engineering offers a proactive approach to identify and counteract these challenges before they escalate.
Broken Access Control & Sensitive Data Exposure: These vulnerabilities revolve around unauthorized access. They represent gaps that might let intruders slip through, bypassing intended security measures or directly tapping into sensitive data. To counteract this, detection engineering emphasizes scrutinizing access request patterns. A sudden surge in access requests or any deviation from the norm could signal a breach attempt, warranting immediate investigation.
SQL Injection: Think of this as someone trying to trick an app by giving it bad information. Attackers use sneaky inputs to mess with how an app talks to its database. To spot and stop this, we watch how the database is used. If something looks strange or unsafe, our detection model will notify us immediately.
Remote Code Execution: Imagine this as someone getting control of your computer from somewhere else. Attackers use weak spots to run their actions on the target computer. To fight this, we monitor system and OS-level activities closely. If something unusual happens or a new action starts that we don’t recognize, we know something might be wrong.
Google Cloud Log Structure: A Foundation for Detection Engineering
In the expansive digital environment of Google Cloud, logs are the diligent chroniclers, capturing a vast array of activities. For those practicing detection engineering, these logs are indispensable, offering a wealth of insights into the system’s operations.
Requests logs, as the front gatekeepers, document the intricacies of HTTP requests and responses. They shine a light on patterns of user access, enabling discernment of not just routine behaviors, but also anomalies. A sudden surge in request rates or unfamiliar data destinations can hint at unauthorized access or potential breaches.
Standard Error logs act as the system’s alert system. Every hiccup, obstacle, or error within the digital machinery is promptly noted. Whether it’s due to software misconfigurations, failed processes, or even thwarted attempts to exploit the system, these logs raise the red flag, providing invaluable insights.
Standard Output logs present the daily rhythm of processes within the cloud. They offer a consistent baseline, painting a picture of ‘business as usual’. Yet, any departure from this routine, however minor, can serve as an early indicator of potential threats or system anomalies.
syslog, the all-encompassing journal, keeps track of messages from various system services and processes. It’s here that routine user logins coexist with critical system changes. By monitoring the Syslog, detection engineers can spot unusual system behaviors, unauthorized changes, or other indicators of potential compromise.
Addressing Broken Access Control & Sensitive Data Exposure Using Google Request Logs
Broken Access Control and Sensitive Data Exposure are issues that can let unauthorized users access information they shouldn’t. To tackle these problems, we turned to Google Cloud’s request logs.
Using Google Request Logs
Google Cloud’s request logs give us a detailed record of who accesses our apps and when. Every HTTP request is like a footprint of user activity. These logs are vital for spotting unusual or suspicious actions which might indicate security issues.
Setting Up Metrics and Alerts
By studying the usual patterns in the request logs, we get a sense of what’s normal for our apps. However, certain behaviors are red flags. For instance, a high number of requests in a short time might suggest someone is trying to guess IDs (like UUIDs) to gain unauthorized access. Similarly, a sudden increase in data downloads could mean someone is trying to pull out (or exfiltrate) a lot of information. We set up alerts for such unusual activities. When something doesn’t look right, we get an alert, helping us spot and address potential security threats quickly.
Working with the Product Engineering Team
Our collaboration with the product engineering team was key. They know the apps best and helped us understand what typical activity looks like based on past data. As our apps change and more users come on board, we update our understanding of what’s “normal” with their input. This ongoing teamwork ensures our alert system remains sharp and relevant.
Detecting SQL Injection Through stderr Logs
SQL injections can perturb a system by manipulating the typical behavior of database queries. However, these attempts often leave telltale signs in the form of specific error codes in stderr logs. Recognizing these signs is crucial to mount a swift defense.
Error Codes and Their Implications:
- Error 42601: Typically, this indicates a syntax error. Within an SQLi context, this error might surface when an attacker introduces characters like ‘ or % in a malicious payload that disrupts the query’s structure. While it can be a sign of an SQL injection attempt aimed at derailing the query, it’s worth noting that this error can also stem from unintentional application bugs or incorrect SQL formulations.
- Error 42501: This signals an insufficient privilege error. When seen in patterns or in certain contexts, it suggests unauthorized access attempts, as if an intruder is trying to view or alter tables they shouldn’t. However, it can also arise from simple misconfigurations or genuine application processes where permissions may not have been correctly set.
- Error 28000: Representing an authentication error, repeated occurrences of this error can be alarming. They might indicate someone’s efforts to guess database credentials, especially if targeted at information schema tables. Yet, occasional instances are login mistakes or application misconfigurations.
Our Proactive Approach:
Keeping a vigilant watch on these error codes in stderr logs arms us with early indicators of potential SQL injection attempts. When such codes appear in unusual patterns or alarming frequencies, it’s our cue to delve deeper. To try this approach, we run our internal test to trigger the new audits for a legacy application recently integrated into our landscape, and we can see the result as shown in the picture below
With the seamless integration of our alerting module to the legacy application following their migration to our cloud environment, our alerting module will process and monitor any anomalies in the application logs while the product engineering team is revamping the features. With this proactive approach, we manage to detect every possible loophole. We then take necessary actions, whether it’s launching a detailed investigation, blocking the suspicious paths, or reinforcing our legacy application code quality.
Detecting Remote Code Execution and Lateral Movement in VMs
Remote Code Execution offers attackers a gateway into systems. From there, lateral movement is their pathway to deeper infiltration, exploring the internal network and seeking opportunities to escalate privileges and compromise more assets.
Signs of Lateral Movement:
Once they gain a foothold, attackers deploy a range of tactics:
- Network Discovery with nmap: Tools like nmap assist attackers in mapping out the network, pinpointing active devices, and identifying potential targets.
The picture above shows our internal scenario to test the boundaries and robustness of the VMs we have access to, including the use of tools like nmap for network discovery and the execution of unfamiliar binaries or scripts. These actions are not indicative of a real case breach scenario but are part of our proactive security measures to ensure the integrity and safety of our systems.
- Privilege Escalation: To broaden their reach, attackers often seek ways to upgrade their access rights. This could involve exploiting system vulnerabilities or capitalizing on weak credentials.
In these endeavors, one recurring theme is the introduction and execution of unfamiliar binaries or scripts in the victim’s environment.
Leveraging Google SCC and Ops Agent:
Google Cloud’s Security Command Center (SCC) is a potent tool for keeping a vigilant eye on cloud assets. Its capabilities, when combined with the Google Ops Agent, become even more robust:
- Monitoring Unusual Binary Executions: With a well-defined baseline of standard operations for VMs, any deviation, like an unexpected binary running, can trigger alerts. Such anomalies often hint at an intruder’s presence.
- Data Movement Monitoring: We keep tabs on data transfer patterns. Any unusual movements, especially between different cloud services, can alert us to potential data exfiltration attempts.
- Integrating auditd for Enhanced Monitoring: auditd is a Linux auditing system, perfect for tracking system changes and calls. By integrating auditd with the Google Ops Agent, we can funnel these detailed logs to a centralized location. This integration provides granular insights into system activities, helping detect unauthorized changes or actions.
Our Holistic Approach:
Using both SCC and Ops Agent, complemented by operating system-level logging, we’ve cast a comprehensive surveillance net over our VMs. The moment an unusual activity or potential lateral movement sign emerges, we receive immediate notifications. Armed with this intelligence, we’re poised to swiftly respond, investigate, and neutralize potential threats.
In Conclusion
In the rapidly evolving world of cybersecurity, staying ahead of potential threats is paramount. By understanding the vulnerabilities and actively monitoring for signs of malicious activity, we equip ourselves with the tools and knowledge to defend our digital assets. Remember, the key isn’t just to react when something happens but to be proactive, always vigilant, and ready. With the right strategies in place, we not only safeguard our systems but also ensure the trust and reliability that our users place in us.
Stay safe, stay informed, and always be prepared.
About the writers
Jeremyah Joel Koesnadi
As a product security engineer at GovTech Edu, he specializes in merging technical skills with strategic management for effective risk management across products. His background included security consulting in the banking and financial sectors in Southeast Asia, and he was responsible for the offensive security operations at Decacorn startup. His current role at GovTech centers around securing the digital transformation process of legacy applications.
