tag:blogger.com,1999:blog-20227807070534839382024-03-13T12:45:55.896-07:00The REdoubtA blog about reverse-engineering, malware, coding, and other random stuff.R3doubthttp://www.blogger.com/profile/11540283914762258246noreply@blogger.comBlogger15125tag:blogger.com,1999:blog-2022780707053483938.post-91677146461968408662021-10-18T12:07:00.001-07:002021-10-18T12:07:08.935-07:00Boldly Going<p></p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/a/AVvXsEiS_YwFvfgpoUNpBbyZMNBhLf1hGNU4AepGYcMwozr1MzbF7PLTYK2cXPZ5t82kQpghUCHOqZ1CLNW7tIVwMkH654bQ3V6CDlmcaNDxid5SPTzmq-o1xc0jHUFMMj86dNyYY5WeZ6VwirJo2e_dJ4mBW9kOerPmf0sFgli352et8M7ksglxM53jLGuPIQ=s1920" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1080" data-original-width="1920" height="180" src="https://blogger.googleusercontent.com/img/a/AVvXsEiS_YwFvfgpoUNpBbyZMNBhLf1hGNU4AepGYcMwozr1MzbF7PLTYK2cXPZ5t82kQpghUCHOqZ1CLNW7tIVwMkH654bQ3V6CDlmcaNDxid5SPTzmq-o1xc0jHUFMMj86dNyYY5WeZ6VwirJo2e_dJ4mBW9kOerPmf0sFgli352et8M7ksglxM53jLGuPIQ=s320" width="320" /></a></div><br /> <div class="separator" style="clear: both; text-align: center;"></div><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/a/AVvXsEg3Vq3AiMdPz2nnplVl7TUSB2soJym1Iac2zoc-ulGwFhb9fFnPNmSRicke9TK9O_W_qEZFm5XTPiHABb0rImZkpjPhogGy7pwZWU_LSJiV1KBYnMm6OHs9-uz_KeFvmRY5ofNtR5aP-hXoMFsQQM4lU81qLo2796MnML_LQzHr5LxlOJ4v5BCpbWk0ng=s768" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="403" data-original-width="768" height="168" src="https://blogger.googleusercontent.com/img/a/AVvXsEg3Vq3AiMdPz2nnplVl7TUSB2soJym1Iac2zoc-ulGwFhb9fFnPNmSRicke9TK9O_W_qEZFm5XTPiHABb0rImZkpjPhogGy7pwZWU_LSJiV1KBYnMm6OHs9-uz_KeFvmRY5ofNtR5aP-hXoMFsQQM4lU81qLo2796MnML_LQzHr5LxlOJ4v5BCpbWk0ng=s320" width="320" /></a></div> <p></p><p>"We prefer to help ourselves. We make mistakes, but we're human--and
maybe that's the word that best explains us."–Captain James T. Kirk</p><p> Watching Blue Origin's New Shepard and William Shatner conquering the gravity well at 90, was a moment that put a smile on my face.<br /></p>R3doubthttp://www.blogger.com/profile/11540283914762258246noreply@blogger.com0tag:blogger.com,1999:blog-2022780707053483938.post-51483434547483927012021-03-27T05:16:00.005-07:002021-03-27T05:16:51.359-07:00Update on the Suez Canal situation<p> </p><p>An update on the Suez Canal situation with the Ever Given. Recovery efforts on Friday failed, meaning that as of this post on Saturday, this appears to be the longest complete shutdown of the canal since it reopened following the 1973 Arab-Israeli conflict. Initial reports in the media are pointing to high winds and a sandstorm as the reason the ship veered off course and ran aground. Dozens of ships have accidents in the Suez each year, but I'm still a bit skeptical. <br /></p><p>In another twist, it has been <a href="https://thebarentsobserver.com/en/industry-and-energy/2021/03/making-fun-suez-traffic-jam-rosatom-promotes-northern-sea-route ">reported</a> that the Russians have been mocking the incident and suggested shippers consider an alternate Arctic route, perhaps facilitated by Russia's fleet of nuclear powered ice-breakers. I am guessing this is just opportunistic propaganda by the Russians, and idea sounds a bit far fetched. It does demonstrate yet another motive and mindset that could lead to a state sponsored cyber attack on maritime choke points. Interesting...<br /></p><p><br /></p>R3doubthttp://www.blogger.com/profile/11540283914762258246noreply@blogger.com0tag:blogger.com,1999:blog-2022780707053483938.post-31983949130170053412021-03-26T09:06:00.007-07:002021-03-26T09:18:03.578-07:00Addressing the Maritime Threat<p>As is so often the case, there is no single, magic bullet "solution" to this public policy concern. There are, however, choices we can make in public policy to minimize negative outcomes. As in other public policy matters, the maritime cybersecurity challenge isn't going to be solved by a quick cash dump from government coffers. Improvement is going to take long term policy focus and a shift in culture.</p><p>The US Coast Guard took some first steps in this direction with <a href="https://mariners.coastguard.blog/2020/03/25/nvic-01-20-guidelines-for-addressing-cyber-risks-at-mtsa-regulated-facilities/">NVIC 01-20 last year.</a> The policy, for the first time "provides guidance to facility owners and operators
on complying with the requirements to assess, document, and address computer
system and network vulnerabilities." For MTSA facilities that have never had a requirement to even think about cyber security until now, a gradual approach to implementing governance standards makes sense. It will take time, and iterative updates and course corrections, hopefully with private sector input. If it is done correctly, it could foster greater cybersecurity awareness in maritime industry culture, and that will have more impact than installing firewalls and IDS.</p><p>A second significant step in the right direction was the US maritime national security strategy released in January. Josh Steinman, the senior cyber policy advisor on the Trump administration's National Security Council staff, and a Navy veteran, certainly seemed like he grasped the essence of the matter. He was even our keynote speaker for the first year of Hack The Sea village at DEF CON. <br /><br />The policy, as I pointed out in my interview with<a href="https://www.cyberscoop.com/maritime-cybersecurity-trump-white-house/"> Cyberscoop</a>, it is far from perfect. There is little in the policy to address the supply chain threats I've mentioned in my previous post. But the policy is a good start. The goals it outlines for workforce development, for instance, are certainly laudable. Poor training and certification standards, and a poor quality workforce have plagued cybersecurity for years. It is my sincere hope the Biden administration will follow through with the strategy released in January, and not let it fall victim to partisan rancor. </p><p>It is also my hope that improvements can be made with respect to the supply chain dilemma. The FDA's efforts to improve security in the medical device industry, for example, could be used as a template for fixing the security of maritime operational technology (OT). <br /></p><p>Gradual progress, along the course set by the two policies I've mentioned here, could partially mitigate the threat of a scenario like the one I described in my last post. <br /></p><p><br /></p>R3doubthttp://www.blogger.com/profile/11540283914762258246noreply@blogger.com2tag:blogger.com,1999:blog-2022780707053483938.post-7757822552047060202021-03-26T08:08:00.001-07:002021-03-26T09:20:22.692-07:00How To Use Maritime Cyber To Cripple Power Projection<p><span class="break-words"><span><span dir="ltr">Imagine you are a Chinese planner tasked with limiting US response to an impending campaign to seize Taiwan. Blocking the Panama Canal, and
capsizing roll-on-roll-off vessels in ports, to block our sea-lift capabilities would certainly be a good start.</span></span></span><span class="break-words"><span><span dir="ltr"> </span></span></span><span class="break-words"><span><span dir="ltr">Let's review the elements involved in such an operation. </span></span></span></p><p><span class="break-words"><span><span dir="ltr">First, an attacker would have to get access to port facilities and their EDIFACT systems. This has already been done numerous times "in the wild". A noteworthy example was the <a href="https://www.bbc.com/news/world-europe-24539417">breach of the Port of Antwerp</a> discovered in 2012. Attackers used physical devices placed onsite, phishing emails, and other techniques to gain access to the port's C-Point Nxtport environment, and manipulate EDIFACT messages.</span></span></span></p><p><span class="break-words"><span><span dir="ltr">While the attack on Antwerp was focused smuggling contraband in cargo containers, the same access and capabilities could be used to sink vessels. Pen Test Partners, in the UK, explained how this was possible back<a href="https://www.pentestpartners.com/security-blog/sinking-container-ships-by-hacking-load-plan-software/"> in 2017</a>.</span></span></span><br /><span class="break-words"><span><span dir="ltr"></span></span></span></p><p><span class="break-words"><span></span></span>Now we can move on to blocking those sea lanes like the DC Beltway at rush hour. We had a glimpse of how this could happen this week when the Ever Given, a container ship from Taiwan's Evergreen line, appeared to navigate in circles then lodged itself sideways in the middle of the Suez Canal before shutting down engines.</p><p><span class="break-words"><span><span dir="ltr">Attacking
the engines on a ship, to shut them down the way it happened in the
Suez this week, was something my team demonstrated with
<a href="https://www.helpnetsecurity.com/2018/10/18/manipulate-marine-diesel-engines/">Auto-Maskin controllers in 2018</a>. I would also like to note that even after we notified authorities, and Auto-Maskin, little was done to address the security vulnerabilities, no effort was made to rush out a patched firmware update. The family of engine control units continued to be used by major marine diesel OEMs, often rebranded, leaving end-users unaware of the danger in their supply chain and engine room. <br /></span></span></span></p><p><span class="break-words"><span><span dir="ltr">Now, let's talk navigation. A lot of researchers have demonstrated attacks on various components of a ship's navigation systems that could have easily resulted in
the "interesting" AIS tracks that preceded the wreck. One of them is my colleague Gary Kessler, who <a href="https://www.youtube.com/watch?v=9xBze1hZVgk"> gave a talk about it at DEF CON</a> last year. He described how an attacker could subvert systems like AIS and ECDIS, and even cause coastal navigation lanes and related buoys to appear to "move", leading to disaster. A number of maritime incidents over recent years could be examples of such attacks "in the wild", but I'll decline making an assessment without more evidence.<br /></span></span></span></p><p><span class="break-words"><span><span dir="ltr"></span></span></span></p><p><span class="break-words"><span><span dir="ltr">So, as it relates to what happened in the Suez this week, what do I think about the chances the attack was the work of a sophisticated, cyber threat actor? Well, I'm not saying it was aliens. But...<br /></span></span></span></p><p><span class="break-words"><span><span dir="ltr"></span></span></span></p><p><span class="break-words"><span><span dir="ltr"></span></span></span></p><br /><span class="break-words"><span><span dir="ltr"><img alt="Ancient Aliens TV Show" class="detail__media__img-highres js-detail-img js-detail-img-high" src="https://external-content.duckduckgo.com/iu/?u=https%3A%2F%2Fwww.monstersandcritics.com%2Fwp-content%2Fuploads%2F2016%2F07%2Fancient-aliens-1.jpg&f=1&nofb=1" style="display: block; height: 262px; width: 262px;" /></span></span></span><p><br /></p><p><span class="break-words"><span><span dir="ltr"> </span></span></span></p><p><span class="break-words"><span><span dir="ltr"> </span></span></span></p>R3doubthttp://www.blogger.com/profile/11540283914762258246noreply@blogger.com2tag:blogger.com,1999:blog-2022780707053483938.post-25072054510352155432021-03-25T17:39:00.000-07:002021-03-25T17:39:30.160-07:00A Salty Week<p>So, ICYMI, someone decided to do some AIS <a href="https://www.vice.com/en/article/pkdjzb/cargo-ship-suez-canal-dick-pic-ever-given ">donuts</a> and then park a container ship in the Suez. </p><p></p><div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-C7WBhp_NEqs/YF0stURnEqI/AAAAAAAAAlI/eB344ELLs7w3uhrcvJE-pH_yBHemOPIEQCLcBGAsYHQ/s1920/suez.jpeg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1080" data-original-width="1920" src="https://1.bp.blogspot.com/-C7WBhp_NEqs/YF0stURnEqI/AAAAAAAAAlI/eB344ELLs7w3uhrcvJE-pH_yBHemOPIEQCLcBGAsYHQ/s320/suez.jpeg" width="320" /></a></div><br /> <p></p><p>Meanwhile, I got to pwn both maritime and aviation systems at <a href="https://www.hackthemachine.ai/">HackTheMachine</a>. A lot of N2K and ARINC 429, lots of caffeine, loads of fun. My teammate Hipu left a minecraft server on one of the "ships" after getting root. I guess they should be glad he didn't engage in any "art" projects.</p><p> </p><div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-5gKEjidi8OM/YF0rchOJzRI/AAAAAAAAAk8/8NEMRYi7xvIL8wt3wwEOhRo3eIxG-ov5wCLcBGAsYHQ/s2048/copter3.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1356" data-original-width="2048" src="https://1.bp.blogspot.com/-5gKEjidi8OM/YF0rchOJzRI/AAAAAAAAAk8/8NEMRYi7xvIL8wt3wwEOhRo3eIxG-ov5wCLcBGAsYHQ/s320/copter3.jpg" width="320" /></a></div><br /> <br /><p></p>R3doubthttp://www.blogger.com/profile/11540283914762258246noreply@blogger.com1tag:blogger.com,1999:blog-2022780707053483938.post-21273469692007075372021-03-22T04:50:00.001-07:002021-03-22T04:50:40.240-07:00VBS Rat<p>This right up by <a href="https://yoroi.company/research/threatening-within-budget-how-wsh-rat-is-abused-by-cyber-crooks/">Yoroi </a>may be of interest to anyone who has taken my script malware course. Commodity RAT dropped using N-day exploit & RTF doc file.<br /></p>R3doubthttp://www.blogger.com/profile/11540283914762258246noreply@blogger.com1tag:blogger.com,1999:blog-2022780707053483938.post-74371656019983035292021-03-20T08:02:00.003-07:002021-03-20T08:02:46.555-07:00Updates for 2021<p> I haven't posted in quite a while, so time for an update. I've been busy getting <a href="https://redoubtresearch.com ">Redoubt Research</a>, LLC off the ground and running Hack The Sea 2.0 in 2020. <br /></p><p> I submitted the application for <a href="https://www.defcon.org/html/defcon-29/dc-29-cfe.html">DEF CON villages</a> for 2021. I hope to see everyone in person in Vegas this year for <a href="http://hackthesea.org/">Hack The Sea 3.0, Deep Dive</a>, as much fun as the virtual conference was last year.</p><p>Currently working on some new
research on maritime ICS/OT, but can't say more than that due to NDA.
Not sure if we will wrap up ethical disclosure in time to enter a talk
at DEF CON, but I will certainly have interesting stuff for you all in
2021.</p><p> Finally, I will be at <a href="https://www.hackthemachine.ai/">Hack The Machine</a> with friends from <a href="https://www.fathom5.co/">Fathom5</a> next week. I am pretty stoked, and prepping some scripts and gear this weekend. The Navy has some very cool content on tap, and there is virtual space participation. Hope to see you there!<br /></p><p><br /><br /></p><p> </p>R3doubthttp://www.blogger.com/profile/11540283914762258246noreply@blogger.com1tag:blogger.com,1999:blog-2022780707053483938.post-52927654605842531622019-06-20T13:42:00.002-07:002019-06-20T13:42:46.093-07:00DEF CON 27I have been pretty busy helping organize HackTheSea for <a href="https://www.defcon.org/html/defcon-27/dc-27-index.html" target="_blank">DEF CON 27</a>. This will be the first ever maritime and ship hacking village at DEF CON. We are going to have a team CTF on a very nifty "ship" environment provided by <a href="https://www.fathom5.co/" target="_blank">Fathom5</a>. There will also be a series of workshops for n00bs to maritime ICS/OT with lots of gear available for hacking practice. Our CFP is currently open, but closing soon for those doing original research in high seas hacking. In keeping with the DEF CON theme this year, Technology's Promise, we will also have <a href="https://www.seasteading.org/" target="_blank">Seasteading Institute</a> on hand to talk about living at sea and to kick-off a hackathon challenge for the Seasteading Community.<br />
<br />
Plus, lot's of pirate meme stuff.<br />
<br />
<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://1.bp.blogspot.com/-TejGL0MG2R8/XQvvlyp_-XI/AAAAAAAAAUc/Kb-yOZH0hN0GYUWZO1x9QtjJfhON9cpygCLcBGAs/s1600/download.jpeg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="400" data-original-width="400" height="320" src="https://1.bp.blogspot.com/-TejGL0MG2R8/XQvvlyp_-XI/AAAAAAAAAUc/Kb-yOZH0hN0GYUWZO1x9QtjJfhON9cpygCLcBGAs/s320/download.jpeg" width="320" /></a></div>
<br />R3doubthttp://www.blogger.com/profile/11540283914762258246noreply@blogger.com1tag:blogger.com,1999:blog-2022780707053483938.post-67088784250045211892019-03-02T22:25:00.000-08:002019-03-02T22:25:17.583-08:00<span id="docs-internal-guid-ec2f63dc-7fff-42d2-0ce7-c6c9df576f31"><span style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;"><a href="http://www.bsidesnova.org/call-for-papers/" target="_blank">BSides NOVA </a>was great. Got a lot of interesting questions and feedback on my malware analysis training. Taught lots of new lock-pickers at the LPV with<a href="https://toool.us/" target="_blank"> TOOOL</a>. Saw some great talks and Watched War Games with an audience of hackers and<a href="https://www.iamthecavalry.org/" target="_blank"> I Am The Cavalry</a>. </span></span><br />
<span><span style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;"><br /></span></span>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://2.bp.blogspot.com/-0s7RdOVRKr4/XHtzJyqT6hI/AAAAAAAAARo/9RUG-kMzLjc1hLj5o2y7ruzZhzle-_KuQCLcBGAs/s1600/D0rkOXgX4AEGAFo.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="900" data-original-width="1200" height="240" src="https://2.bp.blogspot.com/-0s7RdOVRKr4/XHtzJyqT6hI/AAAAAAAAARo/9RUG-kMzLjc1hLj5o2y7ruzZhzle-_KuQCLcBGAs/s320/D0rkOXgX4AEGAFo.jpg" width="320" /></a></div>
<span><span style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;"><br /></span></span>R3doubthttp://www.blogger.com/profile/11540283914762258246noreply@blogger.com0tag:blogger.com,1999:blog-2022780707053483938.post-54620273919227671092019-02-23T18:15:00.004-08:002019-02-23T18:15:43.816-08:00<h2>
Update</h2>
<div>
For anyone attending the malware training, you will need a hex editor installed for one of the labs. It doesn't matter what editor you choose, we won't be using it much.</div>
R3doubthttp://www.blogger.com/profile/11540283914762258246noreply@blogger.com1tag:blogger.com,1999:blog-2022780707053483938.post-45983530482685446162019-02-22T17:51:00.000-08:002019-02-22T17:51:12.945-08:00<h2>
BSides NoVA Training</h2>
<div>
For anyone attending my malware analysis workshop, <b>No Disassembly Required</b>, next week at <a href="http://www.bsidesnova.org/" target="_blank">BSides NoVA</a> here is the very brief list of what to bring and prerequisites.</div>
<h4>
<br />Laptop</h4>
Enough available memory and storage to run a Windows 10 VM<br />
Hypervisor of your choice installed (Hyper-V, VirtualBox, VMware, whatever)<br />
WiFi enabled<br />
*Wired Ethernet adapter if needed (example, Thunderbolt to Ethernet) just in case<br />
<br />
<h4>
Windows VM</h4>
Configured with NATed or Bridged network access<br />
Notepad++<br />
<a href="https://www.decalage.info/python/oletools" target="_blank">OleTools</a><br />
<br />
<h4>
Prerequisites</h4>
<div>
Willingness to learn and a basic familiarity with a Hypervisor software and VMs. The course is designed to let students work at their own pace on the labs, with progressively more difficult challenges to complete. </div>
R3doubthttp://www.blogger.com/profile/11540283914762258246noreply@blogger.com2tag:blogger.com,1999:blog-2022780707053483938.post-63251797558827305402018-10-17T11:48:00.001-07:002018-10-17T11:49:33.886-07:00UpdateAfter a bit of a hiatus I hope to start blogging more regularly, or at least to get caught up on posting about various topics I have spoken on at recent conferences. For those who missed it I spoke on script based malware including so-called "fileless" intrusions, and taught a related training at BSides-Pittsburgh in June. Last week, I co-presented a talk at DerbyCon about hacking maritime SCADA. I will be working on some posts related to those talks, both technical deep-dives, and sharing some of my experiences with "soft-skill" topics like presenting at conferences, writing CFP responses, talking to reporters, and coordinating ethical disclosure of vulnerabilities.R3doubthttp://www.blogger.com/profile/11540283914762258246noreply@blogger.com4tag:blogger.com,1999:blog-2022780707053483938.post-44712204142279262602018-04-20T21:42:00.003-07:002018-04-21T01:02:19.492-07:00Part Three: Triage With Osquery and Obtaining IndicatorsIn this post, I’ll go more in-depth about how I used osquery to find the indicators in part two, and how to use the same technique for threat hunting on macOS.<br />
<br />
<h3>
Osquery Logs</h3>
<br />
There are two styles of results logs in osquery, snapshots and differential logs. Snapshots are exactly what the name suggests, results of queries executed at a single point in time. Differential logs only add a row when results of a query change. As a simple example, I have created an osquery.conf file with scheduled queries for listening_ports and launchd tables. Applying these configs gives us a differential log.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://4.bp.blogspot.com/-OzKPSAW8QTM/Wtqx90PPzLI/AAAAAAAAAFg/KnLLQ3WNLqU5tNF927WLvEi99Wf-LZVoQCLcBGAs/s1600/ioc.conf.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" data-original-height="402" data-original-width="725" height="177" src="https://4.bp.blogspot.com/-OzKPSAW8QTM/Wtqx90PPzLI/AAAAAAAAAFg/KnLLQ3WNLqU5tNF927WLvEi99Wf-LZVoQCLcBGAs/s320/ioc.conf.png" width="320" /></a></div>
<br />
The first set of entries from launchd are the baseline. This log can be saved or discarded. For our purposes we just need the subsequent changes, which will list any new property lists added to the launchd paths. We could get the same information without using osquery by running "launchctl list" and writing a script to parse it, but osquery is a convenient tool to use for this purpose.<br />
<br />
The next step is to take a snapshot on my virtual machine, and then run the trojan PDF from part one. After the Mach-O binary executes, we see the decoy document appear.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://3.bp.blogspot.com/-iO229DWFfoM/WtqyQ5rdzII/AAAAAAAAAFo/TSLu80lAx5QUTqosjf6CYYiQks6yYoN8wCLcBGAs/s1600/pdfDecoy.png" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" data-original-height="654" data-original-width="887" height="146" src="https://3.bp.blogspot.com/-iO229DWFfoM/WtqyQ5rdzII/AAAAAAAAAFo/TSLu80lAx5QUTqosjf6CYYiQks6yYoN8wCLcBGAs/s200/pdfDecoy.png" width="200" /></a></div>
Checking our osqueryd.results.log again, we can see we have a new row in our log. Reading through the results we can see that indeed the malware has added a property list to our launch directory for persistence. We can also see the paths we need to locate the property list, which will in turn yield the path to the persistent executable, as we saw in part two.<br />
<br />
This is an extremely simple example of dynamic analysis. The technique could be automated further with scripting to create our own custom "sandbox". With it we could quickly find low-hanging fruit indicators from our macOS malware sample, or from suspected phishing email attachment. Additional tables can be added to monitor other potential persistence mechanisms like crontab, or even more stealthy methods like @xorrior's <a href="https://www.xorrior.com/emond-persistence/" target="_blank">emond based persistence</a>.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://3.bp.blogspot.com/-pwgQ7czEIvY/Wtq-n39k6aI/AAAAAAAAAF8/pYVHjT8PbyIzKthqC-8svcyyOW4zmXnxwCEwYBhgL/s1600/logs.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" data-original-height="307" data-original-width="592" height="165" src="https://3.bp.blogspot.com/-pwgQ7czEIvY/Wtq-n39k6aI/AAAAAAAAAF8/pYVHjT8PbyIzKthqC-8svcyyOW4zmXnxwCEwYBhgL/s320/logs.png" width="320" /></a></div>
I am still exploring the capabilities of osquery for ways to detect other malicious behavior and indicators. For example, I've been exploring the best way to capture network information using only osquery tables. I experimented with the listening_ports table and found that it does indeed log processes listening on a port. I am able to see, for instance <span style="text-align: center;">the mDNSResponder service listening as expected on UDP port 5353. Unfortunately, listening_ports does not address reverse-shells or other backdoor and C2 mechanisms. </span><br />
<span style="text-align: center;"><br /></span><span style="text-align: center;">Another interesting table I have experimented with is file_events, and it seems to have some cool possibilities. One use case would be to create a substitute for Windows Security Event ID 4663, and use it for something like @neu5ron's <a href="https://nathanguagenti.blogspot.com/2018/01/canary-files-for-legit-access-abuse-with-wef-and-elk.html" target="_blank">canary files.</a> I'll keep exploring, and keep blogging about what I find. </span>R3doubthttp://www.blogger.com/profile/11540283914762258246noreply@blogger.com0tag:blogger.com,1999:blog-2022780707053483938.post-65210631407945556972018-04-16T15:25:00.002-07:002018-04-29T08:17:49.464-07:00Part Two: Writing the Rules<div class="separator" style="clear: both; text-align: center;">
</div>
<h3>
Query Rules</h3>
<br />
As I said in <a href="https://blog.r3doubt.io/2018/04/recently-i-decided-to-learn-how-to.html" target="_blank">part one</a>, I began by checking the osx-attacks pack for existing rules. While there were no rules to detect the newest APT32 associated variants, I used the pack as a template for writing my own. Packs have only a few necessary elements. The first element in the osx-attacks pack is a platform key that specifies the value for macOS ("darwin"). The next element, "queries", quite naturally contains the rules we want applied, formatted as a series of SQL queries.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://3.bp.blogspot.com/-mXX8iOdCtcM/WtT7e_VOSuI/AAAAAAAAAEo/1OXqUIRxMsU0zYThQqNosMPZRYNrb4UBQCPcBGAYYCw/s1600/wirelurker.png" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" data-original-height="340" data-original-width="644" height="168" src="https://3.bp.blogspot.com/-mXX8iOdCtcM/WtT7e_VOSuI/AAAAAAAAAEo/1OXqUIRxMsU0zYThQqNosMPZRYNrb4UBQCPcBGAYYCw/s320/wirelurker.png" width="320" /></a></div>
Looking at the first rule, "Wirelurker", we see that it consists of a query selecting all columns from the table "launchd" where the row has a name value that ends in ".plist". The other keys in the Wirelurker rule are pretty straight forward. "Interval" specifies in seconds how often to run the query. The "version" key is the minimum version of osquery compatible with this rule. The two remaining fields, "value" and "description" are just optional meta-data for informational purposes.<br />
<br />
Before going any further, a brief note on the syntax. The osx attacks pack includes a bash-style backslash to indicate line continuation. Other example configuration files from the Facebook osquery repository include C-style comments (//). These elements were compatible with the json parser in earlier versions of osquery, but may not be compatible with newer versions. The parser in version 3.1.0, the latest version installed by Brew as of my writing this post, throws errors on these non-compliant elements. I have confirmed this via experimentation and talking to osquery developers via the Slack channels.<br />
<br />
Now let's go back to that table "launchd", and answer "what is a .plist?" and "why are we looking for them?" For additional information you can always check out my talks from <a href="https://bupmediasite.passhe.edu/Mediasite/Showcase/bloomcon/Presentation/9441d384b2c047a599bb6c0b3a79ab881d" target="_blank">BloomCon 2018</a> and <a href="http://www.irongeek.com/i.php?page=videos/bsidescolumbus2018/e01-pass-the-apple-sauce-mac-os-x-security-automation-for-windows-focused-blue-teams-brian-satira" target="_blank">BSides Columbus 2018</a>. The short answer is that property lists (.plist) and launchd are the way at least half of macOS malware variants, according to my own completely un-scientific estimate, achieve persistence.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://2.bp.blogspot.com/-KY18Hf35zZU/WtUcWwW6jOI/AAAAAAAAAE0/VQc5hfxgkuEv01fZ1Uy0T-SmM6O24LmQwCLcBGAs/s1600/PlistHidd.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" data-original-height="376" data-original-width="558" height="215" src="https://2.bp.blogspot.com/-KY18Hf35zZU/WtUcWwW6jOI/AAAAAAAAAE0/VQc5hfxgkuEv01fZ1Uy0T-SmM6O24LmQwCLcBGAs/s320/PlistHidd.png" width="320" /></a></div>
The more complete answer is that launch daemon (launchd) is the process started by kernel_task (pid 0) on startup (as pid 1) to ensure that other processes that need to run on startup on macOS get run. Launchd accomplishes this by walking a set of special directories and parsing a set of special xml files, which are the startup "property lists". These directories and property lists serve much the same function as the Windows' "run" and "run once" registry keys.<br />
<br />
Within the .plist file for the APT32 variant shown here, a "RunAtLoad", "true" key-value pair ensures persistence through a shutdown or restart, while "KeepAlive", "true" key-value pair ensures the process is restarted if stopped. Lastly, the "ProgramArguments", "/Users/thewoz/Library/Containers/com.apple.lateragent/Data/Library/Preferences/hidd" key-value pair gives us the full path of the Mach-O binary "hidd", which is written to disk in the victim's ~/Library/Containers folder and concealed with a "hidden" xattribute, which launchd is required to run on startup.<br />
<br />
Using this information, we can create two rules for query pack. The first will follow our osx-attacks pack template and query the launchd table to detect the presence of "com.apple.hidd.shared.plist", a good indicator of this variant. The second rule will query the "file" table for that persistent Mach-O binary. Results can either be logged locally to the default /var/osquery/log/osqueryd.results.log, or can be aggregated for ingestion by Splunk, LogStash, Fluentd, or other options. But that will have to wait for a future post.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://4.bp.blogspot.com/-bOrJMN06tBA/WtUiIxnvXvI/AAAAAAAAAFI/rzKjYyBklSMbaxRXvh4lmnaR5x3Ow5-wACLcBGAs/s1600/lotus.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="389" data-original-width="730" height="170" src="https://4.bp.blogspot.com/-bOrJMN06tBA/WtUiIxnvXvI/AAAAAAAAAFI/rzKjYyBklSMbaxRXvh4lmnaR5x3Ow5-wACLcBGAs/s320/lotus.png" width="320" /></a></div>
<br />
<div class="separator" style="clear: both; text-align: center;">
</div>
A final note, after writing my rules I ran them through <a href="http://jsonlint.com/">jsonlint.com</a>, just to make sure I don't get any json parser related errors. I would recommend it as a practice. Yes, these are pretty static indicators, stick around for part three, where I will discuss more "flexible" queries. Again, the full query pack can be found on my <a href="https://github.com/r3doubt/apple-sauce-in-a-bucket/blob/master/APT32-pack.conf" target="_blank">github</a> repository.<br />
<br />
<h3>
<a href="http://blog.r3doubt.io/2018/04/part-three-triage-with-osquery-and.html" target="_blank">Part Three: Triage With Osquery and Obtaining Indicator</a>s</h3>
<br />
<br />
<br />
<br />
<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
</div>
R3doubthttp://www.blogger.com/profile/11540283914762258246noreply@blogger.com12tag:blogger.com,1999:blog-2022780707053483938.post-69340006410014128452018-04-15T18:29:00.002-07:002018-04-16T19:20:25.162-07:00Writing Query Packs to Detect APT32's macOS Malware Variants: Part One<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="white-space: pre-wrap;"><span style="font-family: "arial" , "helvetica" , sans-serif;">Recently, I decided to learn how to write my own query packs for <a href="https://osquery.io/" target="_blank">osquery</a>. I’ve been spending a lot of time lately learning about macOS malware and osquery has proven to be a useful tool. When I saw a report from <a href="https://blog.trendmicro.com/trendlabs-security-intelligence/new-macos-backdoor-linked-to-oceanlotus-found/?utm_campaign=shareaholic&utm_medium=twitter&utm_source=socialnetwork" target="_blank">Trend Micro</a> of a new malware variant targeting macOS, it seemed like a good opportunity for some practice.</span></span></div>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: "arial";"><span style="font-family: "arial" , "helvetica" , sans-serif; white-space: pre-wrap;"><br /></span></span></div>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: "arial";"><span style="font-family: "arial" , "helvetica" , sans-serif; white-space: pre-wrap;">In this first part of a <strike>two</strike> three part post, I will go over some query pack basics and some info about macOS malware variants. In part two I'll go over details on writing queries. In part three, I’ll go more in-depth about how I used osquery to find the indicators, and how to use the same technique for threat hunting on macOS. </span></span></div>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<br /></div>
<h3 style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: "arial";"><span style="font-family: "arial" , "helvetica" , sans-serif; font-size: small; white-space: pre-wrap;">Query Packs</span></span></h3>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: "arial";"><span style="font-family: "arial" , "helvetica" , sans-serif; white-space: pre-wrap;">Osquery can be used in two modes, interactive and as a daemon. When running as a daemon, lists of rules, supplied as external json files called query packs, can be applied. These query packs are applied by simply adding a file path to the “packs” section of the configuration file, /var/osquery/osquery.conf.</span></span><br />
<div class="separator" style="clear: both; text-align: center;">
<span style="font-family: "arial";"><span style="font-family: "arial" , "helvetica" , sans-serif; white-space: pre-wrap;"><a href="https://3.bp.blogspot.com/-HcmQPCezy78/WtQHMFBLVvI/AAAAAAAAAD0/6IKB2p6lsB0gavaE58WQ0w2Kvw2hv1iKwCPcBGAYYCw/s1600/pack.png" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" data-original-height="292" data-original-width="438" height="133" src="https://3.bp.blogspot.com/-HcmQPCezy78/WtQHMFBLVvI/AAAAAAAAAD0/6IKB2p6lsB0gavaE58WQ0w2Kvw2hv1iKwCPcBGAYYCw/s200/pack.png" width="200" /></a></span></span></div>
<br />
<span style="font-family: "arial" , "helvetica" , sans-serif; white-space: pre-wrap;">Before I started writing my own pack, I first checked the </span><a href="https://github.com/facebook/osquery/blob/master/packs/osx-attacks.conf" style="font-family: arial, helvetica, sans-serif; white-space: pre-wrap;" target="_blank">osx-attacks</a><span style="font-family: "arial" , "helvetica" , sans-serif; white-space: pre-wrap;"> pack in osquery’s github master branch. I didn’t find a signature for the new variant, but I figured this query pack would give me a good template to start from.</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif; white-space: pre-wrap;"><br /></span></div>
<h3 style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: "arial";"><span style="font-family: "arial" , "helvetica" , sans-serif; font-size: small; white-space: pre-wrap;">The Malware</span></span></h3>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: "arial";"><span style="font-family: "arial" , "helvetica" , sans-serif; white-space: pre-wrap;">The next step was to decide what signatures to include in my query pack and to find samples of the malware. The variant reported in early April was attributed by Trend Micro to the activity FireEye dubs Advanced Persistent Threat (APT) 32 / Ocean Lotus. </span></span></div>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: "arial";"><span style="font-family: "arial" , "helvetica" , sans-serif; white-space: pre-wrap;"><br /></span></span></div>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: "arial" , "helvetica" , sans-serif; white-space: pre-wrap;">I am usually pretty skeptical about both the value and accuracy of attribution in open-source “threat intelligence”. Since the goal of this endeavor was simply to learn to build signatures for osquery, I decided to accept the community consensus and focus on putting together an “Ocean Lotus” query pack.</span></div>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span><span style="font-family: "arial" , "helvetica" , sans-serif;">According to a FireEye <a href="https://www.fireeye.com/blog/threat-research/2017/05/cyber-espionage-apt32.html" target="_blank">report</a>:</span></div>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<br />
<blockquote class="tr_bq">
<i><span style="font-family: "arial" , "helvetica" , sans-serif;">Since at least 2014, FireEye has observed APT32 associated ttps and malware targeting foreign corporations with a vested interest in Vietnam’s manufacturing, consumer products, and hospitality sectors...In addition to focused targeting of the private sector with ties to Vietnam, APT32 has also targeted foreign governments, as well as Vietnamese dissidents and journalists since at least 2013...</span></i></blockquote>
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
</div>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: "arial" , "helvetica" , sans-serif;">OK, so now that we know a little bit about APT32 activity, what malware is associated with it? I was able to locate four variants of malware, specifically targeting macOS, publicly attributed to APT32 Ocean Lotus. </span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span><span style="font-family: "arial" , "helvetica" , sans-serif;">The first variant is a trojan Mach-O binary disguised as a Flash update (</span><span style="font-size: xx-small;"><span style="font-family: "arial" , "helvetica" , sans-serif;">SHA-256 </span><a href="https://www.virustotal.com/#/file/12f941f43b5aba416cbccabf71bce2488a7e642b90a3a1cb0e4c75525abb2888/" style="font-family: Arial, Helvetica, sans-serif;" target="_blank">12f941f43b5aba416cbccabf71bce2488a7e642b90a3a1cb0e4c75525abb2888</a></span>). <span style="font-family: "arial" , "helvetica" , sans-serif;">This file in turn writes another binary, named "corevideosd" to disk (</span><span style="font-size: xx-small;"><span style="font-family: "arial" , "helvetica" , sans-serif;">SHA-256 </span><a href="https://www.virustotal.com/#/file/3d974c08c6e376f40118c3c2fa0af87fdb9a6147c877ef0e16adad12ad0ee43a/detection" style="font-family: Arial, Helvetica, sans-serif;" target="_blank">3d974c08c6e376f40118c3c2fa0af87fdb9a6147c877ef0e16adad12ad0ee43a</a></span>). <span style="font-family: "arial" , "helvetica" , sans-serif; font-size: 14.6667px; white-space: pre-wrap;">For more information on activity related to these samples, refer to the </span><a href="https://www.alienvault.com/blogs/labs-research/oceanlotus-for-os-x-an-application-bundle-pretending-to-be-an-adobe-flash-update" style="font-family: Arial, Helvetica, sans-serif; font-size: 14.6667px; white-space: pre-wrap;" target="_blank">Alien Vault</a> post.</div>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<br /></div>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: "arial" , "helvetica" , sans-serif; white-space: pre-wrap;">The second variant is a .zip archive containing a trojan Mach-O binary disguised as a Word document. (</span><span style="font-size: xx-small;"><span style="font-family: "arial" , "helvetica" , sans-serif; white-space: pre-wrap;">SHA-256 </span><a href="https://www.virustotal.com/#/file/b33370167853330704945684c50ce0af6eb27838e1e3f88ea457d2c88a223d8b/detection" style="font-family: Arial, Helvetica, sans-serif; white-space: pre-wrap;" target="_blank">b33370167853330704945684c50ce0af6eb27838e1e3f88ea457d2c88a223d8b</a></span>). <span style="font-family: "arial" , "helvetica" , sans-serif; white-space: pre-wrap;">This variant writes to disk a persistent Mach-O binary, "servicessl" </span>(<span style="font-size: xx-small;">SHA-256</span> <a href="https://www.virustotal.com/#/file/07154b7a45937f2f5a2cda5b701504b179d0304fc653edb2d0672f54796c35f7/%20OceanLotus.B/Noi%20dung%20chi%20tiet.zip" style="font-family: Arial, Helvetica, sans-serif; white-space: pre-wrap;" target="_blank"><span style="font-size: xx-small;">07154b7a45937f2f5a2cda5b701504b179d0304fc653edb2d0672f54796c35f7</span></a>). <span style="font-family: "arial" , "helvetica" , sans-serif; white-space: pre-wrap;">For more info, see the </span><a href="https://researchcenter.paloaltonetworks.com/2017/06/unit42-new-improved-macos-backdoor-oceanlotus/" style="font-family: Arial, Helvetica, sans-serif; white-space: pre-wrap;" target="_blank">Unit 42 </a> post.</div>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<br /></div>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: "arial";"><span style="font-family: "arial" , "helvetica" , sans-serif; white-space: pre-wrap;">The third variant was a trojan Mach-O binary, this time disguised as a PDF (</span><span style="font-family: "arial" , "helvetica" , sans-serif; font-size: xx-small; white-space: pre-wrap;">SHA-256</span></span></div>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: "arial";"><span style="font-family: "arial" , "helvetica" , sans-serif; white-space: pre-wrap;"><a href="https://www.virustotal.com/#/file/f261815905e77eebdb5c4ec06a7acdda7b68644b1f5155049f133be866d8b179/detection" target="_blank"><span style="font-size: xx-small;">f261815905e77eebdb5c4ec06a7acdda7b68644b1f5155049f133be866d8b179</span></a>), w</span></span><span style="font-family: "arial" , "helvetica" , sans-serif; white-space: pre-wrap;">hich in turn drops another persistent Mach-O, "hidd" (</span><span style="font-size: xx-small;"><span style="font-family: "arial" , "helvetica" , sans-serif; white-space: pre-wrap;">SHA-256 </span><a href="https://www.virustotal.com/#/file/a7872fbf84513d1409ce6a13a718a9ff901b3dd92c1671a5ada13f871aaa9975/detection" style="font-family: Arial, Helvetica, sans-serif; white-space: pre-wrap;" target="_blank">a7872fbf84513d1409ce6a13a718a9ff901b3dd92c1671a5ada13f871aaa9975</a></span><span style="font-family: "arial" , "helvetica" , sans-serif; white-space: pre-wrap;">)</span><span style="font-family: "arial" , "helvetica" , sans-serif; font-size: 14.6667px; white-space: pre-wrap;">.</span></div>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<br /></div>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: "arial";"><span style="font-family: "arial" , "helvetica" , sans-serif; white-space: pre-wrap;">Lastly, we have the most recent variant, a malicious Word document (</span><span style="font-family: "arial" , "helvetica" , sans-serif; white-space: pre-wrap;"><span style="font-size: xx-small;">SHA-256<a href="https://www.virustotal.com/#/file/2bb855dc5d845eb5f2466d7186f150c172da737bfd9c7f6bc1804e0b8d20f22a/detection" target="_blank">2bb855dc5d845eb5f2466d7186f150c172da737bfd9c7f6bc1804e0b8d20f22a</a></span>) w</span></span><span style="font-family: "arial" , "helvetica" , sans-serif; white-space: pre-wrap;">hich drops the malicious Mach-O </span><span style="font-family: "arial" , "helvetica" , sans-serif; white-space: pre-wrap;">"spellagentd" (</span><span style="font-family: "arial" , "helvetica" , sans-serif; white-space: pre-wrap;"><span style="font-size: xx-small;">SHA-256 <a href="https://www.virustotal.com/#/file/673ee7a57ba3c5a2384aeb17a66058e59f0a4d0cddc4f01fe32f369f6a845c8f/detection" target="_blank">673ee7a57ba3c5a2384aeb17a66058e59f0a4d0cddc4f01fe32f369f6a845c8f</a></span>). </span><span style="font-family: "arial" , "helvetica" , sans-serif; white-space: pre-wrap;">A detailed analysis was posted by <a href="https://blog.trendmicro.com/trendlabs-security-intelligence/new-macos-backdoor-linked-to-oceanlotus-found/?utm_campaign=shareaholic&utm_medium=twitter&utm_source=socialnetwork" target="_blank">Trend Micro</a>.</span></div>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<br /></div>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: "arial";"><span style="font-family: "arial" , "helvetica" , sans-serif; white-space: pre-wrap;">Luckily, Patrick Wardle has posted samples of most of these variants on his <a href="https://objective-see.com/malware.html" target="_blank">Objective-See blog</a>. </span></span><span style="font-family: "arial" , "helvetica" , sans-serif; white-space: pre-wrap;">For the remaining sample, a hat-tip goes to <a href="https://twitter.com/acalarch" target="_blank">@acalarch</a> for helping me out.</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif; white-space: pre-wrap;"><br /></span><span style="font-family: "arial" , "helvetica" , sans-serif; white-space: pre-wrap;">The query pack can be found on Github, <a href="https://github.com/r3doubt/apple-sauce-in-a-bucket" target="_blank">https://github.com/r3doubt/apple-sauce-in-a-bucket</a>. I hope you'll stick around for Part Two.</span></div>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: 14.6667px; white-space: pre-wrap;"><br /></span></div>
<h2 style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: 14.6667px; white-space: pre-wrap;"><a href="https://blog.r3doubt.io/2018/04/part-two-writing-rules.html" target="_blank">Part Two, Writing the Rules</a></span></h2>
<br />
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
</div>
<br />
<div dir="ltr" style="-webkit-text-stroke-width: 0px; color: black; font-family: "Times New Roman"; font-size: medium; font-style: normal; font-variant-caps: normal; font-variant-ligatures: normal; font-weight: 400; letter-spacing: normal; line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt; orphans: 2; text-align: start; text-decoration-color: initial; text-decoration-style: initial; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px;">
<div style="margin: 0px;">
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: 14.6667px; white-space: pre-wrap;"><br /></span></div>
</div>
R3doubthttp://www.blogger.com/profile/11540283914762258246noreply@blogger.com3