Saturday, November 9, 2013

A Bit Flip That Killed?

During my bitsquatting research I was amazed how many critical RAM chips in a typical PC lack error correcting memory.

It turns out that ECC is missing from an even more critical device: cars.

Details from the recent Toyota civil settlement show that the drive-by-wire control of Toyota cars was lacking error detection and correcting RAM.

From EDN.com:
Although the investigation focused almost entirely on software, there is at least one HW factor: Toyota claimed the 2005 Camry's main CPU had error detecting and correcting (EDAC) RAM. It didn't. EDAC, or at least parity RAM, is relatively easy and low-cost insurance for safety-critical systems.

I can't fathom why that would ever be the case. The amount of RAM required is relatively small, and the extra cost is inconsequential to the total cost of a car. Oh, and the software runs next to a car engine.

"We've demonstrated how as little as a single bit flip can cause the driver to lose control of the engine speed in real cars due to software malfunction that is not reliably detected by any fail-safe," Michael Barr, CTO and co-founder of Barr Group, told us in an exclusive interview. Barr served as an expert witness in this case.

Drive-by-wire systems aren't the only critical control systems susceptible to bit-errors. There is some speculation that a bit-error caused a sudden altitude drop in a Qantas A330. Amazingly, airplane software systems did not have to consider single or multiple bit errors until 2010 (see page 222) to achieve certification.

Monday, October 21, 2013

Git and Bit Errors

Finally, a topic to unite my two most popular blog posts: git failures and bitsquatting.

A friend recently pointed me to an amazingly detailed investigation of a corrupted git repository. The cause of the corruption? A single bit flip. To quote the source:

As for the corruption itself, I was lucky that it was indeed a single
byte. In fact, it turned out to be a single bit. The byte 0xc7 was
corrupted to 0xc5. So presumably it was caused by faulty hardware, or a
cosmic ray.
And the aborted attempt to look at the inflated output to see what was
wrong? I could have looked forever and never found it. Here's the diff
between what the corrupted data inflates to, versus the real data:
  -       cp = strtok (arg, "+");
  +       cp = strtok (arg, ".");


It is quite amazing to see evidence of a bit error resulting in a perfectly innocuous, syntactically valid and yet completely erroneous change in a real program and a real codebase.

How many times does this happen without anyone noticing?


Friday, September 13, 2013

Bitsquatting at DEFCON21 and More

I was very excited to see that several researchers are investigating bitsquatting and writing about it. There were two presentations about bitsquatting at DEFCON 21, a presentation at ICANN 47, and a research paper presented at WWW2013.

Jaeson Schultz - DEFCON 21 - Examining the Bitsquatting Attack Surface
Jaeson presented some excellent ways to exploit bitsquatting that I did not think of -- such as using bitsquats in URL delimeters to target otherwise unexploitable domains. As an example taken from the paper, ecampus.phoenix.edu can become ecampus.ph/enix.edu/.

Additionally Jaeson presents a great mitigation that can be implemented at the local level -- Response Policy Zones. From the paper:
An RPZ is a local zone file which allows the DNS resolver to respond to specific DNS requests by saying that the domain name does not exist (NXDOMAIN), or redirecting the user
to a walled garden, or other possibilities. To mitigate the effects of single bit errors for users of a DNS resolver the resolver administrator can create a Response Policy Zone that protects against bitsquats of frequently resolved, or internal-only domain names.  
 

Robert Stucke - DEFCON 21 - DNS Has Been Found To Be Hazardous To Your Health
Robert demonstrated some new vectors for bitsquatting, such as web applications and hosted email providers. Speifically, he bitsquatted gstatic.com (a site that serves static content for Google properties). Not only was he able to return arbitrary content to people using Google's search services, he could also affect web applications, such as feed readers, that rely on correct resolution of gstatic.com. Robert also bitsquatted psmtp.com, a hosted email provider. This allowed him to potentially receive other people's email.

Nigel Roberts - ICANN 47 - Bitsquatting
Nigel (who runs .gg) presented about bitsquatting to ICANN. Hopefully this will result in more research at the ccTLD level. 

Nick Nikiforakis, et al. - WWW2013 - Bitsquatting: Exploiting Bit-flips for Fun, or Profit?
Nick and his coauthors did a measurement study about the prevalence of bitsquatting and what content appears on bitsquatted domains. They identified several that are used for adverstising, affiliate programs, and malware distribution.  There is also a great graph in the paper where you can see a huge spike in bitsquat domain registration after my Blackhat presentation :).

Thursday, August 1, 2013

Introducing Binfuzz.js

Tomorrow morning I will be giving a demonstration of Binfuzz.js at Blackhat Arsenal 2013. Please stop by the Arsenal area from 10:00 - 12:30. The slides are already available on the Blackhat website.

The Binfuzz.js page on dinaburg.org is now live, and all the code is uploaded to Github.

What is Binfuzz.js?


Binfuzz.js is a library for fuzzing structured binary data in JavaScript. Structured binary data is data that can be easily represented by one or more C structures. Binfuzz.js uses the definition of a structure to create instances of the structure with invalid or edge-case values. Supported structure features include nested structures, counted arrays, file offset fields, and length fields. The live example uses Binfuzz.js to generate Windows ICO files (a surprisingly complex format) to stress your browser's icon parsing and display code.

Features


Binfuzz.js includes support for:

  • Several predefined elementary types: Int8, Int16, Int32 and Blob.
  • Nested structures
  • Arrays
  • Counter Fields (e.g. field A = number of elements in Array B)
  • Length Fields (e.g. field A = length of Blob B)
  • File Offsets (e.g. field A = how far from the start of the file is Blob B?
  • Custom population functions (e.g. field A = fieldB.length + fieldC.length)

The ICO fuzzing example includes uses of all of these because I needed them to implement ICO file generation.

Combinatorics


Binfuzz.js calculates the total number of combinations based on how many possible combinations there are for each individual field. It is then possible to generate a structured data instance corresponding to a specific combination number. It is not necessary to generate prior combinations. This way random combinations can be selected when fuzzing run time is limited.

Why?


The best way to learn is by doing, and I wanted to learn JavaScript. So I decided to create an ICO file fuzzer in JavaScript. I chose ICO files because of favicon.ico, a file browsers automatically request when navigating to a new page. After starting the project, I realized I got a lot more than I bargained for. Icons are a surprisingly complex format that has evolved over time. There are several images in one file, each image has corresponding metadata, there are internal fields that refer to offsets in the file, and the size of the icon data for each image depends the metadata. All of these interlinked reationships need to be described and processed by Binfuzz.js.

Monday, July 29, 2013

A Travel Story

I travel frequently, and not just to the usual tourist destinations. I've gone to places like Singapore, Japan, Ukraine, and India. This is a story of trying desperately to get home from a recent trip.

Be Aware of You Surroundings


The first sign of trouble was when the roof started leaking. The storm outside had only been raging for half an hour before the two droplets landed on my head. The buckets and signs on the floor captured most of the leaks, but there were numerous unmarked drips. You don't want this kind of water in your hair or food. Always be aware of your surroundings.

I arrived at my gate to find a delayed flight. At least there was now time to eat, since all the good food options were in the international terminal. Casually I trekked over, scarfed down some ethnic food and began meandering back to the gate.

An Ominous Sign


The blaring alarm noises and the flashing emergency lights of the fire alarm told me something was wrong. Neither the airport staff, nor security, nor the airline staff knew what was happening. Just that there wasn't a fire. Maybe. There was no smoke, no firemen, and everyone was calm. Carefully and slowly, I continued towards the gate. Then the power went out.

It was still daytime, and the sunlight and fire alarm (probably on backup power) provided enough lights to get by. The gate was in complete chaos. None of the computers worked, and the staff tried their best to assuage angry passengers. Some were just grumpy, others in tears, but all wanted some answers. There were no answers, no air conditioning, and it was getting hot.

The darkened terminal was like an impoverished refugee camp. Uniformed staff handing out bottles of water to angry men, crying women and screaming children. Mobs of people begged staff for answers. It was dark, hot, loud, and no one knew what was going on. This went on for two hours.

Then the plane arrived, but we couldn't board. The jetbridges were electric and wouldn't extend. Other parts of the airport had power, but the airline couldn't or wouldn't use the working jetbridges. The flight was cancelled, but we weren't rebooked to a new flight because there was no power. We had to call the central reservation office. This was never announced, of course. I happened to overhear another passenger talk with the gate agent.

So I called. After 30 minutes, a man with an accent answered. He sounded legitimately concerned, but all the flights for the day were sold out. I asked if he could rebook me on a competing airline; he typed something on a keyboard and then told me that all flights on all airlines to my destination were sold out. No inventory; The earliest possibility was the next night. Begrudgingly, I agreed. I asked about hotel vouchers. "Of course, the gate agents can print them for you", he replied. The power was still out.

Perseverance


I wasn't about to spend another night in this place. There was still one flight on a competing airline, and it was leaving soon. Sure I was told there were no seats, but one can't blindly trust a company to do something that lowers profit. And besides, maybe someone wouldn't show up. The competing airline was naturally in the furthest possible terminal from where I was. It was a long walk, but they probably power.

Turns out there were seats on the flight. There would be a cost, but I would get home, today. Life was also better in this part of the airport. There were no leaks in the roof and the air conditioning was on. Nobody was crying. I sat down near the new gate.

There was a current flight there, delayed indefinitely. Must be weather, I figured. That assumption was shattered when I heard two airline employees talking next to the gate entrance. Turns out the plane needed fuel. They sent for a fuel truck, but it arrived without fuel. For the past half hour they were trying to find either fuel, a new fuel truck, or whomever got them into the boondoggle. No one was answering. After 15 more minutes, they found a new truck. Two hours later my plane arrived.

Don't Count Your Chickens...


As we boarded, I realized there were plenty of seats. The flight was only half full. So much for "no seats available." As we taxied to the runway, I was thankful to finally be out of this wretched place.

My enthusiasm was premature. Literally as we were next in line to depart, our plane was directed to stop. After an hour on the tarmac, the full story unfolded. The original flight path had unexpected weather. We had an approved alternate flight path, but in the time it took us to taxi (about 30 min in the rain at this airport), the alternate flight path also had unexpected weather. It was 30 more minutes before we moved again.

Departure At Last


This time, it was for real. Our wheels touched off the ground and we ascended into the stormy sky. The plane shook violently as we passed through the rain, wind, and lightning. I clutched my seat and thought about this abhorrent place: the fire alarm, the crying passengers, the hot, dark, sweaty terminal, helpless staff, the leaky roof and fuel-less fuel trucks. I just wanted to go home and to never again fly into Philadelphia International Airport.

Sunday, July 21, 2013

Readability Improvements

In preparation for new content relating to my Blackhat 2013 Arsenal presentation, I made some readability improvements to the blog and to dinaburg.org in general.

The layout is wider and the font size for text is now 16 pixels.

Please let me know what you think.

Friday, July 12, 2013

Git Fails On Large Files

Turns out git fails spectacularly when working with large files. I was surprised, but the behavior is pretty well documented. In typical git fashion, there is an obscure error message and an equally obscure command to fix it.

The Problem


A real-life example (with repository names changed):

artem@MBP:~/git$ git clone git@gitlab:has_a_large_file.git
Cloning into 'has_a_large_file'...
Identity added: /Users/artem/.ssh/devkey (/Users/artem/.ssh/devkey)
remote: Counting objects: 6, done.
error: git upload-pack: git-pack-objects died with error.
fatal: git upload-pack: aborting due to possible repository corruption on the remote side.
remote: Compressing objects: 100% (5/5), done.
remote: fatal: Out of memory, malloc failed (tried to allocate 1857915877 bytes)
remote: aborting due to possible repository corruption on the remote side.
fatal: early EOF
fatal: index-pack failed

I pushed the large file without issues, but couldn't pull it again because the remote was dying. The astute reader will notice the remote was running gitlab. The push also broke the gitlab web interface for the repository.

From my Googling, the problem is that the remote side is running out of memory when compressing a large file (read more about git packfiles here). Judging by the error, git attempts to malloc(size_of_large_file) and the malloc fails.

This situation raises conundrums that may only be answered by Master Git:
  • Why was I able to push a large file, but not pull it?
  • Why would one malloc(size_of_large_file) ?
  • What happens when you push a >4Gb file to a 32-bit remote?

I was curious enough about the last one to look at the code: it will likely die gracefully (see line 49 of wrapper.c). Integer overflow likely avoided; would need to read more code much more carefully to be sure.

The Solution


In theory, the solution is to re-pack the remote with a smaller pack size limit. That requires ssh access to the remote repository, which I don't have. So the following fix is untested, and taken from http://www.kevinblake.co.uk/development/git-repack/. The obscure command in question (must be run on the remote):

git repack -a -f -d

Of course, repacking the remote but having non-repacked local repositories around may cause other problems.

Just For Fun


Here is another large file fail:

artem@MBP:~/temp/largerandomfile$ dd if=/dev/urandom of=./random_big_file bs=4096 count=1048577
1048577+0 records in
1048577+0 records out
4294971392 bytes transferred in 437.836959 secs (9809522 bytes/sec)

artem@MBP:~/temp/largerandomfile$ git add random_big_file
artem@MBP:~/temp/largerandomfile$ git commit -m "Added a big random file"
[master 377db57] Added a big random file
1 file changed, 0 insertions(+), 0 deletions(-)
create mode 100644 random_big_file

artem@MBP:~/temp/largerandomfile$ git push origin master
Counting objects: 4, done.
Delta compression using up to 2 threads.
Compressing objects: 100% (2/2), done.
error: RPC failed; result=22, HTTP code = 413 KiB/s
fatal: The remote end hung up unexpectedly
Writing objects: 100% (3/3), 4.00 GiB | 18.74 MiB/s, done.
Total 3 (delta 0), reused 1 (delta 0)
fatal: recursion detected in die handler
Everything up-to-date

Everything up-to-date, indeed.