Monday, July 29, 2013

A Travel Story

I travel frequently, and not just to the usual tourist destinations. I've gone to places like Singapore, Japan, Ukraine, and India. This is a story of trying desperately to get home from a recent trip.

Be Aware of You Surroundings

The first sign of trouble was when the roof started leaking. The storm outside had only been raging for half an hour before the two droplets landed on my head. The buckets and signs on the floor captured most of the leaks, but there were numerous unmarked drips. You don't want this kind of water in your hair or food. Always be aware of your surroundings.

I arrived at my gate to find a delayed flight. At least there was now time to eat, since all the good food options were in the international terminal. Casually I trekked over, scarfed down some ethnic food and began meandering back to the gate.

An Ominous Sign

The blaring alarm noises and the flashing emergency lights of the fire alarm told me something was wrong. Neither the airport staff, nor security, nor the airline staff knew what was happening. Just that there wasn't a fire. Maybe. There was no smoke, no firemen, and everyone was calm. Carefully and slowly, I continued towards the gate. Then the power went out.

It was still daytime, and the sunlight and fire alarm (probably on backup power) provided enough lights to get by. The gate was in complete chaos. None of the computers worked, and the staff tried their best to assuage angry passengers. Some were just grumpy, others in tears, but all wanted some answers. There were no answers, no air conditioning, and it was getting hot.

The darkened terminal was like an impoverished refugee camp. Uniformed staff handing out bottles of water to angry men, crying women and screaming children. Mobs of people begged staff for answers. It was dark, hot, loud, and no one knew what was going on. This went on for two hours.

Then the plane arrived, but we couldn't board. The jetbridges were electric and wouldn't extend. Other parts of the airport had power, but the airline couldn't or wouldn't use the working jetbridges. The flight was cancelled, but we weren't rebooked to a new flight because there was no power. We had to call the central reservation office. This was never announced, of course. I happened to overhear another passenger talk with the gate agent.

So I called. After 30 minutes, a man with an accent answered. He sounded legitimately concerned, but all the flights for the day were sold out. I asked if he could rebook me on a competing airline; he typed something on a keyboard and then told me that all flights on all airlines to my destination were sold out. No inventory; The earliest possibility was the next night. Begrudgingly, I agreed. I asked about hotel vouchers. "Of course, the gate agents can print them for you", he replied. The power was still out.


I wasn't about to spend another night in this place. There was still one flight on a competing airline, and it was leaving soon. Sure I was told there were no seats, but one can't blindly trust a company to do something that lowers profit. And besides, maybe someone wouldn't show up. The competing airline was naturally in the furthest possible terminal from where I was. It was a long walk, but they probably power.

Turns out there were seats on the flight. There would be a cost, but I would get home, today. Life was also better in this part of the airport. There were no leaks in the roof and the air conditioning was on. Nobody was crying. I sat down near the new gate.

There was a current flight there, delayed indefinitely. Must be weather, I figured. That assumption was shattered when I heard two airline employees talking next to the gate entrance. Turns out the plane needed fuel. They sent for a fuel truck, but it arrived without fuel. For the past half hour they were trying to find either fuel, a new fuel truck, or whomever got them into the boondoggle. No one was answering. After 15 more minutes, they found a new truck. Two hours later my plane arrived.

Don't Count Your Chickens...

As we boarded, I realized there were plenty of seats. The flight was only half full. So much for "no seats available." As we taxied to the runway, I was thankful to finally be out of this wretched place.

My enthusiasm was premature. Literally as we were next in line to depart, our plane was directed to stop. After an hour on the tarmac, the full story unfolded. The original flight path had unexpected weather. We had an approved alternate flight path, but in the time it took us to taxi (about 30 min in the rain at this airport), the alternate flight path also had unexpected weather. It was 30 more minutes before we moved again.

Departure At Last

This time, it was for real. Our wheels touched off the ground and we ascended into the stormy sky. The plane shook violently as we passed through the rain, wind, and lightning. I clutched my seat and thought about this abhorrent place: the fire alarm, the crying passengers, the hot, dark, sweaty terminal, helpless staff, the leaky roof and fuel-less fuel trucks. I just wanted to go home and to never again fly into Philadelphia International Airport.

Sunday, July 21, 2013

Readability Improvements

In preparation for new content relating to my Blackhat 2013 Arsenal presentation, I made some readability improvements to the blog and to in general.

The layout is wider and the font size for text is now 16 pixels.

Please let me know what you think.

Friday, July 12, 2013

Git Fails On Large Files

Turns out git fails spectacularly when working with large files. I was surprised, but the behavior is pretty well documented. In typical git fashion, there is an obscure error message and an equally obscure command to fix it.

The Problem

A real-life example (with repository names changed):

artem@MBP:~/git$ git clone git@gitlab:has_a_large_file.git
Cloning into 'has_a_large_file'...
Identity added: /Users/artem/.ssh/devkey (/Users/artem/.ssh/devkey)
remote: Counting objects: 6, done.
error: git upload-pack: git-pack-objects died with error.
fatal: git upload-pack: aborting due to possible repository corruption on the remote side.
remote: Compressing objects: 100% (5/5), done.
remote: fatal: Out of memory, malloc failed (tried to allocate 1857915877 bytes)
remote: aborting due to possible repository corruption on the remote side.
fatal: early EOF
fatal: index-pack failed

I pushed the large file without issues, but couldn't pull it again because the remote was dying. The astute reader will notice the remote was running gitlab. The push also broke the gitlab web interface for the repository.

From my Googling, the problem is that the remote side is running out of memory when compressing a large file (read more about git packfiles here). Judging by the error, git attempts to malloc(size_of_large_file) and the malloc fails.

This situation raises conundrums that may only be answered by Master Git:
  • Why was I able to push a large file, but not pull it?
  • Why would one malloc(size_of_large_file) ?
  • What happens when you push a >4Gb file to a 32-bit remote?

I was curious enough about the last one to look at the code: it will likely die gracefully (see line 49 of wrapper.c). Integer overflow likely avoided; would need to read more code much more carefully to be sure.

The Solution

In theory, the solution is to re-pack the remote with a smaller pack size limit. That requires ssh access to the remote repository, which I don't have. So the following fix is untested, and taken from The obscure command in question (must be run on the remote):

git repack -a -f -d

Of course, repacking the remote but having non-repacked local repositories around may cause other problems.

Just For Fun

Here is another large file fail:

artem@MBP:~/temp/largerandomfile$ dd if=/dev/urandom of=./random_big_file bs=4096 count=1048577
1048577+0 records in
1048577+0 records out
4294971392 bytes transferred in 437.836959 secs (9809522 bytes/sec)

artem@MBP:~/temp/largerandomfile$ git add random_big_file
artem@MBP:~/temp/largerandomfile$ git commit -m "Added a big random file"
[master 377db57] Added a big random file
1 file changed, 0 insertions(+), 0 deletions(-)
create mode 100644 random_big_file

artem@MBP:~/temp/largerandomfile$ git push origin master
Counting objects: 4, done.
Delta compression using up to 2 threads.
Compressing objects: 100% (2/2), done.
error: RPC failed; result=22, HTTP code = 413 KiB/s
fatal: The remote end hung up unexpectedly
Writing objects: 100% (3/3), 4.00 GiB | 18.74 MiB/s, done.
Total 3 (delta 0), reused 1 (delta 0)
fatal: recursion detected in die handler
Everything up-to-date

Everything up-to-date, indeed.