I Love Open Plan Offices

It seems like every month there is a story about how open plan offices suck and how private offices are ideal for software developers. Open plan offices make workers sick, developers hate them, and well, they’re “devised by Satan in the deepest caverns of hell.” The comments from tech workers are similarly negative (see here, here, here, and here). People who would know and several studies have said that private offices make the most productive developers.

Some people's view of an open plan office. The open environment ensures close collaboration and teamwork.

Well, I'm a software developer, and I love open plan offices and hate private offices. I’ve worked in open plan offices, shared offices, private offices, and from home. Open plan offices are by far my favorite.

It will sound cliche, but open plans work for me because of the increased collaboration. Discussing work over email, phone, and chat is simply not the same. Developers work in teams, and it's much easier to be effective within easy conversational range. There is also a social aspect that builds team cohesion. Finally, being in the same environment makes people more accountable. It's much easier to stay on task when you see other people working hard towards the same goal -- and much harder to get distracted and browse reddit.

Conversely, private offices don’t work for me because they are stifling and isolating. My private office experience was sitting alone in a room for 8 hours a day with almost no human contact. Sure, maybe I would catch people going to lunch, or someone would stop by to ask a question, but that was rare. The barriers to communication were simply too high. With private offices, communication required getting up, leaving your office, walking around, and knocking on someone’s door. When people had a question, they would first search, then email, then call, and finally walk over to talk in person. And this is all for team members on the same floor. This cycle could take hours. If everyone was in the same room, it would take seconds. And if the person you needed was on a different floor, well, you'd better hope to meet at lunch. The one floor elevator ride was simply too insurmountable, except in the most dire of circumstances.

My view of private offices. No distractions for 23 hours a day, comes with a private bathroom, and even has a window.

For some people, open plan offices really are terrible. The solution isn’t to force everyone into private offices -- they're also terrible. Just like there is no best Pepsi and no best spaghetti sauce, there is no best office space. I’ve met people who love private offices, who love open offices, who will only work from home, and people who think the ideal environment is a shared office. They are all correct. The answer is to stop forcing a single office layout on all employees.

There is no best office, just like there is no best pasta sauce.

When so many companies make it their mantra to hire the best people, why put them into unproductive environments (or make them to move to San Francisco)? Keep some larger offices to share, leave some as private offices, and make a common area an open floor plan. And if people want to, let them work form home. To get and keep the best people, accommodate what makes them productive. It’ll be easier to recruit, and productivity will go up.

Advertising with Google: It Sucks

Advertising with Google as a small business is much different than using Google as a consumer. Paying Google for ads and calling Google support is a lot like paying Comcast for cable and calling Comcast support. 

The events described in this post happened about a year ago. I finally decided to make this post for two reasons: to show that Google is exactly like every other large faceless near-monopoly and to show my disappointment with how Google handles paying customers. 

Before this experience, I thought the parts of Google that make money must be amazing. Look at Gmail, Google Docs Drive, Google Calendar, Blogger and even Android. They’re great, and they’re all free. Google makes these services to capture you, the product. If the free stuff is great, then the parts of Google that face advertisers (the customer) and take in billions of dollars in revenue have to be absolutely amazing. Sadly, they’re not.

Lets start with some background. A year ago I had a great idea: start a website that would sort pocket knives by legality to carry. You would pick a city and see knives that are legal to carry in that jurisdiction. The list of knives have Amazon referral links. Everyone would benefit: the customers would stay legal, and I would collect a sales commission. 

The story of the site is long and deserves its own blog post, but in summary... creating product sales sites is hard. The site I made, legalknifeshop.com, is still up. Go and take a look, and if you live in Chicago, order a knife. Also, I know the site has many problems. I’ve been too busy to work on it. If you see something wrong, I’m probably aware of it but don’t have time to fix anything.

But this post is about advertising, so lets get back to the story. The first order of business after making a site is to advertise. There is only one choice in online advertising: Google. I signed up for an AdWords account, selected keywords, and paid money. Legalknifeshop was approved and the ads started. Time to wait for the sales to roll in!  

The ads ran for two days. Then I got the first email. 

Urgent Warning - Your AdWords Account May Get Suspended

That's not good. It turns out that after being approved, legalknifeshop was unapproved, despite no content changes.

The site was in violation of the weapons policy that prohibits dangerous knives. The violation was silly: legalknifeshop only sells explicitly legal knives, and knives legal to carry in Chicago (the only supported location) must have <2.5" blade. Boy scouts handle longer and more dangerous knives. Certainly someone at Google could understand and reinstate the ads. 

I called Google support. After 4 different menu prompts there was a human, to whom I explained the situation. 

She put me on hold for 15 minutes, came back and said that she “had no idea what was going on,” but the site needed to be escalated back to review”. “But what about the previous review,” I asked. “I don’t work in that department,” she answered. 

This was the first sign of what was to come.

Then another email. 

Your AdWords account: Ads not running due to AdWords Advertising Policies.

Maybe the review department wasn’t convinced? Having heard nothing else, I sent a support an email asking why legalknifeshop violated the weapons policy.

Call 1-866-2-GOOGLE* for free expert help reviewing your AdWords ads 

That was the title of the next email. It continued: "Having reviewed your AdWords account XXXXXXXXXX, we can see that so far 549 potential customers have seen your ad, and 4 of them clicked on your ad to view your website. We can help you attract more potential customers to your website or answer any other questions you may have".

Maybe they could start by reinstating my ads?  Really, Google? What world do we live in when a Big Data company like Google can't link their suspended account database to their promotional emails?

I called support again and spent 20 minutes on hold. The reply was "oh, I can't really see what's going on, you'll have to contact the people who you were contacting before". This is when I knew I was in trouble. There isn’t even a coherent customer records management or issue tracking.

During the next few weeks I called, used the online customer chat, and sent emails. Finally, after persevering and explaining that legalknifeshop sold no dangerous knives, someone at Google admitted that no, legalknifeshop does not violate the weapons policy. Victory at last! 

The sweet taste of victory was not to last.

The next day, another suspension email. Now, my site was in violation of the bridge page policy. A policy that was not mentioned once during the many prior discussions with Google support. I appealed again, but it was not to be.

It turns out that legalknifeshop doesn't provide enough value to end users. To provide enough value to advertise with Google requires the following changes:

• Provide listings for more than one city (doable)
and one of:

• Use more than Amazon as a referral partner. (pretty much impossible)
or

• Sell the knives myself. (impossible)

During my new week of appeals, I received the following promotions from Google:

Reach the right customers by adding negative keywords to your AdWords ad 'Knives Legal In Chicago'

Your AdWords ads have stopped running. Talk to Google to get help. Remember to call 1-866-2-GOOGLE* for a free review of your AdWords ads with a Google expert

There is no way I could meet the bridge page requirements. Google won, they refused to take my money. I really tried to pay, but they just wouldn’t take it.

Two days later, there was one last email:

AdWords Tune-up: Make your 'LegalKnifeShop #1' campaign ad stand out with a longer headline

That was the last straw. Out of pure spite, I advertised with Bing. But that's a story for another blog post.

What Happens When Your Phone Falls into the Ocean

Short answer: It'll stop working.

But have you wondered, why? What happens to the insides of the phone?

We've all heard that salt water conducts electricity and "fries" your electronics, but what does that mean? Will the phone rust? Will the battery melt? Today we get to find out!

In this blog post we'll look inside my Black 16GB iPhone 5 (yes, that's an affiliate link) that took a swim in the Pacific. Where possible, iFixit's iPhone 5 teardown pictures will model what iPhone 5 internals should look like.

The Initial Opening

This is how an iPhone 5 looks like after swimming in the Pacific.

This is how iPhone 5 innards should look like, courtsey of iFixit.

Right away, there's a huge difference. There ocean phone is full of fine sand, salt stains, and a giant rust spot. If you remember chemistry class, then the rust spot makes sense: rust is an electrical process. The biggest rust spot will be at the anode of the battery leads, which is exactly where it is.

The Screen Assembly and Mainboard

The salted, rusty screen assembly. The rust feels like its simply rubbed off from the other part.

The salted, rusty mainboard and battery. The battery is surprisingly intact.

How an iPhone 5 screen assembly and mainboard should look like. Notice the distinct lack of salt, rust, and sand. Courtesy of iFixit.

Once again, the salt stains and rust is the big difference. Surprisingly, the lithium-ion battery is perfectly intact, without holes or burn damage. Why surprising? Lithium and water tend to react vigorously.

Battery Closeup

The battery leads are extra rusty.

An unsalted battery, courtesy of iFixit.

Other than the rusty leads, the battery seems fine from the outside. The phone would not power on, but I am not sure if its related to the battery or other electronics failures. Probably both.

Mainboard and Chassis Closeup

There's a large salt deposit under the mainboard. The salt water must have pooled there as the phone dried.

A pristine mainboard, courtesy of iFixit.

Getting to this part was a challenge: some of the screws were so corroded and rusted that unscrewing them stripped the grooves used for unscrewing. I had to resort to force and some prying, which didn't matter since the phone was already broken.

A Zoom on the Mainboard

A zoom on the corroded mainboard.

The front of the camera and mainboard.

A pristine mainboard, courtesy of iFixit.

Almost every connector is rusted or otherwise corroded. This is one of the main reason everything stops working: the small connectors corrode and the mainboard components can't make electrical contact.

Everything

All the rusted components in one glorious photo.

All the pristine components, courtesy of iFixit.

The obligatory "all components in one photo" shot. Mine isn't as nice as iFixit's, but the other differences should be obvious. The one interesting thing to note is the serious corrosion on the SIM card.

Conclusion

If your phone falls into the ocean, its going to have a bad time. Don't let your phone fall into the ocean.

Bjarne Stroustrup on the Past and Future of C++ (Including Long Template Errors)

Someone on IRC pointed me to Bjarne Stroustrup's talk on C++11 and C++14 at Microsoft's Going Native 2013. If you work with C++ and haven't seen Bjarne's talk yet, go watch it now.

Stop reading this and go. I'll wait. 

An hour into the talk, Bjarne starts discussing templates, long error messages, and the tradeoffs that were made during the design of C++. The long error messages were a conscious decision to preserve performance and expressiveness with the computing power available back in the mid 1980s.

It amazed me that Bjarne admits template error messages are a huge debacle, and he has been working for 20 years to fix the problem. The solution is near: C++14 concepts will finally allow for sane template errors. Messages like "Member must be CopyAssignable" will be possible, and hopefully normal. This isn't just theory: there is an experimental branch of GCC that supports concepts right now.

Other parts of the talk are fascinating in their own right and have given me a lot more respect for C++ and Bjarne Stroustrup. The man could have rested after creating the original C++ spec and compiler, but he has been working for 20 years to improve the language. That dedication has made C++11 much better than C++98.

Bjarne also brings up a good point: many people who dislike C++ are using it the wrong way. The language should only be used when you need a performance and lightweight abstraction at the same time. If you don't care about performance or you need high-level abstraction, C++ is the wrong tool for the job.

The talk has a lot more interesting content. If you haven't watched it yet, go now.

C++11: better, but still frustrating

Update: jduck pointed out that the before/after code snippets were identical. Oops. Now fixed.

I'd previously given up on C++ due to the many small frustrations: incomprehensible error messages, silly parsing issues (e.g. '>>'), rules to avoid subtle errors, and many other small frustrations that soured me on the language. That was back in the days of C++98 and C++03.

The language has evolved, and recently I found myself working on a project written in C++11. So far my experience has been better, but still frustrating.

A Motivating Example

I'll start with a real example. The project created a lot of Foo objects that were passed by reference to numerous functions. I needed to keep a collection of every Foo object that was passed to a specific function.

My first thought was, "I know, I'll create a vector of Foo&". This thought is simple, elegant, and of course, wrong.

A vector of references isn't possible because references can't be reassigned. That is, references[0] = foo; would update the referenced object, not the zeroth entry of the references vector. More technically, references are not CopyAssignable, a requirement for members of containers.

Errors Galore

But how would someone new to C++ know this? What do compilers say when making a vector of references? Lets find out by compiling this small (and wrong) program.

#include <vector>
#include <iostream>

int main(int argc, const char* argv[])
{
    int a = 1;
    std::vector<int&> test = {a};
    std::cout << "a: " << test[0] << std::endl;
    return 0;
}

Here are the results for Clang, GCC and MSVC:

Compiler	Error List	Error Count
Clang	rextester.com/JMFGP72087	158 lines
GCC	rextester.com/HGKFIT84222	187 lines
MSVC	rextester.com/UXPG39365	107 lines

In classic C++ style, the error messages are hundreds of error lines from obscure library implementation code. They give no indication of what is wrong, and no indication of the solution. I pity someone who doesn't have C++ experience trying to figure out what is wrong with their code. Pretty much any error would be more helpful, even an obscure message like "Member must be CopyAssignable" -- as long as it pointed out the correct line of code.

The Fix

For reference, the corrected program is:

#include <vector>
#include <iostream>
#include <functional>

int main(int argc, const char* argv[])
{
    int a = 1;
    std::vector<std::reference_wrapper<int>> test = {a};
    std::cout << "a: " << test[0] << std::endl;
    return 0;
}

The fix is to use the std::reference_wrapper utility function when making a container of references.

Conclusion

There's definitely upsides: the '>>' parse has finally been fixed. Classes can now be initialized with initializer lists. There is type inference via 'auto'. For-each style loops exist.

C++11 is a great improvement over C++03, but its still frustrating: the obvious solution (like containers of references) is wrong in subtle ways, and compilers still generate hundreds of obscure error messages for a one-character typo.

Do Not Reply Addresses Suck

If you send emails from do not reply addresses, I hate you.
Your customers hate you.
People you've never met also hate you.

Why  you should stop using no reply addresses:

• Something will go wrong and your customers can't tell you.
• You will send email to people who aren't your customers. They will have no way to ask you to stop.

Stop hating your current and future customers. Stop using no reply addresses. Only send email from monitored email addresses.

And one more thing...

If you create accounts without validating email addresses, I doubly hate you... and from now on I will be naming and shaming.

The Do Not Reply Hall of Shame

Ask.fm

Ask.fm lets people register without validating their email. The many follow-up emails are all from noreply@ask.fm. Ask.fm, please stop.

As you can guess, nothing was ever done. Ask.fm, I hate you.

DataViz.com

DataViz lets people buy software without verifying their email. Wish I could reply and tell them, but the email is from do-not-reply@dataviz.com.

My name is not Artem Dubov and I did not buy this software.

Direct Download Link For Adobe Flash Player

Are you tired of seeing this?

"Error: Unable to proceed with the installation". Thats bad. But there's a green checkmark. Thats good? 

Do you find Adobe's toubleshooting page completely useless?

 

Then use this handy direct link and bypass Adobe's broken installer. As a bonus, it wont try to trick you into installing Lightroom or other unwanted products.

http://fpdownload.macromedia.com/get/flashplayer/pdc/12.0.0.44/install_flash_player_osx.dmg

Update as of November 2nd, 2014:

Jerry Leichter very helpfully pointed out that Adobe maintains an official redistributable version of Flash player that doesn't use the awful installer. You can get it here:

https://www.adobe.com/products/flashplayer/distribution3.html

Stupid IDN Tricks: Unicode Combining Characters (or http://░͇͇͇͇͇͇͇͇͇͇͇͇͇͇͇͇͇͇͇͇͇͇͇͇͇͇͇͇.ws)

Nov 3, 2014: The domains mentioned in this article are expiring and I'm not renewing them. All links have been redirected to the archive.org mirror of the original site.

Safari will display Unicode combining diacritical marks in the URL bar (try going to  http://░͇͇͇͇͇͇͇͇͇͇͇͇͇͇͇͇͇͇͇͇͇͇͇͇͇͇͇͇.ws). It is possible to register domains with these marks. Some of these domains will look much like legitimate domains (e.g. apple.com vs. apple͢.com). This is probably not good.

Internationalized Domain Names (IDN)

DNS was only designed with 7-bit unsigned ASCII in mind. However, not everyone in the world speaks English, and they really want to type domains in their own language. So there is a terrible hack to map Unicode characters to 7-bit unsigned ASCII, called IDNA.

Homograph Attacks

Hopefully everyone has heard of homograph attacks using internationalized domain names. If not, here is a recap (taken from the Chrome wiki):

... different characters from different languages can look very similar, and this can make phishing attacks possible. For example, the Latin "a" looks a lot like the Cyrillic "а", so someone could register http://ebаy.com (http://xn--eby-7cd.com/), which would easily be mistaken for http://ebay.com. This is called a homograph attack.

Defenses Against Homograph Attacks

There are multilayered solutions to the homograph attack:

• Browser characters blacklists. These prevent you from registering characters that look like '/', and so on.
• IDN character display rules (see: Firefox, Chrome). These rules restrict non-ASCII domain names to only those languages specifically configured by the user, and prevent display of mixed-language domains. For instance, if your have a Chinese installation of Windows then Chinese characters will be displayed for Chinese IDNs.
• Registrar restrictions. Registrars will prevent you from registering a domain that combines  more than one language. So you can't register a name that is half English and half Russian, for instance.

Another Attempt

So how do we explain http://░͇͇͇͇͇͇͇͇͇͇͇͇͇͇͇͇͇͇͇͇͇͇͇͇͇͇͇͇.ws?

Defeating Registrar Restrictions

Registrars prohibit combining languages in domain names. But there are characters that aren't in any language. The most interesting of these are Unicode Combining Diacritical Marks. These unicode code points will modify the glyph right before them, instead of adding a new character. For example, the letter A when combined with U+0x332 will become: A̲.

But will these characters display in browsers?

Chrome: No :(

Firefox: No :(

Safari: Yes :)

Impact

Someone could register apple͢.com and it would display in Safari as:

This is not good.

I Hate (General Purpose) Computers

I hate computers. More specifically, general purpose computers. They cause me many hours of frustration, mostly due to malware.

Most people don't need or want the freedom to run the malware of their choice. They need a nice computing appliance with a well-designed GUI that "just works". General computing is important, it just shouldn't be the default option.

I propose appliance-default computers with a big red FTC mandated 'general computing' switch. It would save millions of hours in security and support costs, while protecting consumer freedom.

Anger and Frustration

It all started over Thanksgiving. Once again, it was time to answer family computer questions.

My father asked, "How can I be absolutely sure I don't get infected with CryptoLocker?". He was very concerned. It was on the news, and there was a warning email at work.

Unfortunately, there was nothing more I could tell him. He already does everything right, and could still be infected with CryptoLocker. There's nothing I can do: he has a computer and it can run malware. Sure there are precautions, but these are mostly useless.

Malware Precautions: Largely Useless

These (largely useless) precautions to avoid "being a victim" just happened to be on the news as I was drafting this blog post. The news report was about the recent social media password leak.

The precautions:

• Install Anti-Virus Software. AV simply doesn't work. Any criminal worth their salt will ensure their malware works with all common AV products. Also, they're really easy to avoid.
• Avoid Unfamiliar Websites. Thats not going to help you against the next site compromise and Java 0-day.
• Avoid unfamiliar email attachments. People do get legitimate emails from strangers. Also, how many people know that picture files don't end in '.jar'? And lets not forget about phishing.
• Install a firewall. What is this, 2003? Nothing is directly connecting to his NATed machine.

These precautions try to mask the core issue: malicious code can run on a computer, and there is nothing you can do about it except live in fear of every website and email attachment.

Even when following every single precaution, you could still be infected with malware.

Computers vs. Computing Appliances

The problem is that my father has a computer. A computer is a platform that permits arbitrary code execution. This encompasses pretty much all desktops and laptops.

What he needs is a computing appliance with a large monitor and a keyboard. A computing appliance is a platform that only permits execution of pre-approved code, like iOS or Windows on ARM.

In fact, the vast majority of people only need a computing appliance. They will never, ever develop software. They have no interest in running arbitrary, unapproved applications. The only unapproved code they will ever run is ZeuS or CryptoLocker.

A Computing Compromise

Every time OS vendors try to move into a direction of computing appliances, a vocal minority screams bloody murder. Just look at what happened when Microsoft introduced Secure Boot with Windows 8.

To some extent, these people have a point.

Computing appliances have many faults:

• Arbitrary application guidelines and limits
• Opaque and draconian review processes
• Single vendor control of all applications
• Loss of "freedom to tinker" and exploration

Of these, the last is the most important and can't easily be solved by competition between vendors.

It is important to let those who want to modify their computer and their software to do as they see fit. It just shouldn't be the default option.

The best execution of this I've heard of is the Developer Mode switch on Google's chromebooks. You have to physically flip a switch that allows unrestricted code execution. Additionally, flipping the switch wipes all local data.

It's a beautiful solution: there is no accidental enabling, and it prevents 'evil maid' attacks.

There is, of course, little profit in having a general computing mode in appliances. Most customers wouldn't use it, and it would cost time and effort to maintain. The only purpose would be to protect consumer freedoms.

Which is why computing appliances are a perfect target for government regulation. The FTC can require all computing appliances to ship with a 'general computing' switch to protect consumers from malware and controlling vendors. The millions of hours in saved frustration and tech support would be well worth it.

A Bit Flip That Killed?

During my bitsquatting research I was amazed how many critical RAM chips in a typical PC lack error correcting memory.

It turns out that ECC is missing from an even more critical device: cars.

Details from the recent Toyota civil settlement show that the drive-by-wire control of Toyota cars was lacking error detection and correcting RAM.

From EDN.com:

Although the investigation focused almost entirely on software, there is at least one HW factor: Toyota claimed the 2005 Camry's main CPU had error detecting and correcting (EDAC) RAM. It didn't. EDAC, or at least parity RAM, is relatively easy and low-cost insurance for safety-critical systems.

I can't fathom why that would ever be the case. The amount of RAM required is relatively small, and the extra cost is inconsequential to the total cost of a car. Oh, and the software runs next to a car engine.

As little as a single bit flip could, and possibly did, have fatal consequences:

"We've demonstrated how as little as a single bit flip can cause the driver to lose control of the engine speed in real cars due to software malfunction that is not reliably detected by any fail-safe," Michael Barr, CTO and co-founder of Barr Group, told us in an exclusive interview. Barr served as an expert witness in this case.

Drive-by-wire systems aren't the only critical control systems susceptible to bit-errors. There is some speculation that a bit-error caused a sudden altitude drop in a Qantas A330. Amazingly, airplane software systems did not have to consider single or multiple bit errors until 2010 (see page 222) to achieve certification.

Git and Bit Errors

Finally, a topic to unite my two most popular blog posts: git failures and bitsquatting.

A friend recently pointed me to an amazingly detailed investigation of a corrupted git repository. The cause of the corruption? A single bit flip. To quote the source:

As for the corruption itself, I was lucky that it was indeed a single
byte. In fact, it turned out to be a single bit. The byte 0xc7 was
corrupted to 0xc5. So presumably it was caused by faulty hardware, or a
cosmic ray.
And the aborted attempt to look at the inflated output to see what was
wrong? I could have looked forever and never found it. Here's the diff
between what the corrupted data inflates to, versus the real data:
  -       cp = strtok (arg, "+");
  +       cp = strtok (arg, ".");

It is quite amazing to see evidence of a bit error resulting in a perfectly innocuous, syntactically valid and yet completely erroneous change in a real program and a real codebase.

How many times does this happen without anyone noticing?

Bitsquatting at DEFCON21 and More

I was very excited to see that several researchers are investigating bitsquatting and writing about it. There were two presentations about bitsquatting at DEFCON 21, a presentation at ICANN 47, and a research paper presented at WWW2013.

Jaeson Schultz - DEFCON 21 - Examining the Bitsquatting Attack Surface
Jaeson presented some excellent ways to exploit bitsquatting that I did not think of -- such as using bitsquats in URL delimeters to target otherwise unexploitable domains. As an example taken from the paper, ecampus.phoenix.edu can become ecampus.ph/enix.edu/.

Additionally Jaeson presents a great mitigation that can be implemented at the local level -- Response Policy Zones. From the paper:

An RPZ is a local zone file which allows the DNS resolver to respond to specific DNS requests by saying that the domain name does not exist (NXDOMAIN), or redirecting the user
to a walled garden, or other possibilities. To mitigate the effects of single bit errors for users of a DNS resolver the resolver administrator can create a Response Policy Zone that protects against bitsquats of frequently resolved, or internal-only domain names.   

Robert Stucke - DEFCON 21 - DNS Has Been Found To Be Hazardous To Your Health
Robert demonstrated some new vectors for bitsquatting, such as web applications and hosted email providers. Speifically, he bitsquatted gstatic.com (a site that serves static content for Google properties). Not only was he able to return arbitrary content to people using Google's search services, he could also affect web applications, such as feed readers, that rely on correct resolution of gstatic.com. Robert also bitsquatted psmtp.com, a hosted email provider. This allowed him to potentially receive other people's email.

Nigel Roberts - ICANN 47 - Bitsquatting
Nigel (who runs .gg) presented about bitsquatting to ICANN. Hopefully this will result in more research at the ccTLD level. 

Nick Nikiforakis, et al. - WWW2013 - Bitsquatting: Exploiting Bit-flips for Fun, or Profit?
Nick and his coauthors did a measurement study about the prevalence of bitsquatting and what content appears on bitsquatted domains. They identified several that are used for adverstising, affiliate programs, and malware distribution.  There is also a great graph in the paper where you can see a huge spike in bitsquat domain registration after my Blackhat presentation :).

Introducing Binfuzz.js

Tomorrow morning I will be giving a demonstration of Binfuzz.js at Blackhat Arsenal 2013. Please stop by the Arsenal area from 10:00 - 12:30. The slides are already available on the Blackhat website.

The Binfuzz.js page on dinaburg.org is now live, and all the code is uploaded to Github.

What is Binfuzz.js?

Binfuzz.js is a library for fuzzing structured binary data in JavaScript. Structured binary data is data that can be easily represented by one or more C structures. Binfuzz.js uses the definition of a structure to create instances of the structure with invalid or edge-case values. Supported structure features include nested structures, counted arrays, file offset fields, and length fields. The live example uses Binfuzz.js to generate Windows ICO files (a surprisingly complex format) to stress your browser's icon parsing and display code.

Features

Binfuzz.js includes support for:

• Several predefined elementary types: Int8, Int16, Int32 and Blob.
• Nested structures
• Arrays
• Counter Fields (e.g. field A = number of elements in Array B)
• Length Fields (e.g. field A = length of Blob B)
• File Offsets (e.g. field A = how far from the start of the file is Blob B?
• Custom population functions (e.g. field A = fieldB.length + fieldC.length)

The ICO fuzzing example includes uses of all of these because I needed them to implement ICO file generation.

Combinatorics

Binfuzz.js calculates the total number of combinations based on how many possible combinations there are for each individual field. It is then possible to generate a structured data instance corresponding to a specific combination number. It is not necessary to generate prior combinations. This way random combinations can be selected when fuzzing run time is limited.

Why?

The best way to learn is by doing, and I wanted to learn JavaScript. So I decided to create an ICO file fuzzer in JavaScript. I chose ICO files because of favicon.ico, a file browsers automatically request when navigating to a new page. After starting the project, I realized I got a lot more than I bargained for. Icons are a surprisingly complex format that has evolved over time. There are several images in one file, each image has corresponding metadata, there are internal fields that refer to offsets in the file, and the size of the icon data for each image depends the metadata. All of these interlinked reationships need to be described and processed by Binfuzz.js.

A Travel Story

I travel frequently, and not just to the usual tourist destinations. I've gone to places like Singapore, Japan, Ukraine, and India. This is a story of trying desperately to get home from a recent trip.

Be Aware of You Surroundings

The first sign of trouble was when the roof started leaking. The storm outside had only been raging for half an hour before the two droplets landed on my head. The buckets and signs on the floor captured most of the leaks, but there were numerous unmarked drips. You don't want this kind of water in your hair or food. Always be aware of your surroundings.

I arrived at my gate to find a delayed flight. At least there was now time to eat, since all the good food options were in the international terminal. Casually I trekked over, scarfed down some ethnic food and began meandering back to the gate.

An Ominous Sign

The blaring alarm noises and the flashing emergency lights of the fire alarm told me something was wrong. Neither the airport staff, nor security, nor the airline staff knew what was happening. Just that there wasn't a fire. Maybe. There was no smoke, no firemen, and everyone was calm. Carefully and slowly, I continued towards the gate. Then the power went out.

It was still daytime, and the sunlight and fire alarm (probably on backup power) provided enough lights to get by. The gate was in complete chaos. None of the computers worked, and the staff tried their best to assuage angry passengers. Some were just grumpy, others in tears, but all wanted some answers. There were no answers, no air conditioning, and it was getting hot.

The darkened terminal was like an impoverished refugee camp. Uniformed staff handing out bottles of water to angry men, crying women and screaming children. Mobs of people begged staff for answers. It was dark, hot, loud, and no one knew what was going on. This went on for two hours.

Then the plane arrived, but we couldn't board. The jetbridges were electric and wouldn't extend. Other parts of the airport had power, but the airline couldn't or wouldn't use the working jetbridges. The flight was cancelled, but we weren't rebooked to a new flight because there was no power. We had to call the central reservation office. This was never announced, of course. I happened to overhear another passenger talk with the gate agent.

So I called. After 30 minutes, a man with an accent answered. He sounded legitimately concerned, but all the flights for the day were sold out. I asked if he could rebook me on a competing airline; he typed something on a keyboard and then told me that all flights on all airlines to my destination were sold out. No inventory; The earliest possibility was the next night. Begrudgingly, I agreed. I asked about hotel vouchers. "Of course, the gate agents can print them for you", he replied. The power was still out.

Perseverance

I wasn't about to spend another night in this place. There was still one flight on a competing airline, and it was leaving soon. Sure I was told there were no seats, but one can't blindly trust a company to do something that lowers profit. And besides, maybe someone wouldn't show up. The competing airline was naturally in the furthest possible terminal from where I was. It was a long walk, but they probably power.

Turns out there were seats on the flight. There would be a cost, but I would get home, today. Life was also better in this part of the airport. There were no leaks in the roof and the air conditioning was on. Nobody was crying. I sat down near the new gate.

There was a current flight there, delayed indefinitely. Must be weather, I figured. That assumption was shattered when I heard two airline employees talking next to the gate entrance. Turns out the plane needed fuel. They sent for a fuel truck, but it arrived without fuel. For the past half hour they were trying to find either fuel, a new fuel truck, or whomever got them into the boondoggle. No one was answering. After 15 more minutes, they found a new truck. Two hours later my plane arrived.

Don't Count Your Chickens...

As we boarded, I realized there were plenty of seats. The flight was only half full. So much for "no seats available." As we taxied to the runway, I was thankful to finally be out of this wretched place.

My enthusiasm was premature. Literally as we were next in line to depart, our plane was directed to stop. After an hour on the tarmac, the full story unfolded. The original flight path had unexpected weather. We had an approved alternate flight path, but in the time it took us to taxi (about 30 min in the rain at this airport), the alternate flight path also had unexpected weather. It was 30 more minutes before we moved again.

Departure At Last

This time, it was for real. Our wheels touched off the ground and we ascended into the stormy sky. The plane shook violently as we passed through the rain, wind, and lightning. I clutched my seat and thought about this abhorrent place: the fire alarm, the crying passengers, the hot, dark, sweaty terminal, helpless staff, the leaky roof and fuel-less fuel trucks. I just wanted to go home and to never again fly into Philadelphia International Airport.

Readability Improvements

In preparation for new content relating to my Blackhat 2013 Arsenal presentation, I made some readability improvements to the blog and to dinaburg.org in general.

The layout is wider and the font size for text is now 16 pixels.

Please let me know what you think.

Git Fails On Large Files

Turns out git fails spectacularly when working with large files. I was surprised, but the behavior is pretty well documented. In typical git fashion, there is an obscure error message and an equally obscure command to fix it.

The Problem

A real-life example (with repository names changed):

artem@MBP:~/git$ git clone git@gitlab:has_a_large_file.git
Cloning into 'has_a_large_file'...
Identity added: /Users/artem/.ssh/devkey (/Users/artem/.ssh/devkey)
remote: Counting objects: 6, done.
error: git upload-pack: git-pack-objects died with error.
fatal: git upload-pack: aborting due to possible repository corruption on the remote side.
remote: Compressing objects: 100% (5/5), done.
remote: fatal: Out of memory, malloc failed (tried to allocate 1857915877 bytes)
remote: aborting due to possible repository corruption on the remote side.
fatal: early EOF
fatal: index-pack failed

I pushed the large file without issues, but couldn't pull it again because the remote was dying. The astute reader will notice the remote was running gitlab. The push also broke the gitlab web interface for the repository.

From my Googling, the problem is that the remote side is running out of memory when compressing a large file (read more about git packfiles here). Judging by the error, git attempts to malloc(size_of_large_file) and the malloc fails.

This situation raises conundrums that may only be answered by Master Git:

• Why was I able to push a large file, but not pull it?
• Why would one malloc(size_of_large_file) ?
• What happens when you push a >4Gb file to a 32-bit remote?

I was curious enough about the last one to look at the code: it will likely die gracefully (see line 49 of wrapper.c). Integer overflow likely avoided; would need to read more code much more carefully to be sure.

The Solution

In theory, the solution is to re-pack the remote with a smaller pack size limit. That requires ssh access to the remote repository, which I don't have. So the following fix is untested, and taken from http://www.kevinblake.co.uk/development/git-repack/. The obscure command in question (must be run on the remote):

git repack -a -f -d

Of course, repacking the remote but having non-repacked local repositories around may cause other problems.

Just For Fun

Here is another large file fail:

artem@MBP:~/temp/largerandomfile$ dd if=/dev/urandom of=./random_big_file bs=4096 count=1048577
1048577+0 records in
1048577+0 records out
4294971392 bytes transferred in 437.836959 secs (9809522 bytes/sec)

artem@MBP:~/temp/largerandomfile$ git add random_big_file
artem@MBP:~/temp/largerandomfile$ git commit -m "Added a big random file"
[master 377db57] Added a big random file
1 file changed, 0 insertions(+), 0 deletions(-)
create mode 100644 random_big_file

artem@MBP:~/temp/largerandomfile$ git push origin master
Counting objects: 4, done.
Delta compression using up to 2 threads.
Compressing objects: 100% (2/2), done.
error: RPC failed; result=22, HTTP code = 413 KiB/s
fatal: The remote end hung up unexpectedly
Writing objects: 100% (3/3), 4.00 GiB | 18.74 MiB/s, done.
Total 3 (delta 0), reused 1 (delta 0)
fatal: recursion detected in die handler
Everything up-to-date

Everything up-to-date, indeed.

OptimizeVM: Fast Windows Guests for VMware

Do you make Windows VMs? Are they slow? OptimizeVM will make them fast(er).

I got tired of re-googling and re-typing the same commands over and over make my Windows VMs fast, so I collected them on Github: https://github.com/artemdinaburg/OptimizeVM.

OptimizeVM is based on the commands provided in the official VMware View Optimization Guide.

The goal of OptimizeVM is to minimize disk access and remove fancy graphical effects. Certain Windows features, such as Windows Search, System Restore, Windows Updates, and Registry Backup will cause constant background disk access. The disk access makes your VM slow and increases virtual disk size. These features are also unnecessary for VMs that get reverted to a clean snapshot every couple of days.

The script also removes some annoyances like the Action Center, Network Location Wizard, hidden file extensions, and so on.

The disabling of some features, such as Windows Firewall, Windows Defender, and Windows Update do lower the security of your system. If you are very worried, turn them back on. I leave them off since in my workflow VMs get reverted to a clean state every few days.

If you want to speed up your Windows VMs, here are a few more useful links:

• Optimize Windows Guest Operating System Performance
• Optimizing Windows 7 for Linked-Clone Desktops
• VMware Communities Thread

JavaScript Frustrations and Solutions

Since there's no better way to learn than by doing, I've been teaching myself JavaScript by writing a structured binary data fuzzer. The fuzzer currently generates Windows ICO files, and will soon be released. In the meantime, I wanted to describe some frustrating experiences learning JavaScript and include solutions to them.

Object Orientation in JS is Confusing 

Some of this may be because I am used to class-ical inheritance, but considering the number of JavaScript OOP libraries (e.g. oolib, dejavu, Klass, selfish), I'm not alone.

The first confusing thing is that objects are functions declared via the function keyword and instantiated via the new operator. The overloaded use of function doesn't let you know right away if the code you are reading is an object a traditional function. The use of the new operator gives a false impression of class-ical inheritance and has other deficiencies. For instance, until the introduction of Object.create it was impossible to validate arguments to an object's constructor. The deficiency is shown in the following example.

In this motivating example, we want to create an object to encapsulate integers and validate certain properties in the object's constructor. The initial code could look something like this:

function Int(arg) {
    console.log("Int constructor");
    this.name = arg['name'];
    if(this.name === undefined)
    {
        alert('a name is required!');
    }
    this.size = arg['size'];
};
Int.prototype.getName = function() {
    console.log("Int: " + this.name);
};
var i = new Int({'name': 'generic int'});
i.getName();

Running this code would print:

Int constructor
Int: generic int

But now lets say I want to write something to deal specifically with 4-byte integers. The initial code to inherit from the Int object would look similar to the following:

function Int4(arg) {
    arg['size'] = 4;
    Int.call(this, arg);
    console.log("Int4 constructor");
};
Int4.prototype = new Int({});
Int4.prototype.constructor = Int4;
Int4.prototype.getName = function() {
    console.log("Int4: " + this.name);
};
var i4 = new Int4({'name': '4-byte int'});
i4.getName();

This code will alert with 'a name is required'! To set Int4's prototype chain we need to create a new Int object. Arguments to the constructor cannot be validated since they are not known when new Int({}) is called. Luckily this has been fixed by use of Object.create:

function Int4(arg) {
    arg['size'] = 4;
    Int.call(this, arg);
    console.log("Int4 constructor");
};
Int4.prototype = Object.create(Int.prototype);
Int4.prototype.constructor = Int4;
Int4.prototype.getName = function() {
    console.log("Int4: " + this.name);
};
var i4 = new Int4({'name': '4-byte int'});
i4.getName();

All Functions are Function Objects and all Objects are Associative Arrays.

All functions are actually Function objects, all objects are associative arrays. There are also Arrays, which are not functions and but are also associative and also objects. Sometimes you want Arrays to be arrays, and sometimes you actually want Objects to be arrays. Confused yet?

Scoping Rules and Variable Definition Rules that Lead to Subtle Bugs

Scoping rules are a bit confusing, since there is at least three ways to declare variables: assignment, var, and let. Of course, all of these have different semantics. The biggest problem for me was that creating a variable by assignment adds it to the global scope, but using var will keep it in function scope. And when using identically named variables, a missing var in one function will make that function use the global variable instead of the local. Using the wrong variable will lead to lots of frustrating errors.

The solution is to always "use strict" to force variable definitions. Of course, doing this globally will break some existing libraries you are using. Such is life.

Type Coercion With the Equality Operator (==)

Its amazing what is considered equal in JavaScript via ==. Instead of restating all these absurdities, I'll just link to someone else who has:
http://javascriptweblog.wordpress.com/2011/02/07/truth-equality-and-javascript/

When I started my project, I didn't realize that the Strict Equality (===) existed. It should be used anywhere you would expect == to work. It seems more sane to have == be Strict Equality, and another Coercive Equality operator (something like ~= or ~~), but what is done is done.

Problems Modularizing and Importing Code

C/C++ has #include, Python has import, JavaScript has... terrible hacks. There is sadly no standard way to import new code in a .js file, making modularization of your code difficult. I resorted to simply including prerequisite scripts in the HTML where they will be used, but I wish there was a way to include JavaScript from JavaScript.

Browser Compatibility Issues

Not all browsers have Object.create. Not all browsers have console.log in all situations. Not all browsers support "use strict". Turns out every browser is slightly different in a way that will subtly break your code, but of course the main culprit is usually IE.

Solution to Printing Blank Pages Problem in Linux

This isn't an overly technical post but I hope it saves someone hours of frustration printing on Linux. 

In my case the problem was a combination of broken generic printer drivers and a bad default value for the "Print Quality" setting. As a word of caution, according to the Anna Karenina Principle odds are your problem is its own unique snowflake and this wont help you print.

Problem 

• You are trying to print from Linux. 
• The printer starts, makes printing noises, but only a blank page (i.e. one with no ink on it) comes out.
• You verified your printer works by printing from another OS. If you have not, do this. If your printer still prints blanks on Windows/MacOS, you have a printer problem, not a Linux problem.

Solution

The solution is two part; both parts were needed to actually see ink on paper.

1. Install printer-specific software.
   
   The drivers that came with CUPS and claimed to support my printer didn't work. For HP printers, you need to sudo apt-get install hplip, and run hp-setup. If you have another brand printer, look here for help.
   
   
2. Change the "Print Quality" setting to something else.
   
   The setting is in the CUPS web interface. Go to http://localhost:631 (you may need to log in with a local account) -> Administration -> Manage Printers -> Your Printer's Name -> Administration Selection Box, pick "Set Default Options". Clicking that will take you to the following page:

Change the Print Quality setting to something else. Try all the values. For me Normal Grayscale worked, Normal Color did not.

Try all the Print Quality options. Hopefully one of them prints. Yes, the setting is hard to find to and obscure, but hey, at least you didn't have to edit another config file!

My next post may be about trying to get network printer sharing to work between Linux and Mac OS X Mountain Lion, which was its own struggle.