The Dark Resistance

Say you have invented “the next big thing”, e.g. the next C, the next operating system, the next processor, and hope your thing will shine, masses will come to it and you will be forever famous.

You will invest money, time, the best engineering expertise you can get, to build and demonstrate how your thing runs the current top-of-the-line benchmark better than everyone else, or how its “killer app” can become obviously necessary to a large consumer market.

Yet, adoption will remain low.

That is, unless you have a plan to win over the dark resistance.

The dark resistance is formed by well-understood problems whose solutions are complex, foundational and have cheap market value.

“Complex” means it will cost you dearly to develop adequate solutions anew on top of your new thing. “Foundational” means all the other applications that matter need those solutions as prerequisites. “Cheap market value” means nobody will care if you find it difficult to implement these solutions, because the solutions your competion has been selling for 30+ years have become so ubiquitous that everyone expects them for free.

What are those problems?

To find them, you should talk to those people who have been maintaining the deep fabric of the software ecosystem of your competition: compiler / library / OS developers and maintainers of the last 20-40 years. Ask them what it is that they are responsible for, was implemented long ago but still often reused. Ask them why nobody provided new solutions since then. Then judge for yourself.

Some examples from the nearest smartphone, tablet or computer near you:

• fairly sharing access to a common hardware memory or processor between programs written by separate organizations with different programming tools;
• appropriately handling network congestion and delays in the TCP protocol;
• converting floating-point numbers from textual representation to IEEE754 and back again;
• performing arbitrary-precision arithmetic on natural and fractional numbers at adequate speed;
• computing and printing dates and times, while properly handling time zones, leap seconds and calendar adjustments;
• translating keyboard input to text characters across the worldwide diversity of keyboard layouts and languages;
• properly normalizing, comparing, encoding and decoding characters across multiple languages;
• managing typographical data and drawing text on a pixel-oriented canvas while properly handling glyph composition, kerning, ligatures and anti-aliasing;
• scaling, rotating, shearing and otherwise displaying arbitrarily large 2D images on a screen without noticeable visual artifacts;
• accurately converting color values between different color domains, eg. properly handling differences between cameras, LED displays and ink printers.

The dark resistance will capture 80% of your engineering effort. It does not advertise its existence, and typically does not reveal itself until very late in a complex innovation-driven project, when you already have spent 80% of your budget. Then it’s too late, your engineers will eventually give up and your project will eventually suffer the slow death of inadequacy and lack of maintainance.

The discovery and understanding of the dark resistance is the equivalent for language/OS/architecture designers of when an inexperienced entrepreuneur discovers the hidden cost overheads of starting a new business to manufacture, distribute and sell hardware commodities. Supply chain management. Accounting. Dealing with manufacturing defects. Consumer returns. And so on.

Relevance for real-world computing innovations is hard.

Like this post? Share on: Twitter ❄ Hacker News ❄ Reddit ❄ LinkedIn ❄ Email

Raphael ‘kena’ Poss is a computer scientist and software engineer specialized in compiler construction, computer architecture, operating systems and databases.