The Hype Machine and how to avoid its traps

We live in a world where our connectivity and transmission of information is faster and more voluminous than our ability to critically analyse it. Additionally, the world seems to build software faster than we can figure out what to do with it or who it is useful for.

It’s this technological hype machine, driven by our high connectivity, that results in our view of technology being overwhelmed by whatever is the NEW BESTEST THING.

So this is the first in a series of posts I will be doing on this “hype machine”. This post will cover what the hype machine is, and how to best see past it. Following posts will cover how we got burnt by the hype machine and what we learnt.

What is the “Hype Machine”?

When I say “hype machine”, I’m referring to that effect when a new piece of technology is released. A combination of tech media, the easily excited twittersphere and our relentless need to feel like actually know what is going on — results in us choosing technologies that are incorrect for the job. These technologies are often still finding their way — the creators are perhaps still working out what the use cases are, the documentation isn’t quite up to scratch and the project has not matured enough to give itself sensible default configurations. And it’s understandable — the world can produce software faster than the world can figure out how it works and how it needs to be used. The pattern goes as follows:

• Tech giant releases new library.
• Tech giant’s marketing does it’s best to define the use cases for it — but often prematurely, hopefully unknowingly ( sometimes it feels intentionally) and those use cases are wrong. Think realistically how the developers of the technology will tell the marketers and what the marketers will decide to feature. “Can you explain this feature in one word?” “Erm… realtime scalable?”. Wat?
• Tech evangelists and bloggers start writing tutorials on how to do the basics in it. They need to get some pieces out and they’ve found the star of the show.
• Tech leads who like that particular Tech giant begin to play with the technology. Everything looks good from here. They’re tweeting and making a scene.
• The tech is used successfully on a couple of case studies. Bloggers start writing posts and speakers start doing talks about why it’s better than everything that has existed up until now. The inventors of the technology are replying on Quora. The hype machine is running at full steam!
• Tech companies start selecting it to use in production systems. Sometimes based on those premature use cases defined by the tech company itself. Regularly based on the company’s own misjudgement of their needs. Sometimes based purely on the fact people are talking about it and it seems sexy. We’re all to blame.
• After quite some time (after writing inadequate prototypes, developing with this tech, testing and maybe even putting it into the universe), the problems with the choice of technology starts to rear it’s ugly head. Maybe it doesn’t scale, maybe it doesn’t have sensible defaults. Perhaps the API’s have bugs, maybe it leaks memory.
• Those who were burnt start posting articles about the problems that technology caused (this often takes a while, since we often doubt ourselves — is it us? Did we use it incorrectly? Is that why it’s not working?)
• We become wiser. Use cases are updated, the projects matures eventually by which time everyone has already switched back to an industry standard or built something bespoke.

The amount of money that has been lost on simply learning that a new technology is the wrong one for the job is non-trivial. And we should become better at evaluating new technologies. That’s what I’m going to address in this post.

In addition to this post, I will be doing more blog posts on what is behind the hype machine of <insert technologies here>, but for now, let’s talk about the best strategies for peeling back the deceitful words of marketing, and finding out what they really can do and how that suits what you need

Why these techs end up in production systems?

1. The website said it could. It passed a simple “PoC” test.
2. You read countless articles about how great it is. It’s currently THE BEST out there. The most reliable, the most scalable
3. It’s the only one that solves a specific problem you are trying to address

Let’s take on each of these and figure out how not to get caught in the traps

1. The website said it could do it and it passed the simple “PoC” test

Software systems are complex. Our use cases may seem similar on the surface, but deep down, they are often not. A feature might seem to suit our needs, but it doesn’t when push comes to shove. How dare a piece of tech’s fancy parallax website attempt to summarise those complex features into single words?

But whatever. We’re all excellent devs here, let’s quickly write a proof of concept. So the team writes a little proof of concept app (server and frontend) in a couple of hours and it works and everyone goes “Yay! It does the thing! Let’s use this”.

What did we do wrong here? Well, a little PoCis simply not enough to evaluate the entire tech

• Our PoC probably didn’t test our non-functional requirements enough. Is it actually scalable? Is it actually reliable? Is it actually secure? Is it easy to maintain? Do not trust the marketing garbage
• Our PoC probably wasn’t being run in a production-like environment. Was it running with the same load? No, definitely not.

What should we have done?

Extensive extensive tests. Not just a little app. Try test the technology’s limits, despite the marketing garbage. Do you currently have to maintain 1000 continuous connections to your server? Track the performance of the server at 100 and 1000, and 10000 and 100000 connections and see how it is affected. Leave it running for a few weeks at high load. Does it leak memory really badly? Did it fall over randomly?

This sort of testing could take a couple of weeks, rather than a day, but it’s worth it if integrating it into your system will result in months and months of wasted work and many sleepless nights

Digging. If it’s so new that security is not yet documented, then you’ll need to go look at the code. What major dependencies does it need? Apache Spark seemed like it would work for me, but then I started fighting with the hadoop dependencies (I now have a policy for avoiding Apache techs as far as possible). The marketing garbage is biased, as are blogs. You will need to do the digging yourself.

2. The Internet says it’s THE BEST and you want to be THE BEST

Marketers make sure you think that. It’s their job to get people using their piece of technology. It shouldn’t be surprising that they took a use case that has not been truly tested but is a “that seems like a nice idea to the devs” and sold it as a given. Do not trust the marketing lies. Treat them as if you’re being deceived. Because they only talk about the good, they never talk about the bad. And our confirmation bias hurts us. You get so excited that you’ve found this thing, and then fail to think there might be some major downsides. “But Google bought them! They must be good”. Google also has some of the best marketers. Beware!

I think we also forget how complex our use cases can be, and we forget to be critical about those use cases. Just because it worked for LinkedIn doesn’t mean that it’s worth it for you. LinkedIn have loads of support staff and ops people to watch and tweak their stacks. And due to their requirements, they don’t mind picking a technology that requires you to be an expert in it or requires a whole ops team to manage. Say you are a small startup, running off of AWS, with no ops-team. One day you want to be able to scale to the throughput that LinkedIn has, so why not build for it now right? Right?! You have the technology to do it so what’s the problem? Well, it turns out using that new, LinkedIn-promoted open source queue system is more finicky than it’s worth and requires you to understand it’s complex internals, in far more depth than you have the resources for.

I see this all the time with “Big Data”. Everyone thinks they have it, but few people do. Don’t tell a corporate to use the latest shiny new “Big Data” store if they’re not creating terabytes a day and they already have giant Oracle db that they spent millions on. I dunno if you’ve ever played with the major big data techs? More trouble than they’re worth for the most part. They’re only worth it if you have the data (and the accompanying funds and resources) to deal with them. Don’t pick it because it seems shiny.

Not to mention, just because it’s “THE BEST” feature-wise — the most reliable, most scalable, doesn’t mean that it’s the best in every scenario for every use-case or that it doesn’t come with a big set of it’s own downsides.

• BE CRITICAL of the marketing. Constant vigilance. Question everything.
• Never forget about existing technology. Don’t always look at the shiny things, when the old and steady will do (maturity has many upsides).
• Do not select a technology for a future that may never come — especially if it has obvious downsides for where you currently are. I know you hope it one day will, but you can build for that in the future.
• Be vigilant about those non-technical needs (like needing more ops people or requiring bigger servers than other options).

3. It’s the only piece of tech that solves the problem you’re trying to address

I can’t count the number of times that we’ve selected a library because it has a particular feature that none of the other libraries have. And you end up in the dilemma of either having to build your own library to do the thing, or use THAT particular one. No one likes to re-invent the wheel, right?

So even though it might have some issues with maturity, support or, I dunno, runs entirely in the main thread (**cough** firebase **cough**), you dive right in — because reinventing the wheel seems like such a heinous crime and the problem you need to solve is hard and you’d like to not have to fight through it yourself if others have already.

Later on, you realise, that the baggage the piece of tech came with (whether it’s the bulky dependencies or the major learning curve) was not worth it. Often, that single feature you selected the tech for was not even part of that technology’s core philosophy and thus loses support quickly or is hindered by other features. You could, and too late realise that you SHOULD have just implemented your own tool that solves your problem more specifically.

A particular burn story we experienced: Firebase Realtime Database. We needed server-to-client database-syncing. Firebase Realtime Database does that. Brilliant! The trick we missed was with the word Realtime. We were like “oh cool! It’s realtime! That’s not a requirement but totally gonna be awesome for the app”. At the time we didn’t think that Realtime implied it would need to run on the main Android thread — the same thread as the entire interface. We didn’t realise that syncing down loads of data would hold up the interface. Eventually, after much annoyance, when we spoke to the Firebase devs on slack, they told us that it was a design choice that it ran on the main thread… Why? Because it was REALTIME before it was “sync”. It was designed to carry a small amount of data that was imperative that it got updated to the user in REALTIME. That feature trumped the inefficiencies with syncing lots of data. (But the rest of that burn for my next post.)

When choosing that tech, ask yourself the following:

• Did you chose this piece of tech for features that are core to it’s philosophy
• Determine what the baggage is. Is it worth it? For how to determine what the baggage is, check my 1st point.
• If only a few pieces of technology have solved this problem so far, maybe that means it IS something that is worth your time to spent solving yourself for your specific needs.
• If you are going to chose a tech where the feature you need might not be a priority, make sure that tech is open source so you can fork it and change it (keep in mind, this comes with it’s own downsides)

And despite all this, still mistakes will occur

There is no free lunch when it comes to choosing the right technologies. And there is always a risk associated with selecting something hot off the press. Hope you enjoyed reading and good luck!

My 2nd “Behind the Hype Machine”, I will talk about Kafka.