Lukáš Linhart

Programmers are not dentists

written by Almad

Jeff Knupp wrote an insightful article about DevOps movement killing developers which I think neatly illustrates the tension between horizontal and vertical specialization.

Horizontal specialization tends to be default in many professions: you do one thing, and you do it well. You spend years building narrow expertise, and you are great at solving it. However, when crossing the boundary of your discipline (as in handing over your application to production), you need another (team of) specialized person.

Verticals are obvious opposites, although unlike the proverbial Jake, they are master of some of the skills chosen, leading to t-shirt shape.

What should you choose for your professional development and for your company?

The largest trade-off of horizontal specialization is self-sufficiency. You start to heavily depend on external orchestration. In the dentist example, it took me multiple rounds of visits between ENT and dentist to figure out where my problem is, because as the problem was on the boundary, each of them felt it should be handled by the other.

In software, your dependency is going to be someone with the project manager role (or obviously, if you choose to be the conductor, you'll depend on others to actually get anything done).

Those two together introduce a whole new dynamic, to be seen in larger companies:

Team size explosion

It's impossible to get small team, because obviously you need at least:

  • Product Owner (you don't want to think about users, do you)
  • Coder (who will get the design, produces CSS and tweaks)
  • Frontend Developer (who will work with coder to get the responsive SPA done)
  • Backend Developer (to write API for the frontend dev)
  • DBA (to manage database in production)
  • Documentation writer (everybody else is too expensive to write one, or not?)
  • Ops guy (to actually run this thing)
  • QA (to ensure everything works together)
  • Project Manager (I mean, the team is 6 people already, they can't coordinate without adding 7th...)

Talking many languages

Given number of people involved, you need to obviously pay more attention to communication, which becomes a feat itself. This is usually handled by the Project Manager, but that also introduces SPOF and significant delays.

Because of that, it is easy to have diffusion of responsibility among the team, easily leading into "it's their problem" kind of mentality.

This is also caused by foggy visibility into the original stories. In this case, even a simplest user story ("I want to see a list of items") crosses five domains and requires four handover. "Handover" is a corporate codename for a game of Chinese whispers, because even if you have quality and up to date documentation and specification, details are often re-interpreted according to a person implementing it, leading to deviations.

Vertical trade-off

Idea of vertical specialization isn't for everyone to gasp the whole specializations, but that every member of the team should be able to work in every part of the stack. This is enabled by every member of the team to know at least basics of programming.

This is because we found a way how to automate, to certain extent, a lot of non-development work. For navigating the basic operations infrastructure, you can now use Cloud APIs. Avocode can give you what most coders did. Good testing pyramid saves a lot of QA work [1].

Danger with people wearing multiple hats is the potential to cut corners where they shouldn't. However, because they now can own the whole story and are responsible for results, they are also better positioned to know what the time/quality requirements and possible trade-offs are.

So far, those are options and choosing one depends on your environment and scale. There is, however, one more major issue why I think for web, vertical development teams are much better.

Learning and Architecture feedback loop

For SaaS environment, there is an inherent and very tight feedback loop between production environment and development of the code itself.

On the soft side: how are people actually using your software? What are the common scenarios? Are they fast, reliable and covered with tests? Do they work well in both mobile and app?

As you'll write software from "how is user using it" as opposed to "what technical things are enabling the usage", you can answer thousand tiny details that make the user experience much better.

On the hard side: what are your traffic patterns? Where are performance bottlenecks? In which dimensions are your scaling problems? Are you expanding geographically, where, and what are your speed and latency SLAs?

All of those change over time and affect your architecture significantly. I don't believe in any operations being able to have "software package" as a generic black-box unit and run it well; software must be well-designed and well-architectured to do so.

There is nothing that enables -- and even forces -- you to do that as much as being responsible for system behavior in production. Suddenly, designing proper indexes and making sure your app is using them is a much different beast. If you need to know what happens to software in production, you need to have well-designed logs and you pay much more attention to it when writing the code.

Yes, it means less time for writing actual code -- but much more information for doing it well.

The Dentist Analogy

If horizontal programmers are like dentists, it doesn't mean having dental assistant. It usually means having patient prepared in the chair (by PMs) with a diagram of tooth to fix (spec), just somehow fixing it (writing code) and then leaving for another patient. Either someone else will check whether it's actually fixed (QA) and send it back to you, or the patient will discover it (in production), but maintenance will be done by someone else anyway (Ops), so you are fine to go.

I don't think this makes pleasant experience in health care, and I want to avoid it in software as well.

[1] This isn't to say that knowing to program makes you automatically good at those. Things mentioned are both roles and disciplines; in order to do them well, you obviously need to learn at least the principles.

Published on 2016-02-23

Friday Readings

written by Almad

This is part of Friday Readings series: commented readings I find worth sharing. If you want more, follow me on Pocket.


I don't work on hardware level, so Dan Luu's report on Intel CPU bugs in 2015 is surprising to me, and a bit scary.

While on Dan's blog I discovered because of that, Files are hard are very good as well. But honestly, just go and read most of it.

Leadership and company building

On Being Relentlessly Resourceful and how leading by example works. I can only echo this; I still see how culture in Apiary is still derived from founders and the initial team.

Linkedin's pitch to Greylock for their series B is not only an interesting resource for company states, but also an interesting revision of how Internet was like back in 2004, and which predictions worked and which not.

5 Scientific Suggestions Curbing an Employee's Sense of Entitlement provides suggestions on dealing with behavior that can make the group dynamics very toxic.

There is a turmoil in GitHub that also provides new context for previous CEO departure done in response to Julie Ann Horvath's story. Nevertheless, that's "news"; the interesting story is the dynamics of changing a lifestyle company into a VC-funded growth company.

Inspiration for hiring: We got 10 CEOs to tell us their one killer interview question for new hires. I usually walk with only one important question into interviews: "What do you want to do?".

Stop Complaining challenges you, but I find

Heisenberg developers: you can't observe developers without changing their behavior. Story about snowballing oversight and its effect on projects.

World around us

Fascinating story about how Columbian army warned hostages using popular song. I'd totally think it's made up and only a movie plot.

For the first time, we've produced and sustained hydrogen plasma. There is a lot more to harvest it as a source of energy, but after a very long while, it looks like at least some progress.

More pieces into puzzle of price discrimination; it's so widespread on the Internet that nobody notices or cares anymore, but it's interestingly gender-based in the real world:

Hackers paralyzed a hospital in Los Angeles and demand a reward for decrypting patients data. Given we currently don't have a good legal framework on who actually owns the data, it adds a lot of interesting facets to the story, i.e. who is responsible for backing it up. Sucks if you can't have them yourself.

Published on 2016-02-19


written by Almad

If your startup is worth doing, then you probably imagine a world where something is done differently, in a way you consider better.

Creating a product is part of that, but it revolves mostly around bending the real world around your vision and driving it there. Huge part of that is guiding other people to realise that they themselves want to go there.

My mentor once said that the main job of any principal engineer or architect is writing things down.

This is going to be an attempt at that.

There is going to be a short essay or tech writing every Tuesday, and collection of readings every Friday.

Hope you'll enjoy that.

Published on 2016-02-16