Tech&team(software engineering) views

10 August 2022

pro

Table of contents


Target Audience and what to expect

Target audience: junior-mid engineers. Or senior engineers who just want to learn about my thought process. It's okay to use this document for our interview process. I haven't copied things from Google. Still, keep in mind that I've put quite some thought into this before writing, calm and relaxed at home, so I may not be able to recite on the spot, under pressure, some of the things that I've written. There could also be small inconsistencies because I'm learning all the time and my views change with learning.


When to pick tech-stack X vs Y?

Stick with the tech stack you/your team is already used to. That is assuming you haven't picked some unusual stack such as Haskell backend and some transpile-to-js frontend or Arch-based server for production. I like Haskell, I've heard very good things about ClojureScript(transpile-to-js Clojure), and I know that some of the most techie people use Arch distros successfully, without that slowing them down. But I do know for sure that this is not scalable. Not scalable to many people, I can't say it's not scalable to many machines.

DO pick Rust/Haskell/Clojure/OCAML/transpile-to-js/wasm/any of the non-established/less-known tech-stack if you'd like your tech-stack to filter for passionate developers by itself. Or if you are a manager who trusts their engineers, and wants to "spoil" them a little bit.

DO study Rust/Haskell/other tech-stacks that you might not use in production. Look at them as thinking frameworks. E.g. learning about Elixir/Phoenix framework will make you think more about building concurrent backend services. If you're coming from Python and you try Typescript, you will learn how much easier/safer refactoring can be, and vice versa. Reading about Rust might introduce you to some system programming concepts. Learning about Linux might make you appreciate Windows (ha-ha). Learning Golang might make you question what makes a programming language good. There are very interesting discussions e.g. on Quora from functional programmers that give a very solid critique of Golang. At the same time, there are a lot of people with many years of experience, functional programming included, saying that Go is a breath of fresh air. Either way, it will make you understand better programming languages and their ecosystems.


Architecture

TLDR: Here I've listed repositories that no matter how hard I try to explain with my own words, it's still going to be a massacre compared with their quality. It's good to know your limits :)


Performance

You probably shouldn't be worried about performance from the start. There are more important problems to figure out at first. Yet, if you know for sure that your server/app might need some lower-level optimizations, you can still isolate these into separate services and pick a language known for its performance, e.g. Rust, C/++, Go, etc. Still, build your main business logic in the tech-stack that is well-known/high(er) level that is easy to use by more junior developers. Then do REST/RPC/Graphql calls to the high-performance service.

Or if performance is needed for a web-based frontend app, you might try to build that part of the app with e.g Rust(yew framework) and try connecting it via micro-front-ends architecture to your main React/Angular app. I'm not very familiar with the micro-front-ends concept. It may be unreasonable but it's worth exploring before building your whole front-end app in Rust. Btw I like Rust. Or any other language that forces the developer to think hard about what he/she is doing.


What editor/IDE should I use? (Neo)Vim?

TLDR: Stick to the most used/popular editor for your environment.

Neovim, I've spent enough time, around 100h, using/configuring. Now I know that I might come back to use it in the future and that 100h weren't enough to become comfortable using it for my daily work. Though your mileage may vary, I believe that for most people the 100h mark is a sure thing, to be able to understand the editor/configs but not feel comfortable using it for daily work. There are always exceptions, I know.

Emacs, performance is slow for nowadays standards and its dev community is not stable, i.e. slow to fix bugs and introduce new features. I've heard that fewer and fewer people want to work on their codebase.

DO try (neo)Vim extensions for your editor. E.g. VSCodeVim.

Update: I've been actively using LunarVim for a few weeks. I'm slowly becoming more comfortable but I've already invested at least a couple hundred hours in the neo/vim ecosystem. And I still waste a lot of time configuring. Even though Lunarvim comes with saner defaults, once things don't work, you're expected to know even more than if you did a small, simpe config yourself. Compared to VSCode, where I just went into it and understood pretty much everything I needed to in just a few days, the difference is huge.

Update2: No more. 6 months was enough. Biggest time sink, really. Sunk-cost fallacy on steroids. "But, it makes you a better developer", yeah, but you can become a better developer by solving important real world problems with software, and not fighting with your editor. Having fear to udpate because you don't know what will break. Having to read source code for editor packages because things break unexpectedly. Spending half your time in github issues, just not for your own projects but for basic editor features that are supposed to work.

For me, at that point, the conclusion is that Neovim(plus all its frameworks like Lunarvim) are just toys for people who don't have more important things to do. They break way too much and in order to be able to tap into their power, you have to remember a lot. And remember it over time, which is even more difficult. I also found it wasteful because I only have so much memory, and I prefer not to waste it on my editor's internals. One of the worst time waste of my life.


When to leave comments with code?

TLDR: when it's not obvious why a technical decision has been made. Not explaining what the code does. Or, explain what code does if it's really difficult to write best_practices_code, e.g. performance optimizations, working around library problems, etc. MUST explain regexes.


How is Linux better/worse for personal computer/dev environment?

TLDR: do not bother switching unless you have at least 20-40 extra hours to play with it for the first few weeks or even months. The benefits come after at least a month or two of an uphill investment with a lot of effort, and unexpected issues. Assume the slogan "things don't work by default" and you'll be well prepared. Yes, I'm talking about stable distros such as Ubuntu, Mint, and related.

How is it better?

  • The ecosystem is way more techie. The average Linux user is way more educated about software/computers and you'll learn a lot more about software both as an end user and as a developer. When you go to fix one problem, you'll end up learning a few new usually unrelated things. This is of course a double-edged sword. Which as a whole is not hard to translate e.g. to web-dev.
  • Command-line programs will become second nature. You'll not only feel easy using it but might start to prefer using it for common tasks such as searching and manipulating the file system.
  • If you end up in a DevOps(like) position or more backend than frontend, you'll very likely have to deal with Linux and the command line. Having used Linux for personal use will have prepared you well.
  • You want to customize your desktop experience to the max. With Linux, you can customize almost everything you can think of.
  • It's not uncommon to build your software from the source. You might end up reading more code, or at least code configurations such as Make files. This might be useful if you're a js dev, as a context to the whole package.json ecosystem and how things could be done differently.
  • More FLOSS software

Also, it's not hard to find forum threads/youtube videos explaining every little detail of how Linux is better. Read/watch youtube at least some of the most popular ones, until you get a feeling that what you're hearing/reading is repetitive. Then you can kind of say that you know fairly well the benefits. The Linux guys are good at pointing them out. Then it's time to get to know the bad stuff, ha-ha.

How is it worse than Windows?

  • Standards. There are a million ways to do the same thing in Linux and its community likes it this way. No, I'm not talking just about all the different distributions. I am talking about how things should be done within a single distribution. Heck, even about installing a simple program. You'll often find yourself reading hours of unrelated information about things you don't care about, that do not make you a better dev, that are extremely far from solving your problem, just to get a feeling about what's right or wrong. In other words, the best in Linux can be much better than in Windows, but also the worst much worse. In Windows, you know that the solution will kind of suck but you'll likely be over with it within a couple of minutes. In Linux that translates to hours, often days.
  • As a Linux user you're implicitly expected to be a good command-line user. And command line is far from intuitive for the first month or so. And even after the first few months, all the manuals(help pages) are still hard to decipher and many experienced developers still end up searching google for many common tasks that a well-designed GUI application could easily remind us how to use the different options. In Windows, you can easily find a GUI tool to do the job at hand that is very intuitive even for new users.

All that said, it's still useful to at least try Linux through WSL or a virtual machine, even if you're not planning on switching to Linux.


Why is bleeding edge tech bad?

TLDR: it's not. It's bad for the majority of problems/teams because the potential benefits that it introduces rarely outweigh the risks/bugs/problems/incompatibilities with how things are already working, and bad/no documentation for the new features. Say goodbye to Googling because you'll often be the first one to encounter the issues.

A situation to study is Discord migrating one of their services from Golang to Rust(latest). The interesting thing is that in the discussion threads reasonable comments were saying that they could have avoided the migration. And that's for Discord.. with millions of users of their service/s where they might need a specific bleeding edge tech/feature. So, if it had been unreasonable to use bleeding edge tech for an app, that has a lot of unique needs compared to the average app, it's probably unreasonable for most of the other apps, too. It's a generalization, I know. But I find it interesting to think about.


Why do some people dislike Java?

Good reason: the people that work with Java are usually older, with different life priorities than the ambitious, highly motiavated developers. Or in other words, many feel the ecosystem/people there are a little bit stagnant, too bureaucratic, not allowing them to experiment and learn new things. People feeling they don't get to build enough in order to learn. Having a lot of huge codebases to maintain by just changing very little pieces of code.

Reasonable reason: job opportunities don't seem to go in the direction of more Java.

Bad reason: overly eager developers needing to scratch their brain with constantly changing technologies such as those in the front-end ecosystem, not appreciating the stability and maturity of Java, and don't solve important problems but rather like to just build small projects with new tools.


When to use Redux for front-end projects, e.g. with React?

TLDR: When you have a lot of client-side state that is not just a copy/cache of data from the backend. Or state that has a lot of variables that you don't want/need/can't sync often with the backend. When state mostly resembles data from the backend, then ReactQuery, Apollo, or urql should be sufficient with some react context.


React vs Angular vs...

Angular is good if you want a lot of tools/structure/opinions out of the box without having to look at many places and wonder if you're missing out on a lot.

Angular is not good for junior(ish) developers that need to be productive fast. Also if you need flexibility, doing non-standard things, plugging a lot of 3rd party libraries. It's not that it cannot be done with Angular, it's just more intuitive/simple to do so with e.g. React. With React this is the way you're expected to work.

The boilerplate is IMO both good and bad. Bad for obvious reasons that there are more things to read and maintain, therefore more things to make mistakes on. Unnecessary if you're building small projects. Good because it's not just random boilerplate, it's usually boilerplate that gives an extra opinionated structure that is helpful as the project scales. Boilerplate that once you begin to think about how to modularize your React app, to make it more scalable, you end up inventing yourself. E.g. all the seemingly unnecessary creating of modules for everything. There is work done to simplify their modules structure to be more like React, more beginner friendly.

The best reason for React nowadays(2022) is that it's become an industry standard. You can't escape it. Not that it's bad. It's just that it's unreasonable to not learn it if you want to do serious front-end work.

Vue is a mixture between React and Angular with the best of both. I can't think of any serious disadvantages from React, except that its ecosystem is smaller, which might be an issue if you stumble upon edge cases. It's said to be more beginner friendly than both React and Angular, although I can't confirm.

Googling react vs vue(reddit) and going through the first google page results can give us good reasons why we might not want to use vue. TLDR: Typescript support feels like an afterthought. Yes, even for vue 3.


.NET vs Java

TLDR: before .NET core, the main good reason to pick Java over .NET was cross-platform compatibility.


Authentication

The safest/easiest solution is to use some service like auth0, or Firebase's auth. You don't need to use Firebase to use its auth features. May get costly after a few thousand users, so I'd check their pricing pages. By safest I mean safe from developer mistakes and less maintenance. Nowadays with all the political polarities, such services might not be safe if you don't share their political views.

The second best, and still a standard, is to roll up your own authentication service, using the most established libraries for your ecosystem.


Databases

Questions to consider:

  • NoSQL vs SQL vs NewSQL
    • TLDR: I think that relational DBs fit enough use-cases in order not to overthink it. Relational DBs can also store unstructured data if it's required. Unless you expect tens of thousands to a million requests per second(very rare), SQL DBs force you to structure your data, give it types, and think about its future use much more than NoSQL. IMO, NoSQL often encourages sloppy work from developers because the initial effort is close to nothing. This leads to more difficult querying down the road, and more difficult ensuring that the data is consistent(because NoSQL doesn't encourage normalization).

    • And even if you expect millions of requests per second, you can still separate that part of your app with a microservice that uses NoSQL. Just because a few parts of the app need this higher performance, doesn't justify the whole app using NoSQL. In my research, I've found companies with huge user bases scaling SQL DBs for their services. Most companies don't have user bases in the millions/billions. That said...

    • a good case for NoSQL that I can think of right now is chat apps. They need to be able to handle millions of requests per second. Having the accuracy of financial software is not a priority.

    • NoSQL is faster both in terms of performance and initial development speed

    • NoSQL has better horizontal(across many machines) scaling

    • NoSQL breaks ACID principles. E.g. not suitable for financial software

    • NewSQL is relational DBs designed to scale horizontally, although I haven't researched how they work. There might be marketing gotchas

  • OLAP vs OLTP. Or Analytical vs Transactional DB
    • OLTP is for front-facing user apps. The end user is usually a backend engineer, and the end-end user is the average Joe using your app in the browser.
    • OLAP is for Data Science/Machine Learning projects. The end user is usually a data scientist/analyst. Where reading a huge amount of data is more important than a huge amount of transactions. Data is read/written once in a while in bigger chunks. Though probably much more read than written. There are probably automated ETL processes that populate/update the OLAP DBs. Whereas in transactional DBs, there might be a lot of frequent reads and writes but not with a lot of data. E.g. users making posts all the time. Or a chat. A single post = single transaction = many transactions with little data. In OLAP DBs, a Data Analyst/Scientist reads huge quantities of data, if we think in terms of posts, the analyst might need to read millions of posts at once.

ORMs

TLDR: in the js ecosystem, as of August 2022, I can't find anything better documented than Prisma 2. Its Typescript support is also said to be superior out of the box. Yes, I know about the controversy from their v1 migration, where people were just left there with no (easy) way to migrate. I'd put Objection.js second. It's much more barebones, it's built on top of Knex.js, with better Typescript support out of the box.

I think although ORMs have become very complex, they still save enough boilerplate writing for the majority, let's say 90% of queries. And almost every advanced ORM(not just .js ecosystem) allows its users to fall to raw SQL when needed. A lot of flexibility is lost by not directly writing SQL, so for that, there are barebones ORMs like Knex.js and Objection.js. They still keep most of the flexibility of writing SQL but feel like you're still doing regular programming.


Typescript

TLDR: Please pick Typescript for any project longer than 5k LOC. Or less than 5k but where you'd expect to come back in the future and remember your objects/functions.

TS' biggest strengths are refactoring and not having to read your colleagues' objects/intentions for what objects/functions are supposed to have and not have. I'd have said intellisense a few years back, but nowadays with VSCode and/or its LSP(Language Server Protocol) with e.g. Neovim, the editor can give you very good hints for .js files as well.


Frontend vs backend vs devops/sys_admin

Frontend is easier to begin with and more difficult as the project grows. Engineers are predisposed to not think about the complexity in advance. There are still not a lot of standardized practices, except for maybe Angular on the frontend about how an app should be built.

Backend is much more difficult, to begin with, but once you get the hang of it, it feels a lot more consistent and stable. Easier to scale not because it's easy but because best practices have emerged from decades of tinkering that somehow do NOT change like the ones on the frontend.

DevOps, for me, requires the most upfront investment to grasp. It's a strange mixture of bad coding practices with a lot of trying to get used to commands and their options. That said, DevOps and backend feel like the natural progression of my software engineer interests. It feels like I need to study harder upfront but once I start getting used to it, it becomes easier. In contrast to frontend, it feels very easy at first, but the more I start learning how to do it properly and consistently(professionally), not just the tools, but the ecosystem as a whole, the more difficult it feels. We'll see.


What tech questions matter more than others?


FP vs OOP

TLDR: learn some FP even though you might not be using it a lot in real projects. A lot of the new ideas in languages come from FP, such as lambda functions, Promise object is also said to originate from FP.

OOP is not dead. Although it's said to be very far from what its originator Alan Kay had in mind, it's still an abstraction that is heavily used in industry. IMHO as far as one values composition over inheritance, it's not too bad. Smalltalk is said to be a good(though old, and not used in production) language to study the original intent behind OOP. Learn Smalltalk in Y minutes. FP repositories I've found interesting


Python vs bash vs Perl vs...

TLDR: Pick Python if you don't want(you/your team) to have to learn and remember a lot of new syntax. Don't pick Python if importing its standard library for very tiny tasks might be an issue e.g. some Raspberry-Pi-like embedded hardware or small Docker instances(though it's still probably fine)

Pick Bash for really small scripts, less than 30 lines of code. Or if you're proficient with it, and know that no other is going to work with your scripts.

Pick Perl if you want a powerful language that can succinctly do a lot within a few lines of code, and you don't mind that it's been used less and less in the industry. Perl can be used as a substitute for Bash because of its succinctness, and it has been used as Bash's substitute for some time in the past. Perl is said to play excellently with regex and plain text manipulations. Perl stands for - Practical extraction and report language.

Pick Zsh if you want a little better command-line completion and awareness. Bash is said that it can imitate a lot/everything that Zsh can do with plugins as well. But even if it weren't, for me, the benefits of Zsh are too few/little to deviate from something as established as Bash. Although Zsh is said to be POSIX compatible and Bash compatible, there are still differences that you have to remember. It may be worth it for sys-admins who spend many hours in the terminal.

Pick Fish, if you want to play with a shell that is better than Bash, often compared in quality to the highly praised Powershell. Not POSIX compatible.

Pick PowerShell if you're going to do Windows sys-admin/DevOps tasks. It works better with Windows(it's designed specifically to do so, while trying to avoid some of the problems with Bash such as input/output being plain next, as opposed to structured objects). IMO, even though it's said the be better than Bash, it still doesn't work as well with Linux as Bash does. The whole Linux ecosystem is expected to work with Bash(something like it). Plus the ecosystem around Bash is much bigger.


AWS vs smaller cloud providers like Linode, DigitalOcean

With smaller cloud providers pricing is much more straightforward. Much less time is spent on studying their services, more time spent building your product. With AWS and its competitors like Azure and Google Cloud, it's almost overwhelming with all their details and options to choose from, always leaving you slightly insecure if you've picked the right option.


BAAS(Backend as a service)

TLDR: as long as you don't have a lot of business logic that should be hidden from the client, BAAS should be sufficient for your needs. If you have a little business logic, you can still use the BAAS provider's cloud functions. Not good if you have a lot of data transformations with the DB data because the transformations will end up being executed in the client(usually a browser with much fewer resources than a dedicated server). You can still use cloud functions for that as well, but IMO, if you end up using BAAS with cloud functions more and more, you're just better off not using BAAS services and building your own thing, be it again deployed with serverless functions or a monolithic server.


Rust vs Golang

TLDR: start with Golang. In case you end up needing to manually manage the memory in some part of the server/app, you can isolate that part into a separate service/s(with e.g. Rust) and still build the majority with Golang, which is fast enough but way less complexity.


AI/Machine_Deep_Learning/Data Science/Analytics