Yeah, that linked article is a good summary of why just doing browser-side rendering is a problem for performance and the associated UX. Particularly on mobile devices, which is a lot of your audience these days likely as not.
Anything that gets trapped by the Request > Parse > Execute > Render cycle can be a problem if you don't happen to be on a top-end developer machine. Even having cached JS, means you just skip the first step of that. There's a perf hit on time to interaction, one devs often don't see.
Do some user testing with actual people on real setups and you'll see this more clearly. We actually integrated with a competitor's API recently, our plain old server-created HTML solution is orders of magnitude faster to use than their Angular-based client side solution. Users notice that speed and have a better reaction to the app.
These things always vary though. It depends what your site does and how it's used. Nothing is absolute. It is worth checking if your setup can do a server-bootstrapped render, isomorphic is a keyword to look for.