Giving back - how I cleared L6 System Design - Part 1
Long term prep strategy Part 1 is out: https://www.teamblind.com/post/YA2RKEqY Coding Part 1 is out : https://www.teamblind.com/post/ikvVrRXa Part 3 is out : https://www.teamblind.com/post/qubF6fS2 Part 2 is out : https://www.teamblind.com/post/rBrt5bV8 Edit 6: I will be posting the rest/commenting as user flung2 from Red Hat going forward. Edit 5: I know I promised to reply to every comment/question but I left it at 90 comments and now it's 450 and grows faster than I can reply. I'll have to be selective. Sorry. Edit 4: I will be posting the links of part 2 of SD and future coding post here. Edit 3: "This is booky knowledge and not real life experience stuff". DDIA is condensed experience. Nishtala et al are as close to *relevant* real life experience as it gets. In fact, I had the feeling some of the experience of my interviewers was irrelevant experience in scaling glorified CRUD apps which had left them stale on knowledge. It's a balance, but I stand by my hints here if your goal is to clear the SD interview. This method got me 3 SH and 1 H in F/G interviews so like it or not, it gets the job done. Edit 2: Some people feel this is too much detail and you never get asked these things. Wrong. It's all up to you to steer the conversation. The question is pretty vague. The key is to identify unassisted the main pain point(s) of the problem and once you do, dive as deep as possible after you cover the high level design. It is not uncommon that the detail you waved away turns your high-level design to dead end. Everything that I mentioned as examples here were actually brought up in my interviews. Less preparation could get you there but, as I mentioned, I go for the jugular. Edit: Throw your questions in the comments. I'll get to everyone of them eventually, I promise. DMs also are welcome but it needs to be something you can't put on the comments. 1. Intro I spent 6 months preparing overall. In the evenings before going to sleep I'd spend some time on Blind to get my hopes up and see a joke or two. There are some assholes here and there on Blind, but the majority are good people. I love you all. So now that I'm 'across the river' I decided to give back and provide some hints specifically on the System Design rounds. I interviewed at Amazon, Twitter, Google, Facebook, Databricks and Elastic for L6 (or L6 equivalent). I cleared them all with exception of Twitter. I cleared Google and Facebook first then had 'emergency' interviews with the rest of the companies to ensure I was well-placed for the inevitable lowballing dance. One Friday I had half-onsite with Elastic in the morning and half-onsite with Twitter in the evening. I completely messed up the System Design for Twitter and that killed my prospects. I regret that though, as I liked Twitter. I am not a genius, I'm just a hardworking tenacious guy. Don't let anyone tell you you're not smart enough for L6. It's all about mustering the motivation, channeling the work and executing it well. I made the decision to try 6 months ago, and I prepared throughout that time while being at work. So my preparation hints will suppose long-term prep. If you're expecting a two-week crash course this is not for you. I'm conscious some of this intro will read like a flex - it is genuinely not - but I'm sure the context will be helpful to many so I decided to make that tradeoff ;) 2. Content Where do you start? Get the Designing Data Intensive Applications book. This will fill your theoretical gaps. Read it slowly. *Do not make fake progress*. If you don't understand something stop, use the references, research the subjects, get out of the book and back. There is *nothing* useless for you in that book. Nothing too much. It's all for you cover to cover. Properly grasping that book is half of the whole work. Part 1 is super useful to teach you how to pick data stores. Part 2 will dispel your fears of sharding and choosing a replication mechanism. Part 3 will give you a full idea on how to piece a big system together from smaller systems. The separation of System of record and derived data is key to understand there. When you think you're done with DDIA, if I wake you up in the middle of the night you should be able to explain to me how LSMT-based databases use Red Black Trees to keep a sorted log in memory and why they do that, and then get back to sleep. I'm serious. Next up, pay for Grokking the system design interview. I know that Grokking can be too shallow. Nevertheless, I want you to be able to recite the solutions in Grokking down to the smallest details. Details are more important than the big picture for L6. Do you think your done with typeahead suggestions? Did you mention exponential moving average on how you weigh results? No? Failed. You can leave the pastebin and bitly and that easy crap uncovered, but I'll need you to recite the other grokking solutions to the smallest details. Take extra care on the geospatial problems. If I show up at your lunch and ask you about quad trees I want you to be able to estimate the index size on the spot. You can't? Not ready yet. Next up: videos. InfoQ and @scale videos about real life systems. Look up Nathan Bronson and suck up all his videos on Tao, Caches etc. Look up Nishtala and the memcache video. Look up Kulkarni and the Facebook live video. Look up Susheel Aroskar and his Zuul push video <- ultra useful. Again, never make fake progress. Take this last one: Aroskar's Zuul push. He mentions Apache Netty. Read up on it, understand it. Go deeper, understand epoll and its red black trees. Go deeper and understand the TIMED-WAIT trick in TCP protocol which saves some web sockets connections. Another thing: he mentions some load balancers can't handle websockets. Why? Go figure it out. I did and I impressed my Facebook interviewer as I went deep until I was stopped as the interviewer lost it. L6 means depth, depth, depth. If you draw me a block that says 'Server' and leave it at that I'll slap you back to L3. Now I went the extra mile. But I wasn't targeting barely making it - I was targeting mind-blowing performance. If you want the hardcore stuff, let me know and I'll give you content for that as well. If you're up for anecdotes, a Google interviewer challenged me on Paxos when I mentioned Spanner to him. I drew the multi-Paxos detailed flow for him with estimated latencies supposing a 5-replica configuration with the pivot replica on north-east US. He smiled and said 'ok' (Strong hire). Next up, the Google SRE book. Did you say you're not interviewing for SRE? Don't care. You don't need it all. You do need the chapter on Non-Abstract Large System Design. You need to be able to recite that in your sleep. Especially the estimations part. You don't have NALSD interview? *rolls eyes* Don't care, learn the NALSD chapter or fail. 3. Speaking of Estimations Ultra important. If you can't handle these, you're most likely ducked. How do you prepare for them? Practice power of 10 calculations. See how the NALSD chapter does it. Practice with fake numbers. Drag the units with you and reduce them when dividing/multiplying. When calculating storage, consider cold storage. Also consider e.g. cassandra needs ±30% free space to do compacting (see, I told you to properly learn LSTMs in Part 1 of DDIA), also keep in mind that only ±85% of the disk space is usable (OS files, formatting tables etc.), also keep in mind that 3-5% of disks die in a year so account for that, also keep in mind that you need to multiply by replication factor, also keep in mind that cassandra says no more than 2TB disks otherwise it gets slow. Have a strong grasp on the -bound concepts. Is something storage-bound (Log dumps)? Is something cpu-bound (live stream encoding)? Is something RAM bound (in-memory timeline serving)? Once you grasp those you design tiers with separate scaling techniques. That's why you need numbers, not to show off your math but to figure the -bound part and then decide how to scale a specific tier. I'll wrap up this part here. In the second part I'll get into some sample walk-throughs of some popular questions. Then I'll give a you a detailed plan on how to spend the 45 minutes. I did 9 System Design interviews in the space of 5 weeks so it's fresh in my head. Now you might think this is a lot of work. It is. It depends on how much you want the job. Want it badly? Throw in the effort then, that's all it takes. Was it worth the effort for me? Yes. Old TC: 175K USD, New TC: 593k USD #SystemDesign #Interview #L6 #E6
This is actually an inefficient way to study and a huge waste of time. Learning details like this is useless because A) they mostly don’t get asked during interviews, B) engineers who have production experience can figure out that you just memorized some generic bullshit in a second, C) it prevents you from using creative thinking.... The better way to study is to learn big ideas and learn as much detail as long as it feels natural to you. Then use raw brain power to solve problems on the spot rather than regurgitate crap you memorized.
Good write up. I’m not sure what your current position is, but given the TC it seems it might be quite a bit lower than L6 at a FANG. If I was you, I’d be slightly concerned about whether I can do the actual job. You seem smart, but the job at the L6 (L7 at Amazon) isn’t just about being technically smart, it’s about leadership. If you haven’t come from a role where strong leadership was required, I’d try to focus on that a bit. It’s unfortunately hard to do with just books. The principle level you’re talking about isn’t really for the most part about solving hard problems yourself, rather enabling others to solve those problems. Doing that takes experience. Keep this in mind when you join.
encourage the author, if he has done so much in 6 months, he could do much more. All it matters is hard work
I would agree here. Expectations from IC6 are high at Fb. How could one learn these things? Any good books?
How did you learn? from books?
Mind blown, so you didn't have strong prior experience in distributed systems and yet cracked all of this? This is fking dedication.
Are you not scared though to compete with fellow L6 folks who come with strong experience in your next company
Limited experience, yes. I'm not scared at all, the people that interviewed me were L7 and they didn't look too scary. I can handle them.
Congratulations OP, you deserve it. Any suggestions on all papers you read?
Great post! Thanks for sharing. You mentioned hard core stuff. Could you elaborate more on what you did? Was it along the lines of reading white papers, implementing the tech in your pet projects, etc?
Properly digesting Lamport clocks, happens-before relationship, ordering guarantees, original Paxos paper, Nishtala et al. memcache paper, Bronson et al. TAO paper, Jeff Dean's Spanner paper, various storage sizing papers from Google, test drove Netty, Kafka, wrote performance tests between epoll and select/poll, tweaked internals of cassandra to see difference in performance etc.
What tweaks did you make to Cassandra? I once dove deep into distributed atomic counters class and I’m s*** scared now lol Did you run them in local docker containers?
Cassandra (and most other LSMT-based DBs) uses a compaction strategy to tidy up 'messy' SSTables. I went in and wrote a compaction strategy from scratch, similar to leveled compaction strategy. Completely useless in practice of course but I wanted to see the pieces of the system moving. I just ran it on my local laptop, no containers. It got messy, but I got what I wanted and then wiped the whole directory. I still remember the fan sounding like a tornado while I was load testing it ;)
imo 80% of DDIA is too deep for most interviews, including E6/L6. The SDI is an art, and a big part of it is communication and telling a coherent story. It's knowledge you need to know, and you need to know *something* about the ideas it brings up, but many chapters are just way too detailed. If you start rambling about LSM trees or tombstones, you are probably detracting from your overall design and ideas. There's deep, and then there's too deep, and DDIA is often just too deep to tell a nicely packaged, believable story.
You are correct, but I think OP said to learn them to have concrete ideas about designing storage for large scale data, so when you share your design you will have confidence to defend your design choice by going deeper if needed
It’s not about just mugging up the concept and regurgitating in the interview. Reading it cover to cover like OP said gives you enough material to build a nicely packaged story. OP didn’t start talking about LSM trees right off the bat he only brought those out at the end when he needed to go deep.
SDI is not about telling a story. It's about showing that you can talk the talk on the high level design and then walk the walk on the specific pain points that a specific problem carries. To do the latter you need depth. To have that depth you need solid a theoretical base which you get from DDIA.
First post that talks about going in depth Love it. Please post more resources for folks like me. Highly appreciate. Alternatively cab you make a post on Leet and share here it's easier to use LC discussion.
Glad you liked it. If people like this stuff I'll post about my coding journey as well.
OP, don't want to miss out on future post. please link the new post on this post once your coding journey is published. This is superhelpful
Will do
Interesting, I guess you had a plan, so for these 6 months how long did you LC/study daily? Did you plan to cover X problems and X parts of system design every week or how did you monitor your progress? Great post!
Yes I had a plan. Actually I had a system in place which I perfected throughout the months. On average 2 hours per working day was spent preparing, no exceptions. You can always find 2 hours. I should probably make a post about that as well as it's hard to fit in a comment. Will keep you posted if I do.
Would appreciate it OP
2 hours a day? That's doable. Thanks OP! You just inspired the crap out of me. 2.5 YOE, $180k TC Making me think I can get to $500k in the next 3 years or so.
Amazing, such a confidence booster with clear steps to follow, really appreciate it.
Boosting the confidence of people that feel down, incapable, not smart enough, not enough of this, not enough of that is the main reason I took the time to write this. Chin up, heads down and it's doable.
Thank you brother, it’s very rare to see people like you — don’t blame others though, it’s human to have this feeling of exclusivity that stops most of us from helping others. You are the best!!
I can't do this 🙏 I am happy with my low TC
Aren’t you working in microsoft
When your ambition picks up and you decide to do it, the post will still be here ;)
@Robert Microsoft is not known for great TCs
As someone who is actually familiar with Cassandra - you just failed my interview. hard. And you are not the first. Unfortunately, you can’t replace real world experience with just a book.
agree.. the theory is sometimes way different than what happens in real production systems
but without the real world systems knowledge this is the only way to go.. read as much as possible.. do you have any suggestions ?
The author hacked the interview process which I’m not a fan of. They did it by demonstrating knowledge that associated with experience, so good for them. The question now - will they be able to keep above the bar after they are hired. Interviews are 5 hours, work day is 8.