5 lessons from 5 years of building data infrastructure
Reading time: 4 minutes 36 seconds
I honestly didn’t think I’d stick around 5 years in Stitch Fix. Let alone in the same team. But I did. I want to share what I managed to learn and retain during this time.
Background
Back in early 2017, I joined this 3 member team building Compute Infrastructure for the Algorithms organization in Stitch Fix. It was a different role coming from my customer-facing role in Cloudera.
The mission statement of this, somewhat newly formed team, was to build tooling that made it easy for Data Scientists to access and use data in the organization. Not to mention, making the tooling easy to use, reliable, and, available.
There are a bunch of things I learned along the way. These aren’t specific to technologies, but generic lessons across my tenure. Allow me to share the 5 important ones.
Solving problems the right way
When solving a problem in Data, seldom do we wait and ask ourselves, should we really be doing this?
I’ve learned this after a series of tried and tested attempts at solving problems in a certain way. You can’t be right, but it helps everyone if you are.
What helped me get better:
Asking a lot of questions about the problem context. I asked till it made sense to me.
Setting the right expectations at every step of the way.
Honoring the expectations and being open about the progress or sometimes lack thereof.
If anything changed, it was best to get a consensus before moving forward.
Emphasize on Quality
Cranking out code at record speed sounds great. Build fast and break things, right? But, what about quality? Technical debt? The constant worry about having to maintain poor quality projects.
Sometimes taking a step back to focus on quality really goes a long way. It prevents unforeseen disasters and also sets a precedent for operating well as a team.
How do we do this?
Having discussions for every design and approach for a problem. This helps create a sense of accountability and fosters a healthy technical discussion between peers and others.
Enforcing a set of standards for codebases. Like using formatting tools like black in Python and ensuring that test coverage is present in any code that is meant to be checked in.
Doing a retrospect on completed projects to understand what could we do better. This helped us understand and not repeat mistakes.
Communication
Not just how you speak or discuss things. How do you send emails, how you respond to questions, how you give feedback, everything.
This is a crucial part of the role that no one tells you when you start. Engineering involves a lot of communication than merely pure coding. I learned this early on that it was an important skill to master.
What can help communication?
Listening and then responding. It helps to really understand what you are being asked to respond to. Be slow, not abrupt.
Give the reader/ listener the most important message first and maybe skip or park the details elsewhere. Not everyone is interested in the nitty-gritty. Give the bullet points and add references for the rest.
Keeping an open feedback channel. Let your customers/ users communicate freely with you and vice versa. This helps in understanding situations and being quick to respond.
Be open to hearing the good and the bad. Not everything might be pleasant. Listen, learn and adapt accordingly. It’s important to be emotionally intelligent.
How to manage oneself
As I grew as an engineer, I reflected on my 1:1 with my manager. They started off with tasks being handed to me and me being told what to work on. And later they became more self-driven and my chance to capture/update on the important pieces.
This didn’t happen overnight. I gradually learned how to manage myself. It took time, effort, and testing.
What I did to help me over the years:
Starting to plan my tasks on my own. Breaking down a project into individual items is what I did on my own after a while.
Coming prepared with answers to problems rather than listening to problems. I looked for problems and came up with suggestions rather than merely pointing out that something needs fixing.
Understanding my priorities. What needs to be done now versus later was crucial to my growth and self-learning.
Asking for help sooner than being isolated. This unblocked me a lot more quickly than trying to break my head fixing something that I might not have context on. Don’t be afraid to ask dumb questions. They tend to help.
Helping in planning the quarter for myself and the team. Giving suggestions and ranking problems for an upcoming quarter were helpful to know what to work on.
Empathy
Finally, a thing that goes unsaid in software development; empathizing with the user/ customer helps an engineer understand where to build better software.
Will this extra line of documentation help users or confuse them? Can they reach this reference easily? Will onboarding to this new style of software be easy or difficult? Asking some of these questions to yourself while envisioning the impact of what you build can help steer you in a better direction.
Here are some things that helped me:
Understanding the real pain that affects users/ customers.
Do they hate hours of debugging? Yes, everyone does.
If their lives could be easier with one change, just do it.
Surveying users to keep the finger on the pulse. It’s hard to retroactively fix something, but it would be great if you had the information to do it.
Reading bug reports carefully to fix things the right way. What is the sentiment of those sentences? Positive or negative? React accordingly in the fix.
I’m hoping empathy becomes a software engineering principle someday if it isn’t already.
Until the next post, thanks for reading.
Stay encouraged.