Always a contributor, never a committer
Reading time: 7 minutes 40 seconds
My career has revolved around the Apache Software Foundation and its open-source projects. I’ve been fortunate to be a part of this ecosystem and learn from them. But I promise you it wasn’t a rose garden [cue the song].
In a nutshell, a contributor to multiple projects, but a committer in none. That’s me. My attempts at open source project contribution left me with bits of experience, and lessons, but they lacked that last bit of validation.
Here are my contributions in one place:
Now for the story.
What do these terms mean?
Contributor - Someone who contributes code or documentation to the repository of an open-source project.
Committer - A consistent contributor who has proven to be a shepherd of the project along with increasing the quality of the project itself. This also includes helping other/ newer contributors and users of the project from time to time.
PMC - A leadership role in the community. Driving the momentum forward for the project and helping coordinate with the Apache software foundation about general project health and updates.
Reference: Apache Software Foundation: Contributors, Committers, PMC
Background
I joined Cloudera as my first ever professional job right out of graduate school. Little did I know what I was signing up for.
It put me in the throws of open source and all these tools that are now commonplace in conversations. Apache Spark was barely mentioned and Hadoop was the big beast in the data world back then.
I saw my colleagues write code to open source and it felt like the right thing to do. I mean it was the products that my company was offering, albeit, packaged and cleaned up.
Let me tell you my story from thereon.
The reality of open source
While it is the nicest boost to one’s career and an instant rush of knowledge, being part of an open-source project isn’t trivial.
To simplify, open-source projects are projects governed and contributed to, by people who volunteer their time to do this. This isn’t sometimes respected or the most rewarding work. Writing tooling that powers the world is good but it sits on the backs of people who do this out of the pure passion for the craft.
It takes time, planning, effort, and a lot of dedication to get open source projects to be successful and adopted. You know the famous ones right, there are a bunch that still exists and not receiving their fair share of the limelight.
I, too, struggled with a few things along this journey.
Struggles
It wasn’t a singular item that hurt my open source contributions, it was definitely a collection of them.
I’m going to use the Apache Iceberg community as an example here since that’s the most recent one I was a part of. But, these points are relevant to my experience with every community.
Here is the current distribution of folks in the community. I’m one of the ones in the 253. There are overlaps in the other two since everyone is a contributor.
I used the script below to identify the Contributors’ numbers since Github’s insights only provide the top 100. The date is included to give you a sense of the timing of the numbers. I got the Committers and PMC numbers from the Apache Iceberg committee page.
#!/bin/zsh
now=$(date)
echo "Contributor script run on:$now"
git clone git@github.com:apache/iceberg.git
cd iceberg
git shortlog -s > contributors.txt
wc -l contributors.txt
------------------------------------------------------
# Output:
Contributor script run on:Sun May 8 17:34:06 PDT 2022
253 contributors.txt 
Figure 1: Contributor, Committer, PMC breakdown for Apache Iceberg
Drinking from the firehose
You subscribe to notifications and alerts to be part of a community. The Slack channel, the email list. The email lists typically involve information about issues, a dev list to discuss topics, a commit list that tracks commits, etc.
As a newbie when I joined, I signed for all of this. It helped in getting to know things, but it was overwhelming. How do you know what to read or what to dive into when you are new.
Most communities have a “good first issue”, as a tag for their issues that can help you get a sense of something to work on. But, there’s no overview to start from. You find your way and somehow get to the point of throwing code at the issue at hand.
If you can’t keep up with the pace of things, you tend to get left behind and lose traction with the community. And remember, you aren’t alone in this community.
Competing with others
Yes, there’s competition here too. There are 2 kinds of competing forces.
Other contributors
There are much more non-committers than committers as you see in Figure -1. You are one of them searching for your piece of the pie.
You try to jump to contribute and, more often than not, there’s already someone who is either interested or has already begun the work.
Not everyone gets the shiniest piece to showcase their work. You might end up being unnoticed while your peers shine.
Open-source developers from companies.
These are the real deal. They get paid to do the work you do in your free time.
Iceberg has hardcore contributors/committers from Apple, Netflix, and Tabular who work and breathe the project. They are the driving force of the project, and it’s difficult to compete with them.
While I was doing things in somewhat my spare time, sometimes on weekends, I would see a barrage of activities from this group of people that made me wonder if it were worth the effort to work on this.
Relevance to work
Another piece that adds more complexity to this journey; is the open-source work relevant to your job?
I jumped onto Iceberg only when I had to stand it up internally. It helped me orient and understand the ecosystem by being in the mix of it. But you might not have this perfect fit. And I don’t call mine perfect either.
What the relevance to work brings is the actual need to stay in touch and be engaged with the community. If your work depends on your commits to this project, it helps you the most. But, if you are doing something one-off or just in your spare time, it can fizzle out or you will burn yourself out.
This happened to me with Apache Beam. I was interested in it but my work never pivoted to using it or seeing it as useful.
Consistency
A person I consider a friend/mentor advised me to be consistent. And this was a while ago (2015) and the project was Spark back then. You guessed right, I couldn’t be.
My work back then had such a ton of context switching that I couldn’t keep up. I was burned out by the end of each day. I tried on weekends but I really hated that too.
I even had a plan of doing a JIRA per week, but I never managed to do that. ☹️
I might have got 1 fix into the Spark project in that period. It definitely needed much more push than I could give at the time.
In hindsight
I reflected a lot on this. Not just while writing this piece, but over the years, I thought about what could I have done differently or better.
Better planning
I wish there was a magic pill for this.
Jokes aside, I wish I structured my time better. It could have been worthwhile to give a dedicated set of hours a week, consistently, for a few months to see what comes out of it.
It takes rigor and discipline to have this, and I think I still lack that in some regards. I want to get better. Planning my day and task management are helping me currently. But I do have a long, arduous road ahead.
Being patient
Building trust takes time. Building trust in one’s skills and decision-making prowess takes a lot of time. I wish I were patient enough to understand that important piece.
You can’t lose 10 pounds in a week - at least not the right way. It’s a slow process that can’t be rushed.
I guess I got carried away awaiting the result that I forgot to put in my time for it.
My own project perhaps
This is my ambition now. To start a project of my own.
I’ve thought about this, I have something in mind, and I do want to make a dedicated effort to push something out.
Don’t hold me to this. I might fail. But it’d be great if I succeed.
Final Thoughts
It wasn’t all bleak in the years that I have done open source. The silver lining has been the communities and learning I managed to get and retain in all my adventures. I’ve met and interacted with easily the smartest people in the industry, and I have learned from their collective experience. I fondly remember a dinner in Seville, Spain with Beam folks back in 2016. :)
But, back to my journey, I wish there was a light at the end of the tunnel for this story. No happy ending yet. Someday, I hope to have contributed something more tangible to the world, but that remains to be seen.
If you are jumping on to open source, try to focus on the right things. I’d hate for anyone to struggle as I did.
Don’t let this deter you from throwing yourself into a project. It can be chaotic, beware. If you know what you are doing, more power to you and good luck.
Until the next post, thanks for reading.
Stay encouraged.