Navigating the Shadows
Some people think programmers should only do what they're told—one ticket at a time. Nothing extra.
Scrum has inscribed this sentiment into their official guide: the Product Backlog “is the single source of work undertaken by the Scrum Team.”1 And most engineering managers are careful to keep programmers locked out of the backlog—except for pulling and completing items exactly in the order they're given.
But, if you take this sentiment seriously, you're stuck. You're doomed forever to march to the beat of someone else's drum. The fast track to an unfulfilling career.
Thankfully, there's a simple way to break out of this trap: the shadow project. A shadow project is work you do off the books, on your own initiative. It's something you want to do, for your own reasons.
When you get a hankering to fix something, try something new, or learn something that might help you down the road, fire up a shadow project. Don't ask; just do it.
Forget the backlog.
Luckily, programmers are uniquely equipped for doing shadow work. We don't need special equipment or capital to run our experiments. We can run them on the computers we already have. We don't have to present the ROI or ask for budget to acquire new machinery. We are free to try anything we like, and with so much free software available, it's often as simple as typing 'apt install'.
Of course you'll be picking something that helps the business too. But that's not why you do it. You do it to better your position, make your life easier, or increase your future prospects. Backlogs focus on what other people want. Shadow projects restore the balance. They let you take the wheel for awhile and establish mutual benefit—yours and the company's.
It probably goes without saying, but while you're working on a shadow project, you don't tell anyone. Don’t put a ticket into Jira. What they don't know can't hurt them, right? I know it probably sounds subversive (to be honest, that's part of the fun). But is it really? You're working for the good of the company, in a way that will motivate you far more than working from a never-ending list of tasks. This way, you'll be far more productive.
Just because your manager doesn't understand this concept, doesn't mean you should rot away at your desk updating Jira. Doing shadow work is like pretending your manager actually does know how to motivate you ("Sure! I'd love to research a new language. Thanks!" "A little time for some clean-up work? Don't mind if I do.") and delivering the superior results they don't deserve. Who knows, it may even get them that promotion they are in no way qualified for.
Here's another way to look at it: shadow projects are a way to diversify. If you only do what you're told, you're putting all your eggs in one basket. You're counting on your employer to direct your efforts in a way that will be good for you long term. You're totally dependent on them. I'm much more comfortable taking my fate into my own hands and working on several fronts at the same time. Some sanctioned stuff and some secret stuff. If the sanctioned route works out, then great (for whatever reason, that's rarely the case for me). But if it doesn't, I have other irons in the fire.
Additionally, shadow projects fill an organizational gap. Because of their narrow focus and lack of context, backlogs leave many important things undone, things that programmers are acutely aware of. When you do shadow projects, you're naturally drawn to those neglected areas. By doing yourself the service of addressing things you find painful, you're actually doing the organization a service as well. These are exactly the things your organization needs to address.
Shadow projects can be anything. Sometimes they're directly related to the customer: a new feature or a GUI to hide a tricky command-line. Sometimes they focus on making your day-to-day life easier: some elisp code, a new plugin for your editor, or something to make Jenkins go faster. What's important is to trust yourself to be the guide. If you feel pain, it's relevant. You don't have to stuff it down anymore.
Another reason shadow projects should be exciting to you personally, is so they are interesting enough to motivate you to do them (secretly) alongside your normal work. Many of my shadow projects are moon shots—ideas that are radical, but if they work, they can make a dramatic difference. I've found going after big things is the best way to keep me interested.
Example Shadow Projects
For inspiration, here are a few shadow projects that have turned out well for me:
Using vagrant and/or docker to create development environments that match production, which eventually turned into a way to create repeatable production environments. (This is common now, but back when I started trying this approach, it was more radical)
Exposing C code as modules to other languages (ex. gambit, python, C++, zig) to make the system more scriptable—especially useful for writing tests.
Building an A/B partitioning scheme for an embedded Linux device. Each update is an OS image that's installed to the inactive partition—which then becomes the active partition.
Converting complicated makefiles and visual studio projects to cmake, which can then automatically generate makefiles, visual studio projects and much more.
Building gameplay components for level designers in a visual scripting language they were already using, to allow them to design level functionality themselves, instead of having to rely on developers to script it all.
Using python to generate C code for state machines instead of coding them by hand.
Investigating literate programming tools to help documentation and code stay in sync, which eventually led to an exciting open source project: https://github.com/adam-ard/organic-markdown
Adding parallelism to Jenkins builds to take them from hours to minutes.
Looking back, I realize some of my best projects were shadow projects. Lots of good feelings. I'll spare you the details of my failed efforts (but even failures teach you something).
Here are some additional things to keep in mind as you navigate the shadow lands.
Stay Strong, Stay Quiet
With shadow projects, it's rarely useful to tell anyone what you’re doing until you're completely done. That way, if it doesn't work out, you don't have to bring it up at all. No one's the wiser, and people don't start getting the idea you're addressing your own needs concurrently with their priorities (even though you are).
Remember to make at least a little progress on something in Jira every day so you have something to report in morning stand-ups. Otherwise, people might start asking questions and you'll be forced to reveal your stealth project. At which point, they'll start a pressure campaign to make you stop. ("We don't have time for that right now." "Can you make a ticket for that, so we can (de)prioritize it?" "Let's get your current tickets done first.")
Once a shadow project shows promise, and you have some real working code to show off, then you can take a stab at sharing it—perhaps first with a trusted colleague or maybe more widely. Whatever you think gives your idea its best showing. Trust your intuition.
How much working code you need before sharing depends on your goal. How hard is it going to be to convince other people that your idea is viable? If you're dealing with more stubborn coworkers, it might be best to wait until you've implemented a lot of it. Then you can say, "If you don't believe me, here it is. I already wrote it."
If you're dealing with coworkers that are easier to engage, just a little code or research might do the trick. Every situation is unique.
A Cup of Rejection with a Pinch Glory
From the beginning you should expect that nine out of ten shadow projects won't go anywhere, at least not at first. You either won't find them worth presenting or they'll be rejected (by your boss, the team lead, and/or the rest of the team). That's OK! Just file it away for now. You'll have chances to bring it up again down the road—especially if the problem you're trying to solve is painful to others as well. Also, now you've established yourself as a pioneer of the alternative path. People will remember you. They'll come to you in the future when they're finally ready for a change. But no matter what happens to the project itself, you'll retain what you've learned and be a better engineer for your effort.
And what about the one in ten ideas that do get accepted? They can totally change your career. When you manage to get people onboard, opportunities open up. As a result of shadow projects, I've been offered positions on other teams, shifted to new specialties, been saved from getting fired, and maintained connections that have led to future opportunities. In all honesty, I don't know where I'd be in my career without my shadow projects. They've made all the difference.
Some Honest Talk
I'd be lying if I said that shadow projects don't come with some risk; you never know how management is going to respond to someone who goes rogue occasionally. The worst case scenario is you get fired. Most of the time, though, that doesn't happen. Someone who does extra work, for the benefit of the company, is hard to get mad at—even if they are a little subversive. Most managers are smart enough to appreciate free work.
This is how I think about it. If a shadow project gets me fired, I am likely working for a control freak. It's unlikely I'll ever be happy working for a control freak, no matter what I do. So, as difficult as it may be in the moment, getting fired is probably my best outcome; it gets me on to working somewhere with more potential. For that reason, shadow projects are a risk I'm willing to take. I realize that not everyone has the same risk tolerance as me though, and some may see a rare loss of employment offset by occasional moments of shining glory as an unacceptable trade-off. I respect that. The goal is to make work better. If a shadow project is just going to tie you in knots, it's probably not for you.
Are you game?
But if you're ready for an adventure, and your job feels like an endless stream of uninspiring errands, shadow projects might be your thing. They’re not just a way to escape monotony; they’re a way to reclaim agency in your career, sharpen your skills, and discover new opportunities. So proceed with caution, embrace the challenge, and enjoy the ride—because sometimes, the best work happens in the shadows.
I’ve never seen shared shadow work get rewarded, and in environments where they reward individual local optima and # of stores completed in a sprint (instead of innovation and teamwork), you may want to keep your shadow projects to yourself and use them to beat out other workers metrics-wise.