USDS Digital Services Playbook
playbook.cio.gov
playbook.cio.gov
Excel macros working one day, then prohibited the next by new IT policies.
Directors demanding that service line bosses meet with me to discuss "data-driven decision-making", followed by a solid week of said chiefs telling me that my job is a joke.
Actual honest-to-God employees trying to make a difference in financial openness having their coworkers walk by their desks and loudly shout they "ain't doing that shit" and walking away.
Two years hunting for a database to store 2 GB of data, and only finding one when a new contractor onboarded and knew a guy in the next office over who would let them have a partition of SQL Server.
I wish I was making any of this up. The federal government has massive, massive employee culture issues. This list is... a nice dream.
Most importantly, USDS is seen as non-partisan, and so has been able to have good impact in both administrations.
Interestingly, the 21st Century IDEA Act mandates compliance with the USWDS now (at least for publicly facing sites) and there is a maturity model built around it. We’ll see whether or not agencies comply, but it’s a step in the right direction.
I'm new to the system, so this is the only one I know (wrote about joining here: https://twitter.com/abachman/status/1217795232449859585), but I'm currently on a team of 6, all feds: 1 designer, 1 product mgr, 3 engineers and I'd say we're fairly to extremely self-directed. User-research based product development, pair programming, short iterations, deploying daily, all that jazz. Just a variation on the same sort of thing I've seen in the tech-first / software-product-company gigs I had in the past, but I've mostly worked for smaller indie or niche companies.
That is, I get to work in a way and with people and tools (Rails + React at the moment) that makes sense to me. How far outside the norm are we and where could I go to learn more about the whole system?
It kind of sucks being on the bidding end, as their tech challenges cost a ton on top of the normal costs with a proposal. But I can understand why they do it and from an acquisition perspective, it makes total sense. I think they will need to strike a balance at some point though - some of their tech challenges are too complicated for the timelines provided.
In terms of development practices, it is pretty much all DevSecOps and they make heavy use of independent verification and validation/QA, and even more importantly, are pretty mature in their adoption of it.
So to answer your question, you guys are pretty far outside the norm in a good way :-).
Not sure the best way for you to be able to learn more other than to do some networking in the DC area (post COVID). If you get the opportunity, there’s a ton of great meetups in the DC area, although if you’re taking the MARC those can be a challenge to attend after work due to train schedule limitations. Other than that, move around to/visit different agencies and get exposure to their work. Work with your PM to see if you can do some site visits to other agencies/organizations as part of professional development and such.
I was on the cusp of attending some cross organization "for feds" tech meetups back in March, but... :| I'll keep on the lookout.
My kinda of place... no stress, drama (for chill .. go with the flow types), nice co-workers and you can pay the bills & then some!
For example, here's a project started by the USDS in 2015. It's responsible for managing the complex bureaucratic process of VA legal appeals. (https://github.com/department-of-veterans-affairs/caseflow) It's open source, continuously integrated and deployed with close to 100% test coverage, deployed on an AWS GovCloud VPC, and was built on a fraction of the budget of similar systems.
Active development on this system is now done primarily by contractors now that the relevant bureaucratic barriers were removed.
How do we scale this story? More talented people with the desire to serve their country. It's not easy, but it can work.
Most reasonable people might wanna work for their government for a 20% discount in either, but my fear is that it’s much much more and in both.
So many people complain about the government, so few are willing to fight to fix it.
IMO, USDS makes this worse as someone like Matt Cutts has the cachet to bring in the playbook and get execs to listen. Whereas, I’ve been doing this for 3 years and routinely have to fend off coup attempts as the term “mvp” sounds like a joke to other career employees. As such, it’s often easier to check the box of the federal acquisition requirements and collect a paycheck versus hustling to build a vanilla web app using JavaScript a rest api and Postgres before the checkbox folks shut you down because the baselined version of Jenkins is 5 years old and you installed version 2.26x without a request for change.
So how do we scale this story? Inform politicals and bureaucrats that incentives matter, successful tech start-ups are exception not the rule, sunk costs are real, and you don’t need a catchy name like 18f nor the catchet of celeb dev to right a failing project.
You're totally right that if the incentives don't change, the choices made by execs won't change either. For example, a contracting officer I've met saved their agency a billion dollars by not continuing a failing contract, and was punished (to the point of leaving that agency) because that wasn't what the incentives supported.
How can USDS/18F be helpful to you and others in your position? The messages you list at the end are all true.
The list is a nice dream, and it is with the will of those in the federal work force who can break down those silos and bring about a common understanding with urgency that'll make it happen for their project.
Much of what you describe is not unique to the U.S. government. I've run into many very similar scenarios in the jobs I've had over the years, from mid-sized travel company to billion-dollar healthcare companies.
Also used to do govt work. I'm sure everyone has stories.
My takeaways.
1) Understand what people need -> AND LET THEM DO IT.
2) DO NOT ADD A SINGLE NEW THING with the IT / automation. If the old system doesn't have it DO NOT ADD IT. No 20 extra fields for demographics if you didn't track that before. That can be added later IF it's a MUST.
If they would take away all various fifedoms and hassle - people actually would bring a lot more tech into gov. Let the local business / division people make a decision. The only requirement be that they go through a 1 hr training and be exposed to 3 packages in their space (ie, do demos no powerpoint).
Take away all other requirements. Be OK with smaller failures. Govt is so scared of just letting folks try stuff out that everything turns into a 100M project that goes totally of the rails in terms of scope etc (ie, what people need / want is long forgotten, it's what MANAGERS want that gets emphasized).
Result - even if successful (rarely) for downline workers the software brings TONS of extra (not less) work.
----
My own examples. The IT folks say that passwords have to change every 90 days. Google for example does not default to this (stupid) rule - they push two factor without SMS (which is better). Bam - you are out or have to pay for a "security" solution on top of whatever standard platform you are using with admin rights that can force rotate passwords (and is a MASSIVE backdoor itself vs self-serve password mgmt).
Then purchasing. We need an ipad to edit some videos. OH, Apple is not supported. This is 10K employees but you can't get an apple product to use unless the city atty will sign off on a variance (no chance).
This is explicitly not the NIST recommendation (the group in gov that sets some security standards), but the word is not getting out quickly.
The current IRS requirements are 90 day password changes. Ergo - many people write their passwords down in a text document next to the software launch icon.
What I don't get - if your computer is hacked, and you force people to write down their passwords on the computer being hacked, they will even more easily be able to access the systems you have access to.
Google seems to get this right. I have had same password for 20 years, if I login with a new device I use my MFA (no SMS). If I do a security sensitive op I need to login and do an MFA again (password reset etc). I imagine they actually monitor and rate limit bad login attempts etc. A 10 character password is really fine then in my view as an example.
Government technology workforces are thus composed largely of either people who are below average in competence or above average and willing to take a pay cut because they care about the cause, but we shouldn't expect people to do that, and there aren't very many of them. We should give government a fighting chance.
That said, you raise an important problem - how do you manage poor performers without allowing a bad-faith political official to fire people unfairly?
The playbooks aren't wrong, but it's silly to think they'll actually fix the issue of technology not being a government's core competency.
Your premise that the government should just contract out tech is bad because it already does contract out the vast majority of it. And the results are still bad in a lot of cases.
This playbook has become something that a lot of contracting companies that specifically focus on doing good dev work specifically call out in their websites and marketing. Ex: https://digitalservicescoalition.org/#/story
The contracting process itself is an incredibly onerous burden with a huge number of factors other than mere technical ability (are you a disabled veteran minority-owned business?). And the companies with the fortitude and resources to actually make it through the contracting process tend to be the exact companies you don't want doing the work - it's not your scrappy 30 person startup that gets the bid.
Furthermore, those companies are the only ones that can, frankly, put up with the government's shit. The problem isn't just that the government doesn't have the technical ability or the culture to get the job done, they don't even have the ability to oversee a technical contract, and it inevitably results in weird and obtuse requirements, stumbling blocks injected from the contract coordinators, etc.
You can sometimes have a setup where one contractor executes and one contractor oversees, but anyone who's worked on a project overseen by one of those contracting management companies (DeLoitte, Accenture, etc) knows that doesn't fix anything either, and the contract coordinator isn't going to stop their bullshit either, now you just have twice as many layers of management shitting things up.
There's no magic bullet here, but management and IT services within the USG drastically need a shakeup, and increasing pay scales to match the private sector has to be there too. On top of that you basically need a top to bottom review of the regulations and SOPs. It's simply too hard to actually get work done, at every level. There is too much security theater, too little real security, too many fiefs, too many bosses, too many approval processes, too many dumb requirements that were coded into law 20 years ago when they maybe made a little sense but have long-since ceased to be best practices.
Also, contracts are for a specific and limited period of time. Often development and maintenance are under different contracts - which will be bid out separately. So the first contractor doesn't really care about anything except getting it out the door, then the USG will be super surprised that they're pulling the plug once they've met the bare minimum of the requirements, and then there's no money to actually maintain it on an ongoing basis. If there is, that money will be bid out again and probably goes to a different contractor.
Keeping it internal at least means that somebody has ownership of the project. Contracting turns everything into a game of hot potato with everyone racing to throw the potato as soon as they've done the bare minimum to accomplish their contractual obligations and get paid.
I worked for a non-Federal US government, and my team was an amazing group of talented and passionate people.
Despite all of the nonsense of living in a bureaucracy where 500 people can say “no”, the mission and the impact on our neighbors was worth it. Compensation meh, total compensation was good.
But as time went on, the terms of the deal changed for new employees — generous pensions and benefits were tweaked in a way that marginally reduced cost, but dramatically reduced value to the employee. End result was average turnover of new hires was ~70% vs ~20%, and after I’m dead, the state pension fund will be sitting pretty as the new folks pay more to collect dramatically less benefits.
This is not true! On average, the federal government is paying quite a bit of money for tech people. I'd seriously estimate they're paying almost FAANG rates in a lot of cases all told.
The problem is that a lot of it gets siphoned off in the contracting process. The contracting process needs to be fixed. There's a ton of huge established players doing really bad work. IT specifically gets treated differently in government contracting, and is almost always procured through something called a GWAC which is basically a moat that it's hard for new companies to get into (the goal is to streamline procurement by having "vetted" companies in this pool already but it usually means it's very difficult for new players to get involved.)
That is changing, just very slowly. The agency that created this playbook, GSA, is helping facilitate that with things like introducing good vendors to other agencies and smaller more limited engagements.
But even doing a quick comparison, levels.fyi estimates about $160 - 180k cash compensation for new hires for FAANG. GS-15 Step 3 in DC is 150k. Plus excellent government benefits. It's spitting distance at least.
Also GS-15 is the absolute max. 2 yrs at FAANG and your comp will be closed to 220-250k, which is already passed the absolute max you can get paid on the government pay scale.
And admittedly, it is a little hyperbolic. I don’t have actual data to directly back it up beyond some research I did earlier this year. But FAANG really represents the most you could get paid as a software engineer and it’s pretty clear to me that these salaries are at least average/middle of the road. There isn’t some widespread low pay issue for software engineers doing this kind of work.
The other major recruiting problem is marijuana prohibition. Government has a lot of problems retaining high-skill IT people because a huge number of computer-touchers smoke. iirc FBI has commented on their inability to hire good technical staff because they can't find anyone who doesn't smoke.
And it's not just a question of willingness to smoke or not going forward, for most public-trust positions you're going to have to abstain for at least a year before even applying.
What are you willing to give up for a job offer that you might not even get? For computer-touchers with lots of other options, that answer is "not much". And federal jobs are basically low-paying, highly-competitive, and high-bureaucracy, when you could be making bank and having a reasonably satisfying work experience in the private sector.
And thus you filter out a huge number of the exact people you're trying to hire - young techies who are familiar with modern IT and development practices.
Marijuana usage is so pervasive that they basically can't exclude anyone who's ever used it, instead they want to see that it's been at least a year (preferably more) since you've used it and you will probably get approved. "Harder" drugs they are more strict on.
If you just smoked pot in college no worries, you'll be fine.
Well many many of us have the mentality that the US Government and various local governments SHOULD NOT be doing some of the things they are attempting to do, even if they could
See we believe in LIMITED government, VERY LIMITED government as such we do not believe in giving the government more resources to take over things it has no ethical authority to be meddling with in the first place
You can estimate here: https://www.federalpay.org/gs/calculator
Also, the sense of duty and serving the public is real. And this is not expressed in a tongue and cheek way... it’s an earnest desire to do meaningful work.
Many people who join USDS, 18F, and PIF from the tech industry are there because they genuinely want to do good work.
Agencies like USDS, 18F, etc. who are hiring at this level are atypical, and have to do so in order to attract talent away from private sector. GS-13+ slots are typically senior leadership positions and not entry level for anyone unless they have a PhD.
If you want to make change in the government, and don't want to become a federal employee, being a contractor is an alternative that you might consider: https://www.adhocteam.us/join/
It can be frustrating work, but it can also be enormously satisfying; I've been doing it now for ~6 years. Feel free to ask me questions if you want, I'm not a recruiter, just an engineer that's been building government products for a few years and trying to improve how they get built.
Technology really drives those organizations, just very slowly, as they've finally got the hang of those whole "client/ server" concept, and the notion of providing "services" at all.
Of course the most hilarious aspects of this are when departments foist hours of farcical forms on your projects and insist they are providing you a "service," where the service is compliance with their gating demands to fill out forms. (looking at you, infosec) Many of these organizations have evolved precisely to deflect accountability, as authority without responsibility is the ultimate measure of bureaucratic success.
I am an advocate for this playbook, but change will take time, probably until we've moved on to something better...
My country does not have anything like it, but what countries do?
Australia: https://www.dta.gov.au/
I find #7 especially hollow:
> We need talented people working in government who have experience creating modern digital services.
Sure, no one would argue against that, but the solution to this can't come from the team implementing or providing the digital service. Rather the whole of USG's incredibly obtuse, opaque, and inflexible centralized hiring system needs to be addressed. This is no better than telling a USA Special Forces team commander that in order for his operators to be more effective he needs the Army to recruit better soldiers.
Another one that comes to mind is Coding It Forward, which has the express mission of creating government tech internships that rival FB/Google internships in terms of pay, experience, and mentorship:
Bureaucracies have infinite ways to stop something if you don't get cover from certain people.
Coding It Forward is a civil society initiative, not a government one, and is analogous to thinking that a weak diplomatic corps can be fixed by sponsoring more high school Model U.N. chapters. There is no lack of viable and willing candidates _outside_ of the system, the issue is getting them _inside_ and letting them inform it.
Sure, there are some people who will see working for USDS or 18F as a call to serve their country, and will accept a pay cut in order to do that, but I imagine that's a pretty small slice of the pie.
Beyond that, it's the bureaucracy. I would honestly consider said pay cut in order to help make dealing with government computing systems better, but what I will not deal with is layers upon layers upon layers of bureaucracy. I've watched the 50-person startup I started at 9 years ago evolve into a 3500-person public company, and even here I find the bureaucracy and hierarchy stifling at times. At a government job I imagine it'll be orders of magnitude worse. I wouldn't even last half a year in that environment.
Yes, Play #4 in their list in theory suggests that they'll try to cut through all that, but I don't believe for a second that will go well.
I feel the absence of job training upon hiring is also why there is no concept of "company men" after the Boomer Generation; companies stopped investing in the employees they wanted and started treating them like pre-packaged commodity.
I agree with your frustrations about training in general too. I wish companies provided that _and_ provided real career progression goals. Instead it’s learn the tech for your job after work and some vague one page sheet from HR about what the next level is, which really comes down to a manager’s opinion more than anything really measurable.
> go all in on 18F, where it provides a whole career progression from basic training on up through SES.
should read
"go all in on 18F so that it can provide a whole career progression from basic training on up through SES."
You're correct, as far as I'm aware 18F, or any other government org that isn't the US military, does not offer actual training.
I think there's some confusion here on what and how the USDS works.
They embed into government organizations to improve services and systems, bringing expertise that doesn't normally exist in government, while not working directly for the organization they're embedded in.
USDS doesn't follow normal hiring practices either. They recruit directly for limited employment terms, max of 4 years. This helps offset the entrenched beaurocratic mindset that exists in a lot of USG organisations, and allows them to recruit current experts in a particular field.
From the Special Operations analogy this would be equivalent of the Military giving them direct hiring/recruitment ability from both the Armed Services and direct from civilian
#7 "Bring in experienced teams" isn't hollow, it's their core mission.
Going back to our US Special Forces team, the US Army actually did (attempt)to solve for this by simplifying the pipeline with the 18X "X-Ray" Program.
We do work on hiring as well
It's akin to the Postal Service visiting everyone's mailbox each day - not efficient, but important.
THAT SAID, there are massive improvements that can happen in hiring - if agencies have incentives to pursue them!
https://github.com/usds/playbook
Along with full commit history.