Comment by stevesycombacct | Hacker News Reader

Comment by stevesycombacct | Hacker News Reader

Parent Full thread

stevesycombacct

I have a lot of heartburn reading something like this. I spent a decade in government consulting trying to implement anything resembling "data science" or "DevOps" and got repeatedly shut down by higher ups or by random bureaucrats that saw any modern practices, including any technology, as a threat.

Excel macros working one day, then prohibited the next by new IT policies.

Directors demanding that service line bosses meet with me to discuss "data-driven decision-making", followed by a solid week of said chiefs telling me that my job is a joke.

Actual honest-to-God employees trying to make a difference in financial openness having their coworkers walk by their desks and loudly shout they "ain't doing that shit" and walking away.

Two years hunting for a database to store 2 GB of data, and only finding one when a new contractor onboarded and knew a guy in the next office over who would let them have a partition of SQL Server.

I wish I was making any of this up. The federal government has massive, massive employee culture issues. This list is... a nice dream.

These plays can work. Why? Because USDS employees are government employees with escalation paths all the way up to the very top which gives them a lot of power to break down bureaucratic barriers to modern software development that government contractors & consultants would have no chance of doing. They then can bring in contractors to work in the relatively modern shell that they've created.

For example, here's a project started by the USDS in 2015. It's responsible for managing the complex bureaucratic process of VA legal appeals. (https://github.com/department-of-veterans-affairs/caseflow) It's open source, continuously integrated and deployed with close to 100% test coverage, deployed on an AWS GovCloud VPC, and was built on a fraction of the budget of similar systems.

Active development on this system is now done primarily by contractors now that the relevant bureaucratic barriers were removed.

How do we scale this story? More talented people with the desire to serve their country. It's not easy, but it can work.

Contractors can also be part of the problem. Very few contracting firms with access to blanket purchase agreements are willing to dump proprietary tech—as it’s their cashcow. For example, Oracle is huge in the federal gov, and Oracle DBAs are billed at a premium over a dev who can just make Postgres work. Worse, Oracle isn’t the problem rather the problem is some legacy tech or middleware no one has ever heard of that is incredibly fragile and has been hacked into a partially working solution by a vendor over a five year period and management isn’t willing to expend the financial, social, nor political capital to correct and forget about management acknowledging sunk costs.

IMO, USDS makes this worse as someone like Matt Cutts has the cachet to bring in the playbook and get execs to listen. Whereas, I’ve been doing this for 3 years and routinely have to fend off coup attempts as the term “mvp” sounds like a joke to other career employees. As such, it’s often easier to check the box of the federal acquisition requirements and collect a paycheck versus hustling to build a vanilla web app using JavaScript a rest api and Postgres before the checkbox folks shut you down because the baselined version of Jenkins is 5 years old and you installed version 2.26x without a request for change.

So how do we scale this story? Inform politicals and bureaucrats that incentives matter, successful tech start-ups are exception not the rule, sunk costs are real, and you don’t need a catchy name like 18f nor the catchet of celeb dev to right a failing project.

I've also heard of contractors selling nearly identical software to multiple agencies and charging each the full price of development (one reason open source has faced opposition), and sometimes decent excel macros can solve the actual problem.

You're totally right that if the incentives don't change, the choices made by execs won't change either. For example, a contracting officer I've met saved their agency a billion dollars by not continuing a failing contract, and was punished (to the point of leaving that agency) because that wasn't what the incentives supported.

How can USDS/18F be helpful to you and others in your position? The messages you list at the end are all true.

The Playbook is helpful. So is the DoD’s Detecting Agile BS. More publish references similar to these that can be presented to management. Another Avenue would be to engage GAO and OMB in order for their reports to align with above. Thank you!

At market comp with market frustrations?

Most reasonable people might wanna work for their government for a 20% discount in either, but my fear is that it’s much much more and in both.

It's both more difficult than any private company job I've had and it paid less, but knowing the impact my work had on veterans was well worth all of it. I'm extremely proud of the work that my teammates and I did there.

So many people complain about the government, so few are willing to fight to fix it.

No civil service job can make us replace political appointees with well compensated civil servants like Singapore and the UK have.

Off-topic : I always wondered if serving one country also deserves the whole world.

I wish I was making any of this up. The federal government has massive, massive employee culture issues. This list is... a nice dream.

Much of what you describe is not unique to the U.S. government. I've run into many very similar scenarios in the jobs I've had over the years, from mid-sized travel company to billion-dollar healthcare companies.

The USDS doesn’t hire people through normal channels or for permanent hire, but hires people who have experience in the private sector and want to change the government. While it’s true that they hire significantly more junior people than before the trump admin (if you compare things like years of experience pre USDS then and now, or pre USDS salaries then and now), it’s still often experienced people who want to make a difference for a short period of time before leaving government.

FWIW, I don't know if there's a significant difference along a senior vs junior axis. I think the set of people willing to consider gov employment at USDS certainly (and reasonably) changed in after the 2016 election, so USDS lost access to some talent, but there is good work happening now.

Most importantly, USDS is seen as non-partisan, and so has been able to have good impact in both administrations.

Have certainly seen and experienced what you did, so I hear you. However, there are improvements slowly being made to be optimistic about. DevOps (or DevSecOps as they like to refer to it now...) is definitely becoming a more ingrained practice. If you’re interested, take a look at initiatives such as Cloud One, or some of the work being done by GSA (especially 18F) and USCIS (of all places).

Interestingly, the 21st Century IDEA Act mandates compliance with the USWDS now (at least for publicly facing sites) and there is a maturity model built around it. We’ll see whether or not agencies comply, but it’s a step in the right direction.

I'm curious how you've heard about USCIS' work / style?

I'm new to the system, so this is the only one I know (wrote about joining here: https://twitter.com/abachman/status/1217795232449859585), but I'm currently on a team of 6, all feds: 1 designer, 1 product mgr, 3 engineers and I'd say we're fairly to extremely self-directed. User-research based product development, pair programming, short iterations, deploying daily, all that jazz. Just a variation on the same sort of thing I've seen in the tech-first / software-product-company gigs I had in the past, but I've mostly worked for smaller indie or niche companies.

That is, I get to work in a way and with people and tools (Rails + React at the moment) that makes sense to me. How far outside the norm are we and where could I go to learn more about the whole system?

Other than previous work experience at DHS, bidding USCIS solicitations was the biggest difference I had observed with USCIS/way I became familiar with their practices. They’re pretty unique/developed with both their acquisition and DevOps practices.

It kind of sucks being on the bidding end, as their tech challenges cost a ton on top of the normal costs with a proposal. But I can understand why they do it and from an acquisition perspective, it makes total sense. I think they will need to strike a balance at some point though - some of their tech challenges are too complicated for the timelines provided.

In terms of development practices, it is pretty much all DevSecOps and they make heavy use of independent verification and validation/QA, and even more importantly, are pretty mature in their adoption of it.

So to answer your question, you guys are pretty far outside the norm in a good way :-).

Not sure the best way for you to be able to learn more other than to do some networking in the DC area (post COVID). If you get the opportunity, there’s a ton of great meetups in the DC area, although if you’re taking the MARC those can be a challenge to attend after work due to train schedule limitations. Other than that, move around to/visit different agencies and get exposure to their work. Work with your PM to see if you can do some site visits to other agencies/organizations as part of professional development and such.

Ah! This is interesting info and helpful, thanks!

I was on the cusp of attending some cross organization "for feds" tech meetups back in March, but... :| I'll keep on the lookout.

[deleted]

Have you tried reaching out to Nicolas Chaillan or anyone within the AF CSO about some of your observations regarding where Cloud One/Platform One are missing the Mark? Nicolas is pretty active on LinkedIn and he may be open to input that could lead to some positive changes.

I agree with the sentiment that there are cultural issues with the federal space, and it requires continuous diligence of dedicated people to change the landscape and move the needle. Devops take a long time to grow organically - and it's longer in the federal government due to silos, compliance, finance, regulations.

The list is a nice dream, and it is with the will of those in the federal work force who can break down those silos and bring about a common understanding with urgency that'll make it happen for their project.

Were you working for the USDS though? My understanding is that when they're brought in, red tape doesn't have to be cut so much as it jumps out of their way in advance of their arrival. Basically a "special forces" digital intervention team with situational authority over the normal obstructionist BS.

Sounds like you want to do more then make a nice living in IT. Govt. IT for the most part isn't that, rather it's going with the easy/chill flow.

My kinda of place... no stress, drama (for chill .. go with the flow types), nice co-workers and you can pay the bills & then some!

I also work in government consulting. The govt thinks “DevOps” is developers and ops folks sitting in the same room.

Reply on news.ycombinator.com