Hacker News Reader

Comment by rsweeney21 | Hacker News Reader

I tried to build a VC backed startup that sold a SaaS product to state and federal governments. The procurement process for the government is so horrible and broken that it blew me away. From first conversation to signed contract will take 2 years minimum. It's filled with legal challenges, back room deals, inefficiencies, a ridiculous RFP process. You run out of money before you even get your first contract signed.

It was so strange to me to sit in meetings and talk about how we could make them so much more efficient and they would talk about having the problem solved in 5 or 10 year timelines. Having spent most of my career at Microsoft and Netflix where a year feels like an eternity, it's just a completely different world.

View on HN

The problem isn't that the process is really broken (though it is in many ways) - the problem is that too many companies fail to understand there is a major difference between government and commercial markets. Two totally different mindsets.

Yes, the sales cycles in government are a lot longer (2 years is at the extreme end though, 3 to 6 months is more average but with a lot of caveats) and yes, there are a lot of rules and regulations that don't exist in commercial deals (FAR, DFARS, etc. etc.) - but at the end of the day, you're talking about business-changing scale when you get awarded a decent sized government contract (millions of dollars over a period of 3-5 years on average or longer). You're talking about being able to grow to 100's of employees with just a half dozen or so government customers.

The problem here really is that people need to invest the time and energy into understanding the government market and they can't expect it to be like the commercial one. If you do this, then the payout is very much worth it (and, no, it doesn't require "deep pockets" but it DOES require relationship building, a strong commitment, and a lot of learning).

If you give up in frustration because "it's too convoluted" compared to the commercial market, then, yeah, it will never pay off and it will feel like everything is stacked against you. But if you stick it out you'll find yourself with some incredibly stable and predictable long-term revenue streams.

I think the concern is that the reason it is so different is disturbing to taxpayers: If it is filled with more bureaucracy than a similar commercial deal, then they're wasting taxpayer money. If that bureaucracy is the only way to prevent corruption, then they must be horribly easy to corrupt. If you can get a guaranteed revenue stream to grow your company to 100s of people with no day to day risk of losing this customer, then they're not being good stewards of that money. In short, them being different in these ways are bad sign for the taxpayers who fund this.

But the bureaucracy is sometimes for good reason (usually driven by Congressional mandate). What people need to do is learn to work within (and around) the bureaucracy. That's key to the government market. There is a reason for the way it is, and you need to learn to work with it instead of fighting against it.

I'm not sure you're understanding really how the system works, which is exactly what I was pointing out in my statement. I never said "guaranteed" - I said "stable". There is ALWAYS risk of losing the customer and the government can terminate at any time for almost any reason. But if you perform, then you can maintain the same customer for a decade or more.

This is how the system works and it works this way for a reason. There is more accountability in the government market than people realize - moreso than in most commercial transactions even.

I think people would be more willing to accept the bureaucracy if it delivered results. If you've ever been the customer of a government tech product (even a public-facing one like healthcare.gov) you know it does not.

Take a look around the world and look at the average state of governance and you will find a lot of appreciation for American bureaucracy.

And it is indeed independent bureaucracies that are responsible for making agencies and institutions run. Bureaucracy gets a bad name in many places, but the alternative to bureaucracy isn't just automatic improvement, it's most often politicization. Which is significantly worse.

Historically, government contracting and the public sector has been incredibly corrupt, as it's basically unlimited funds with no market-based mechanism to address it (in a private entity, the owners will be angry about money taken from their pockets by employees, and in places where corruption is too high, you end up with family firms or other external trust structures, very small entities, etc.)

This is why the heavy-handed regulations exist in US contracting (along with massive risk aversion and other things). The Government uniquely cannot be subject to market forces and punishment for bad actions.

That's true in places like India, but not really in the U.S. I own a company that does government contracting and have never "paid someone off" or done anything remotely "corrupt" but we have been wildly successful nonetheless.

There are NUMEROUS mechanisms for oversight and redress at every level - but the typical commercial-focused company doesn't understand them, so they don't research or learn what they are and just give up thereby developing a very misguided interpretation of the system akin to what you believe it is like.

This was true in the US before we created all these stupid regulations, which is why they were created. 1700s/1800s, "spoils system", etc. That is the whole reason the regulations (and attendant overhead) exist.

(I bootstrapped a US government contractor and have worked for several others.)

Can you clarify your point, were you saying that the US had 3rd world-levels of corruption before the regulation, or were you saying that the US was as fine as it is today, before the regulations.

The former- before the regulations in the 1800's there wasn't even really a concept of 'third-world levels of corruption' as it was universal to every government.

The former. Google the "spoils system."

I've worked for commercial, government and semi-government organizations. It's not corrupt in the sense that it's illegal. The problem is that while the stakes are high within the private sector there are no such stakes within the government sector. Yet people do get payed high stakes salaries. There's no incentive for efficiency. In the government realm when the end of year is nearing, departments start burning through their budgets to ensure they don't receive less the next year. Inefficiency is rewarded. It simply means more people get to pay for nice mortgages.

> If that bureaucracy is the only way to prevent corruption, then they must be horribly easy to corrupt.

The bar is much higher for government procurement, though. As an example, if I decide to use Codacy as my static analysis tool at work because I know the founder, nobody would bat an eyelid. If that same dynamic were reproduced at the government level, it would amount to corruption.

Having worked for a government contractor, it was my impression during my time there that they're so worried about oversight of the money and what miss-spending and corruption looks like that they spend most of the money making sure it doesn't get miss-spent. Often the amount spent recovering miss-spent money far out-weighs the cost of what was miss-spent in the first place.

One example my attention was drawn to is that a government employee overspent by $1.50 for a drink that was misassigned an expense code for the amount they spent. If they'd used the correct expense code or spent $1.50 less, no eyebrows would have been raised. Instead everything was processed and the person was pulled into a tribunal in front of other government employees who were being paid to oversee spending. Lord knows what those employees were paid to investigate this miss-spend. On top of the fact this employee was paid to attend the tribunal. This kind of behaviour is a gross waste of time and money... for $1.50 at a cost of many thousands of dollars to reclaim - or more likely re-assign to the correct expense code... and for what?

If you want to see your tax dollars at work, it's right here, making sure none of your hard earned money gets miss-spent.

It's accountability in the age before we had AIs that could spot strange patterns of behavior across millions of government employees. Presumably now there might be more efficient ways to do this, but it's a combination of inertia and fear of what the consequences would be if such a program didn't work out. The upside of the current system is that it leaves huge paper trails and everything is auditable; the downside is that actually auditing all this stuff is so painful.

It also doesn't really help that a majority of the politicians and a majority of voters foam at the mouth about government waste, no matter how minuscule and how much it would actually cost to eliminate. If we implemented more efficient algorithms and saved 90% of administration but some waste slipped through they'd say the government was getting sloppy. You see this a lot when people rail against, say, fare evasion on public transit; people would rather have 10x of a bunch of highly paid employees to manually check every ticket, instead of having proof-of-payment and random inspections with fines, because it feels like the right thing to do.

Here's the scale of problem the government deals with:

https://www.youtube.com/watch?v=4JkcrtvN60s

That's an experimental YF-22A crashing due to a software glitch. That's a $300-400M hand-crafted state-of-the-art-on-a-hundred-dimensions supersonic jet, almost killing the test pilot (an engineer who flies planes) 50 feet off the ground at 150 mph. Just the cost of training that pilot is easily in the millions of dollars.

Everyone agrees the budget process is nuts, right up until something like that happens. And then everyone who ever signed any document related to that airplane is able to sleep at night. Except the software teams that wrote or approved the code. They're going to loose sleep, and they know why.

Oh, and the DoD is a rounding error compared to CMS. They have routine fraud complaints that would sink a major defense contractor.

I think fraud detection and holding people accountable for safety lapses are two completely different problems calling for different solutions. Just because you have a hammer doesn't mean everything's a nail.

If we didn't spend so much money on fraud detection via immense bureaucracy, that would increase the amount of money available to provision services.

Serious investigations of seemingly-trivial events can produce some startling information. Look into the ongoing Kealoha scandal in Oahu. One broken mailbox is tearing down a massive criminal conspiracy.

Stupid question, but how can it not require deep pockets when it can take up to two years to get to a single signed contract?

I think because the implication of the phrase 'deep pockets' for most people implies a graft or bribe of some kind.

Reading it as - you need enough money to make it two years without payment - is fair, but I would argue not how most people read that statement.

From what I gathered when I got an offer for a government contract position: the company has an idea of how many people they want to hire (or things they want to buy) to fulfill the contract.

This can involve making "contingent offers" to get people interested, which to my knowledge carry no actual legal weight and is really just an informal agreement.

This way, you get a reasonable pool of candidates without actually having to hire anyone.

The dirty secret of government contracts is that most firms, no matter how good their tech and implementations are, couldn't win the contracts because they don't understand government procurement well enough. This leads to the same companies winning contracts over and over again just because they know how to win the bids, and not because their solutions or prices make sense. Many big contacting and consulting firms are really just experts in the procurement process.

This then leads into third parties acting as procurement contractors, where their only job is to win government contracts for others.

There are other middlemen taxes like this. Ever seen a proposal that says that it must go to a small business or a certain percentage of the award must go to a small business? Just find a small business to bid on the contract, take their cut, and then sell your product as the solution.

Government procurement is a dark place.

Overpromise and underbid, there are few if any consequences. YMMV

This is 100% spot on in my experience.

> It was so strange to me to sit in meetings and talk about how we could make them so much more efficient and they would talk about having the problem solved in 5 or 10 year timelines.

Maybe their experiences or observations with "VC backed startup[s] that [sell] a SaaS product" has shown them again and again that you cannot trust those in any way to be even around for 2 years.

Seems unlikely to me. The barrier to entry for startups selling to the government is so high because of the regulations to prevent corruption force such a long sales cycle.

Another thing that surprised me when it came to selling to the government is that it seemed like most employees had no idea how to purchase something, even if they had budget and authority.

The federal government is generally biased in favor of large entities that'll be around in 20 yrs because that is the timescale over which the government operates, which is why their technology is always significantly behind.

As a startup your best bet is partnership with the big guys, meaning you'll have to craft your solutions in a way that it can piggy-back on all the checklists (things like FeDRAMP) the big guys already have checked.

Don't even get me started on the bureaucracy full of people whose sole purpose for existence is to ensure nothing can get done:)

Not true. 23% of all government money goes to small businesses. Over 30% in certain agencies and up to 60% in some. Like I've been saying here, you need to take the time to learn the market because if you don't, you will just believe it is stacked against you when it really isn't and is actually more friendly to smaller companies than certain commercial markets.

My response was in relation to a VC funded startup business, that is very different from the "small business" segment you're referring to.

There are loads of "small business" entities out there who get government business merely by being in existence and meeting the sizing requirements.

I however wouldn't quit my job to start one of these because these are really just people grazing on the federal acquisition process and provide in most cases little to no value, it is not much money either.

I know this because I have experience with working on government projects where you can easily have 20+ contractors involved and it is not clear what their roles or input are to the project.

question:

are most of those small businesses partnered with larger entities on RFPs to win those contracts/money?

A large percentage of them are. When starting out many small businesses exclusively work as subcontractors to larger entities which must partner with smaller businesses to meet set asides.

Some contracts require the prime contractor to be a small business. Often when this is the case and the contract is large the small business acts almost purely as a pass through to a larger contractor. Taking a percentage off the top and filling a couple of slots on the contract while the larger contractor does fills most of the slots and takes the majority of the money.

true in some cases, but some set asides require the small to do a certain percentage of the business

Which is why I said often and not always.

> From first conversation to signed contract will take 2 years minimum.

If you are trying to sell to government for a “need” they aren't actively seeking solutions for, which your description sounds like, that seems plausible. Then again, a major purpose of the safeguards in the procurement process is to prevent sales going in that direction in the first place. The intent is for the government to identify a need, and then solicit solutions, not to have a vendor with a vested interest in the government perceiving a need in a way which favors their solution sell the government on the need.

the other side of it is the decision you make like this in government will have a 30-50 year impact. making changes is not easy for anyone, but you can change an entire infrastructure at microsoft in a tenth of the time you could do it at the DoD

Well said, buying for an entity that needs to survive to the end of time is a totally different ballgame.

For comparison, the DoD's 2019 budget ($686B) is six times Microsoft's 2018 revenue ($110B). It is one-hundred and sixty-three times Netflix's $4B.

Add that to relative technical aptitude, regulatory compliance, etc. and it doesn't surprise me Government is going to take longer.

However, if you are able to land a contract or two, you have a lot of money pouring in. Maybe enough to the point where you don't need any further outside funding, or a reduced amount of outside funding. And since the contracts are for 2+ years and you have to really screw up to have it canceled, then you have guaranteed income for years. You can't run it like a consumer startup.

This can be a very difficult market, but there are a handful of different projects designed to help with this. In-Q-Tel [0] has been accelerating adoption of new technology for the Intelligence Community for many years and often invests in and helps deploy technology from startups of many sizes. SBIRs [1] can be an effective way to quickly get gov money for research and even culminate in a procurement vehicle for sole source contracts if you get through Phase III. There are also a handful of accelerators like DIU [2] and MD5 [3] designed to help small firms navigate this difficult space. It still isn't easy, but it can help level the playing field a bit when you are just a startup.

[0]: https://www.iqt.org/

[1]: https://www.sbir.gov/

[2]: https://www.diu.mil/

[3]: https://community.md5.net/md5/landing

Pro tip: don't sell to the government. Regardless if it's in the US or outside

Rather, sell to someone (or partner someone) who is experienced and let them do the talking

Also don't do it as the 1st thing you do, you'll have much more success selling it to people/(small) business first.

I would only advise selling to gov first if 1) you absolutely know what you're doing and it is something that only them can buy and 2) you have money for someone to walk in the higher circles (as in both salary + "extras")

Large enterprise customers can take just as long with an even less transparent process. It's not a particular problem of gov't, it's a problem that comes with size.

This is further compounded by how Gov employees are compensated. Gov IT is not a meritocracy with RSUs, so "efficiency" and "performance" are not intrinsic goals.

To truly succeed in Gov contracts, solutions must be aligned with the retirement and pension goals of procurement officials who want to "ride out" long-term projects to the sunset.

I work for a Danish municipality, we’ve bought a few SaaS products in recent years. The fastest implement was two days, and we had a working product before we were even done with the contract. We bought it from a local software house, that knows how political organisations operate, they host in AWS, but they could have hosted it in Azure for all we care. (We would prefer Azure because they are better in Europe, but AWS is fine).

So government operations and contracts can work fast. Until you meet a certain amount of cost that is. We have laws that govern how we buy software, and once they kick in, the process is long, tiresome and really, really, ineffective.

Because often the company who wins a bidding war, underbids, and end up delivering a lot less than promised unless you pay ridiculous extra money, and nothing in those laws protect us from this, not really.

The big issue is that government isn’t similar, so you have to govern in for the lowest common denominator. My ciry has technical competences on staff, this allows us to actually utilise SaaS products that require integration into our business processes and 300 existing systems. Our neighbouring municipality doesn’t, and as a result, they don’t even really know what the word SaaS means. The government has to write laws that work for both of us.

I think the real issue is lack of technical competences in the public sector, and the reason is really beyond me. Sure I have a vested interest, but I think I can defend it by my ability to write software in a few weeks that would cost us half a million on the private market. That’s not the primary importance of technical staff though, I don’t think we should build all those 300 systems ourselves, but we need to know how to cooperate with the private sector, and we need to build an enterprise architecture to support that, and to do that, we need technical staff.

The fact that this isn’t obvious in the political command structure in 2019 is just silly. Especially in Denmark where we’re competing with Estonia for having the most digitised public sector in the world. But to add some numbers to it, we have 7000 employees, they use a digital device or a computer for an average of 5 hours a day. We have 245 physical locations with network (schools, libraries, child care institutions and so on). We have more than 300 different systems, some major some minor. We’re expected to operate this with 5 people in operations, two people in development and architecture and 3 people in support.

I think I know why though. We don’t have a single elected official in my entire country who knows how to program.

The procurement process for the government is broken because our political system is broken. You need deep pockets and political connections.

This is actually Elon Musk’s greatest triumph; almost all of his post-PayPal businesses are based on exploiting weaknesses in government regulatory (Tesla, PayPal) or procurement processes (SpaceX, boring company).

SpaceX was legitimate innovation in a broken government contracting process, but Tesla is starting to look like it was only a viable business with the subsidies — I would anticipate their business model to shift away from vehicles soon and turn towards large-scale grid infrastructure (aka government contracting).

Yes and no.

Separate from the political issues at the top level of funding, the entire FAR (Federal Acquisition Regulations) is a steaming pile of reactionary bullshit, grown by accretion to minimize risk to bureaucrats at the expense of serving the tax payer efficiently.

The whole "bribing and political connections" side is sexy and interesting and headline grabbing, but the bulk of the issue is just far more banal and depressing.

The sad part about it is that the state of the FAR is such that our government, even if you are a person that wants it to play a bigger role, isn't really capable of executing on big things anymore. Combine that with the inept and apathetic staffing of (typically) mediocre at best clock punchers, and you end up with an organization that has a deep, deep spending efficiency issue.

The two political parties are both clueless. One thinks a massive, bloated organization with a track record of spending inefficiencies needs more revenue. The other thinks that a nation of 350 million just doesn't need a funded federal government.

> Separate from the political issues at the top level of funding, the entire FAR (Federal Acquisition Regulations) is a steaming pile of reactionary bullshit, grown by accretion to minimize risk to bureaucrats at the expense of serving the tax payer efficiently.

I don't think people understand just how much of the procurement system has been built out of risk aversion.

A thing needs doing, politicians / government workers procure a solution to that, and that solution ends up going wrong somehow. Lots of egg on the relevant politician's faces, and plenty of ammo for anyone running against them. "$foo wasted millions of taxpayers money on solutions that don't even work!" So they put in a new mechanism to try to ensure that mistake never happens again. Each individual mechanism on its own generally makes sense.

But over decades and decades it becomes such a complete mess that if you take a step back and look at it, it's all kinds of absurdity (There's a fun parallel there to the infrastructure and systems we tend to make as developers and sysadmins). Worse, the kind of absurdity it has become actually ends up costing taxpayers more money than if a bunch of the regulations didn't exist, cheaper things were tried and failed, and replaced with one that does work.

By the same token, if you normalize failure in a structure that large, it’s hard to get anything done. It’s more enriching to get paid to fail 5 times than it is to do the thing once successfully.

Failure is great when there is oversight and feedback mechanisms that help the organization learn from those failures. Less good when repeated failure is profitable.

The major driver of procurement bureaucracy is exactly this sort of blatant, somewhat cynical, distrust in government.

Every requirement is designed to protect against arbitrary decision making, especially corruption but also just general "I like the vendor/product".

Corruption is self-explanatory, so let's focus on the second:

Instead of buying a Toyota because you liked the last one you drove, or because they offer a fancy color you like, you have to create a 500-item checklist of features you want in a car, and a scheme to evaluate and weigh all possible options.

If you forget a single important criterion, say "available in colors other than pink", the police department will now be driving in pink cars.

I've worked in Canadian Federal sector after decade+ in commercial sector. My mind was blown - they do not compare:

1. Speed is an order (or three) of magnitude lower. On anything - from macro-scale items such as signing a contract, designing the solution, creating the plan; to micro-scale items such as opening a port (2weeks-2months), creating a VMWare VM (2weeks-2years), obtaining a standardized laptop for new team members (2-weeks-3months). (Yes - you read that right - there is a set of two well defined VMWare windows VMs we have literally been waiting on for two years)

1b. After my first project, I reviewed the openly available "lessons learned" from variety of Gov't IT projects, and in the top three, uniformly, you'll find a sentence like "External vendor did not have sufficient appreciation for nature, complexity and speed of government processes". Truer words never said. We became faster when we slowed down to the oragnizational level: if we tried to do something in 2 days that government does in 2 weeks, it'd take us 4 weeks of hitting our head against the wall and causing havoc. Once we understood the 2 weeks it takes and built it into our plans, then 2 weeks it took.

3. As a citizen, I think of "The Government" - a single, big, overarching entity. Forget about it - it's really a lose and overly-formalized set of practically-independent organizations. IT integration of Department of Agriculture to Ministry of Labour is like integrating Walmart to Tesla. Different standards, processes, networks, firewalls, etc. It can take couple of years to agree and consume standardaized webservices across departments

4. Like many large organizations with variety of responsibilities and products, stability and predictability are much, much more important than speed. They need a solution that will last 20 years and a vendor that will exist and support it for that long. They can't change frameworks across 42 datacentres and 300,000 employees every 2 months. When their budget works at best on yearly basis, they cannot change their CIO roadplans quarterly.

5. This is a personal revelation and people may disagree: we want, or rather demand government to be that way. Due to requirement on transparency, fairness, documentation, we the public end up pushing government in the corner. They cannot just pick the easy right choice in two weeks - that'd be in the Globe & Mail (think Washington Post;) as unfair process, favouritism, lack of transparency... other companies would outright sue. So you get a two year procurement process that ultimately WE demand. Same for every decision at every level - it must be examined, discussed, commented, documented, covered. Everything has to be supported at an order of magnitude higher level than in commercial world (and note it has to be a well-supported, not necessarily right decision). Government procurement and processes have documentation and checkpoints which in some ways can be wonderfully stable and structured, in other ways immensely frustrating.

My mentor told me early on: think of everything you do in the context of front page headline: "Consultant does X on public money" So I can't buy lunch to my colleague I've worked with for years ("Vendor tries to bribe public servants"), cannot have a kick-off dinner for the project ("Consultants eat and cavort on tax-payers' money!!!"), etc.

My personal fear/nightmare is how I'll integrate back into the market having adapted to the public sector way and speed of doing things. Reading about neck-breaking speed of JavaScript frameworks and public cloud changes out there "in the real world", doesn't help my sense of peace :P :P :P

>we want, or rather demand government to be that way

This is something so many people miss. So much of the endless red tape is a direct result of public concern over wasting taxpayer dollars. The irony of the situation is the government would be able to save a lot of money if taxpayers stopped caring about where their money was going.

I often wonder whether we'd be better off if we "tolerated" a small amount of corruption. What's the point in spending $1 million on red tape and auditors to prevent $10,000 in fraud?

Just like the war on drugs, you're fighting a moral crusade against the basest part of human nature. You're always going to lose - better to just accept it and do what's needed to minimize net loss.

Is it horrible and broken? Or is the grueling process a feature? By filtering out VC-backed start-ups they are left with only bidders who they can be sure aren't going to vanish if funding dries up or a few key staff leave.

It's what Avisare [0] is trying to address (I have no connection to the company).

We have some government contracts pending and we decided to give them services free of charge rather than going through the procurement hell.

[0] https://www.avisare.com

Government is inherently inefficient, yet some people still insist that it controls everything.

Classic reverse causation there.

I've worked as a government contractor in a few industries and my experience is just like OP's. The government is slow, inefficient, contradictory, expensive, and it under-delivers. This isn't an accusation, this is just fact, and it's not even a secret.

Reply on news.ycombinator.com