President Signs Government-Wide Open Data Bill
datacoalition.org
datacoalition.org
I am very interested in what this type of lifecycle might look like considering most data I feel should be kept forever. I wonder how a lifecycle might collide with the challenges that bit rot[0] are dealing with.
[0] https://www.theguardian.com/technology/2015/feb/13/what-is-b...
For the most part, common file formats like PDFs, JPGs, and TIFs are likely to be understood for a very, very long time, but you don't just have file storage, you have systems to manage, index, and find those files, and those systems are likely to need constant maintenance.
Paper can rot if it's not kept climate-controlled, but not nearly as fast as bits stored to non-volatile media if the power goes out and there isn't a hot backup.
Physical storage space definitely does have a cost, but so does software licensing, hardware maintenance contracts, etc. And a lot of government organizations have physical space in abundance, but IT resources, less so.
A lot of this type of work is already being/been done.
"Forever" is a very long time indeed. There's a reason the UK still keeps some records on scrolls of velum: https://www.bbc.co.uk/news/magazine-35569281
[0] - https://news.softpedia.com/news/All-Videos-on-Google-Video-W...
But nothing digital, that's for sure.
So one day Jeff Bezos issued a mandate. He's doing that all the time, of course, and people scramble like ants being pounded with a rubber mallet whenever it happens. But on one occasion -- back around 2002 I think, plus or minus a year -- he issued a mandate that was so out there, so huge and eye-bulgingly ponderous, that it made all of his other mandates look like unsolicited peer bonuses.
His Big Mandate went something along these lines:
1) All teams will henceforth expose their data and functionality through service interfaces.
2) Teams must communicate with each other through these interfaces.
3) There will be no other form of interprocess communication allowed: no direct linking, no direct reads of another team's data store, no shared-memory model, no back-doors whatsoever. The only communication allowed is via service interface calls over the network.
4) It doesn't matter what technology they use. HTTP, Corba, Pubsub, custom protocols -- doesn't matter. Bezos doesn't care.
5) All service interfaces, without exception, must be designed from the ground up to be externalizable. That is to say, the team must plan and design to be able to expose the interface to developers in the outside world. No exceptions.
6) Anyone who doesn't do this will be fired.
Perhaps instead of it being "all teams" it should be "all cpu processes"?
Bezos didn't build AWS, people who worked for him did. They also didn't do it as part of the normal day to day at Amazon, no matter how much Amazon pushes the "it's the tech we use already" narrative.
The comment to which you’ve replied said exactly that.
The people that worked on AWS didn't build it, the other people that worked on AWS built it. That's the meaning of your statement.
Bezos works for Amazon, he decides whether AWS gets built or not, how large resources are deployed in the company, and what the priorities are over time. He worked on AWS. If you aren't an engineer, that doesn't mean you didn't help build a product or business. There are a lot of non-engineers that were required to build AWS, including managers, support staff, sales, marketing, HR, accounting and so on.
Saying he didn't build it and that everyone else did, is the same style of mistake as pretending that Bezos built it by himself would be (and nobody actually thinks he built it by himself).
The Bezos-as-Demiurge theory, or “Gnostic Amazon” theory? I like it.
What it really means is that if you're building something that you run on your teams hosts it needs to be exposed through services. For most of the services at amazon that means using the common service interfaces that everyone uses and not just giving out DB credentials (which I've seen happen it took (think it's still in progress 5+ years later)) to untangle.
It's not a "you must find some data or service to expose". If you have data and someone wants to use it, they do it via service.
I'd say you're being a pedant but you're not even technically correct.
https://martinfowler.com/bliki/TellDontAsk.html
I'd also like to point out that with all design decisions there are trade offs. Bezos made these trade offs specifically with Amazon's scale in mind and likely would not make the same trade offs with a team of 5 devs or even a team of a thousand devs that had good habbits around adhering to data boundries.
Clearly the dev culture at Amazon had a habbit of abusing those boundries or it wouldn't have gotten to the point where such a black and white mandate made a net positive change to the status quo.
It's more nuanced for Amazon - they intentionally ignored language skills when looking to hire good developers (difficult in 90's Seattle), so the end result was a hodgepodge of services written in many different languages talking any number of protocols which including Service A directly accessing the database of Service M. This mandate was his answer to teams complaining about working with each other and the bullshit they had to go through when integrating features across service boundaries to get things done.
I'm assuming you're more likely talking about something like GRPC/protobuf which I have similar gripes about.
I advertise a rule set, you give me a function - pattern really - and i return all the data related to that pattern.
There are other organizations that accept a function. effectively the result is return the record in the True case, and don't in the False case.
- you build a matrix math system as a service... or
- you build cloud rendering (as they have done several times)
That sounds pretty awesome (and expensive)!
I'd argue that keeping those public is also a net good for society.
Of all the comments I've made that should have been down voted, the one I made above was it. HN gods are having mercy on me it seems.
Countries with sham elections tend to be open ballot, as the voters need to show they support the current ruler. This is why the selected candidate wins with 98% of the vote or some such.
You can't even get 70% of people to agree on which direction the sun rises.
A secret ballot is necessary for free elections.
Of course one could argue that all primaries should be non-partisan. Or that governments shouldn't spend money on running partisan primaries on behalf of political parties which are private organizations.
I just don't understand why which party I voted for needs to be "public" record. I don't believe that companies & the general public need to know. I've used campaign software that shows how incredibly easy it is to select a list of people & market towards those who voted one way or another in the primary.
Laws about this vary from state to state.
I just don't understand why that needs to be "public" record. I don't believe that companies & the general public need to know that in 2018 I chose to be part of the Pastafarian party. Maybe I want to avoid all the noodle marketing that comes with that affiliation.
Even more funny are when specifications for engines for weapon systems (tanks, aircraft, not the guns/missiles, etc) are tightly guarded secrets domestically but considered public information overseas.
I'm just pointing this out because I'd like to see a movement towards actually more open government.
Source? How were they able to do that?
If you include year, birth dates carry an awful lot of information.
The 87% paper is https://dataprivacylab.org/projects/identifiability/paper1.p.... It says:
”It was found that 87% (216 million of 248 million) of the population in the United States had reported characteristics that likely made them unique based only on {5-digit ZIP, gender, date of birth}. About half of the U.S. population (132 million of 248 million or 53%) are likely to be uniquely identified by only {place, gender, date of birth}, where place is basically the city, town, or municipality in which the person resides. And even at the county level, {county, gender, date of birth} are likely to uniquely identify 18% of the U.S. population. In general, few characteristics are needed to uniquely identify a person.”
More info likely available via http://latanyasweeney.org/work/identifiability.html (discussed in https://news.ycombinator.com/item?id=2942967)
Kind of depends on what they say is "sensitive". Probably not much of a facility for appealing that designation either.
I sorta look at it like the "First Step" prison reform act, it's only the start.
Baby steps.
I have been working for two years with a group trying to add Michigan to the list. Tomorrow I am publishing an open letter to our governor asking her to support our efforts. I will post a link on HN.
But perhaps that's the point. Shutdowns are supposed to be inconvenient.
Or by having the government continue working according to last year's budget planning, like many other countries do. I know of no government shutdown in the Western countries except in the US. Even for "third world" countries such things would be disastrous as armies don't like not being fed and paid.
In addition to the administrative burden, it appears to ignore the fact that non-sensitive information, in sufficient quantity and correlation, becomes sensitive information.
Perhaps my skepticism is misplaced, but my initial reaction is that this sounds better in the abstract than it will turn out to be in practice.
It's a meta-comment, but this term sounds slightly misplaced when you're talking about bureaucrats having to do more paperwork.
But--it's possible that better access to open data and a stronger culture in govt. around using data-driven decision and policymaking could improve on this in the long run. Not to mention the fact that much of the pre-existing paperwork requirements could be automated (if implemented carefully).
As a data analyst working for a state government, not consolidating or creating metadata really hurts my efficiency. I've gotten too comfortable with munging tables in PDFs.
>it appears to ignore the fact that non-sensitive information, in sufficient quantity and correlation, becomes sensitive information
This is something we're trying to figure out. The problem is, I doubt many agencies are actually maintaining privacy with their publications. The Census is adopting differential privacy strategies [0], but my own agency relies on practices from the days of printed reports. I know for a fact some of them don't work, but government is slow to adapt.
[0]: https://privacytools.seas.harvard.edu/why-census-bureau-adop...
There's no doubt that getting this data into a sane format will take the states some extra resources.
But when you consider how much more efficient this will make my wife's company, and every other provider of Medicaid services, it's bound to be a huge win on net. And improving efficiency of delivering healthcare should be important.
The government is big, but the private sector is still much larger. So there's great leverage to make our overall systems more efficient because an investment in efficiency on the government side will be multiplied many times over as seen by the many private entities that the government is overseeing.
However, you're right that they are notoriously stingy on buying new things if the economics aren't immediately apparent, and will never buy into something that demands radical workflow/org changes.
I (briefly) thought the same thing when I first read about it, but I think the efficiency gained from having digital, standardized formats will eventually outweigh the inefficiencies of the initial conversion to that format.
I'm also happy that otherwise "dead" data (e.g. papers sitting in boxes in a basement somewhere) could now be used more effectively in aggregate to further increase operating efficiencies. Imagine trying to put together a comparison between a specific subset of finance reports across departments when Department A uses one digital format, Department B uses another digital format, and Departments C through Z all have them in boxes. What would have otherwise been a beaurocratic headache _before you even get to data munging_ now becomes an ordeal that's easier on all fronts, and that data can then be used to fight back against otherwise unknown inefficiencies.
To make a cartoony analogy: flight security would be more efficient if everyone flew naked with no hand luggage, but that would defeat the purpose of people traveling from place to place for their own reasons.
Likewise: the government has collected or generated that info; let's put it into a reasonably clean and accessible format so others (who, in the US, have funded its collection/generation anyway) can build upon it.
I think closed data or data in pdf masks a lot of technical debt that causes manual labor, expensive proprietary licenses (looking at you SAS for archived data sets).
Poe's law strikes again.
That would be ideal, although a large portion of grant funding goes to medical research (e.g. via NIH), where it would be difficult to anonymize the data. They could require that it be sanitized (differential privacy, etc.), but I don't know how that could be verified effectively/efficiently. The grants process is quite time-consuming for the grantees (and the grantors) without this requirement.
*not a lawyer
"To balance the need for public access while protecting the research process, OMB’s revision limits the kinds of data that will be made accessible (it excludes personal and business-related confidential data) and limits applicability to federally funded data relating to published research findings produced under a federal award and used in developing an agency action that has the force and effect of law."
1) Open is better
2) Bipartisan legislation, indicates some progress is possible
Edit: It was an honest question I wanted to know the answer to, but I've got my answer.