Nationwide FAA weather reporting outage
nco.ncep.noaa.gov
nco.ncep.noaa.gov
the technology was certified and frozen in the 90s. many of these devices are running windows 3.1 for networking!
they cost more than 2 million dollars a piece to install (at least around here) and require 4 trips per year to calibrate.
since there was little incentive and herculean constraints for the team in charge of METARs to improve the situation, another team at the FAA decided to create their own non certified version. to be installed in places that can't afford a METAR. They were able to build a better system using the same hardware but updated software for a hardware cost of $80k, 250k installed. this system only needs calibration once every 2 years. so it represents about a 1/10 cost of install and 1/8 cost of maintenance.
naturally this pissed off some people. I'm not going to get into internal politics beyond saying the project is currently frozen until a number of large efforts imposed on the program are completed. I believe it will still eventually come out because the cat is out of the bag and there are many groups desperate for a better alternative.
Cameras list/images: https://weathercams.faa.gov/
People adapt and say “meh it’s working well enough” because they don’t have a sane alternative.
* METeorological Aerodrome Report
Woah. Really?
I noticed that when the day was long, and I was feeling tired and lazy it was a lot easier to find some excuse why not to run that day. This excuse was often the weather. But on the other hand I didn't want to say no matter the weather I must run, because that is obviously excessive. So I made up a simple "algorithm" to decide if the weather fits the minimums, and if it did I must go and run. And I choose to base it on the measurements from the METAR of the local airport.
(I also remember I was willing to skip running in case of anything “exciting” being reported. Like tornados, or sandstorms or vulcanic ash. Not that any of that was a real possibility.)
For folks who haven't seen what's in a METAR/TAF: https://www.faa.gov/air_traffic/weather/asos/ (By default, clicking a station gives you the abbreviated format; you can select the "decode" radio button and click "update" to translate it into English.)
That doesn't help a pilot plan ahead for conditions on their route or at their destination, though-- that's where METARs come in.
Per the source, 167 airport weather stations are not reporting correctly (either due to a highly unlikely concurrent fault with the stations, or more likely an issue with the system they report to).
Local weather is always used, but that's precisely what is unavailable right now.
If the tower is closed, or it's an untowered field, the system can run in automatic mode. In that case the radio broadcast gets updated every minute, much more frequently than METARs get updated (every hour).
Stopping takeoffs was absolutely the right move, since they couldn't handle the phone throughput.
Others are potentially useful, but not essential for safe flight. Things like closed taxiways, nonavailability of services at an airport, etc.
And a very few are really critically important. Runway closures, correction to vital chart information, airspace changes, malfunctioning navigation aids.
The FAA's reaction to the NOTAM outage was probably the correct course of action. But make no mistake, the volume of spam NOTAMs combined with the lack of an easy way for a pilot to quickly sort for important NOTAMs makes us all less safe.
The system is in desperate need of some modernization, no argument there. The fact that there isn't a simple criticality filed with them that makes it easy to see what will actually impact flight planning (airspace closure / runway closure vs stupid chart updates) is insane.
I can’t say one way or the other, as I’m not a pilot, but that’s the argument I’d make to keep the system w/o priority levels.
Similarly, chart updates are very important if you're flying IFR. Going below minimums while in the soup ends... very badly. If somebody's changed the MEA/DA/MDA, you can bet I want to know.
NOTAMs do have keywords to tell you the subject (i.e. airport closure vs tower light), and most briefings will highlight the ones likely to be urgent.
(That said, I'd argue printing out all the tower notams in textual format is somewhat useless. The NEXTGEN FSS briefings are plotting them graphically now, which is an improvement.)
We have mines being blasted to our north in temporarily restricted military airspace - lets just allow pilots to roll the dice?
You look at the closest available weather station and you look at the forecasts — human-authored (TAFs) and computer-autored (GFA/MOS). And when you arrive, you look outside at the actual conditions — the windsock, actual flight visibility, etc. If the conditions are worse than expected, you divert to your alternate.
Once that happens, you have to hope that your backups work - why bother having disaster recovery drills for a static system?
Once you realize that your backups are hosed, you may experience a creeping suspicion that software is never truly static when human operators are involved.
I would not be surprised to learn that this was the case here, too.
https://twitter.com/PatrickMoorhead/status/16139375120027729...
$5 says this is the XKCD for this. Corrupted DB file was the culprit, as announced by the FAA. It's a system that relies on user inputs, often manual, and it's all being put into a database. My bet is a user decided to put in a bunch of "fun characters" to make their input easier to read. You can't account for and sanitize all levels of stupidity on user inputs.
That is false and a defeatist attitude.
Sanitization is not the right solution anyway. If you are working with any form of a database and don't know from the top of your head how to avoid query injection attacks then you should look up in the manual of your database. The solution is most often called "parameterized query" or something similar.
When arriving at an airport, a pilot is required to confirm they have the latest METAR information for the site. It's presumed negligence when pilots don't know the weather. This is because weather is the #1 environmental factor in accidents.
But the actual use of weather in flying (esp. for commercial flights) is not via METAR but via in-cockpit radar. This is an FAA-sponsored service, but it's unclear if that's affected.
So it seems to be more administrative than safety, but just as disabling.
This is the system where weather info (among other information) is transmitted from a network of ground stations.
ADS-B gives you METAR, for instance.
...METAR OUTAGE...
AOMC reported that there is a metar outage going on...167 missing sites at last check. The FAA is investigating the issue.
That seems like a lot but curious to know what percentage of their total sites this is.
I think I found the full list here: https://www.aviationweather.gov/docs/metar/stations.txt
This list is worldwide and I see no information about this outage being region specific so I'll assume it's worldwide.
Seems like they have about 9,517 sites so this is about 1.75% of their sites worldwide
(The title of the submission is also "Nationwide FAA weather reporting outage")
Yeah, but the title is made up. There is noting in the official report about this being "nationwide".
The report only states that there is a metar outage consisting of 167 stations.
It doesn't provide any context on which stations are missing or how they are distributed.
Adding "nationwide" to the title is pure FUD IMO.
The report is that 167 are missing.
That would be roughly 5% of US stations or 0.2% of worldwide stations.
I was able to look up meters for airports that I'm familiar with. https://www.aviationweather.gov/metar
So, it is real but far from a total collapse of the system.
Once you build up credibility for being right, you can expand into other more difficult areas. If you keep predicting for the underground, then it will also help keep you correct average up.
The thing is he is not. Even with the negotiations over the railroad strike, he was not talked about. It was Joe Biden in the news.
Buttigieg's biggest asset is likely his ability to get on Fox News for an interview and steamroller over their usual rhetorical tactics.
Is some group pen-testing?
It could be Russia, Iran, China, climate direct action groups (probably not y'all qaeda though). Or maybe it's just ancient software running on older hardware that's starting to fall over after a couple years of lighter loads and a sudden increase in complexity and weirdness. Everything is breaking at the moment.
Classic buffoonery. Until it's not.
Just a tiny bit of blackmailer-sabotage or vandal mischief-making – the sort of thing often underreported for legitimate reasons! – could've been the margin between some mild embarassment & the total collapse we saw. (For example: tying up the call-in lines to turn a normal hold time of minutes into hours, or long busy signals.)
Simple monocausal stories are for unsophisticated spectators.
So you just inherently believe all that is reported as fact?
Remember when the Gulf of Tonkin incident was “for real really real” and we invaded Vietnam?
Or the WMDs in Iraq?
I’m not tinfoil hatting - Just calling out how interesting it is 3 major events in a documented as fragile industry happen so close.
That’s literally all I mentioned in my orig comment.
Well, no, the US military was (openly) in Vietnam in support of South Vietnam long before the Gulf of Tonkin incident. Heck, the US military was there (again overtly) in support of France for several years before South Vietnam existed as such. And, ironically, they were there in support of the Viet Minh – which later became both North Vietnam and the Viet Cong – even before that, with almost no break between, starting in July 1945.
Insofar as there was an international incident that led to the US invasion of Vietnam it was – though it took a few years for the response – Pearl Harbor, not the Gulf of Tonkin. All the subsequent fighting in Indochina was breakdown in relation between erstwhile allies who were all already present, after the Japanese were driven out and had surrendered.
Tying that to this thread: why is what’s reported about the airlines taken to be fact? (And I’m not speaking as if I mean this is malicious (as the Tonkin incident was) - just reported incorrect from fact for other “capital friendly” (face saving) reasons.)
Didn’t someone say “believe half of what you read…” somewhere?
I just don't think it's ruled-out by the simple official story. Some but not all of the reasons why I believe there might be more to the story are based on decades of life experience:
* Whenever the media reports on something I know about intimately, they get it wrong, preferring pleasing simple narratives to the full details. Hence, I assume their stories on things I don't know intimately show a similar skew towards oversimplification, missing-details that are embarassing to the key actors, and audience-pleasing explanations.
* I've not ever been in IT security for a major or public company. But even in tiny organizations, I've observed strong incentives – not all misguided! – to downplay malicious mischief as a contributor to any problems. Organizations don't want to encourage the perpetrators with publicity, nor encourage copycats. Orgs also don't want to be embarassed by lax measures. From direct reports from individuals at larger organizations – and reliable public accounts of late-reported hack/extortion incidents – I believe these incentives can be even stronger at large, slow-moving, distributed-responsibility public companies (though of course the penalties for explicitly-misleading statements also larger).
* Plenty of mean, crazy, or self-interested people may have it out for Southwest, from previously-angered travellers to disgruntled employees to motivated short-sellers (individuals or formal funds). And even if the potential for sabotage was under formal investigation right now, the investigators – private or public – might want to hide that fact until definitive evidence collected & perpetrators are prosecutable. It can take months or years for the real story to emerge!
A fragile outdated system finally reaching a chaotic breaking point is one possible & sufficient answer, of course.
But it's also a potential weak-point to be pushed-over-the-edge by motivated saboteurs or extortionists. In fact, such a weak point is ideal for certain criminal schemes, because of its deniability by both perpetrator and victim as merely a problem of aged systems & incompetence.
So, only the naive would rule it out entirely based on only self-serving public narratives.
> So, only the naive would rule it out entirely based on only self-serving public narratives.
I don't technically disagree, but it is also just as naive to ignore our human proportionality bias and discount the relative probability of the available evidence.
I'm not saying it definitely was an attack, nor even that it likely was. Just that it's premature to "definitely" rule it out so soon, a mere 3 weeks out, given the organizational incentives involved & base rates of both extortion/vandalism & (often well-meaning but at the very-least ass-covering) reporting-misdirection about the same.
You seem to agree with me that it remains a possibility, so not sure we actually disagree about any particulars of the event, just the discussion.
Imagine there were a well-refereed, bettable proposition like, say, "By the end of 2030, will either (a) someone be criminally charged for contributing to the Southwest service disruptions of late 2022; or (b) will a knowledgeable Southwest insider or law enforcement agent report they saw evidence that intentional acts worsened the Southwest service disruptions of late 2022?"
Just from base rates of such mischief, & without yet digging deeper, I'd consider an answer of "YES" to have around a 2-3% chance. And something with a 1-in-50 chance of having happened is absolutely a valid topic of speculation deep in forums like this!
Such tail events are where lots of the big wins, & big losses, for industry & society arrive. But also, such real-but-rarer outcomes get habitually ignored by simple mainstream summary coverage, which needs to put neat bows on stories for uninformed & distracted audiences, by short deadlines, reliant on spin from involved entities.
To protest deep in the threads, and insist that a few-weeks old official-sources story is "definitely" the whole explanation, case-closed, stop-speculating-its-hurting?
That's actually hostile to helping curious people understand a complex world based on limited & conflicted information sources.
Are you sure you are not seeing selection bias because the majority of users on this site are Americans?
“We” don’t know the causes, only what was reported.
I’ve worked on a lot of old systems needing to be updated, migrated, or sunset. For sure it’s 99% of the time due to management being lazy. I get that.
And I know correlation/causation, but hey my orig comment was a simple curiosity.
Sure, my nature distrusts what a corporation says because it’s always filtered through stockholder functions, but I still think “3 major airline issues in as many months” is kind of hilarious. Even if they seem to be in different facets.
Does this happen often but only now is it reported to “us”? Or is this truly an odd and rare timing?
OK, but in Southwest's case, you had former and current employees from various positions within the company publicly coming out and lambasting Southwest for the failure to actually invest in upgrading their scheduling system that led to the outage. It wasn't just a corporation coming out and going, "Here's what we hope you believe happened," it was pissed off employees publicly fucked off that their years of complaints went unheard and now they were all paying the price for being ignored.
>Or is this truly an odd and rare timing?
It's just odd/rare. I would wager that your brain's rush to try and tie them together like this is just it's way of trying to make sense of something it normally doesn't see.
Meaning - a system is so old, slow, bug, and vulnerability prone a bad actor can pen-test around and mess stuff up?
There doesn't seem to be any flight delays though or notices on the airport website...
This isn't sufficient of course but when the FAA repeatedly demonstrates incompetence, when do we eventually just tell them no, you can't have any more of my money, you're obviously wasting it and making us worse off?
If you did you’d be a consumer, not a taxpayer.
Wow, it could be. Can you share the FAA's budget this year compared to 2019 so we can see how it's been constrained?
Have you looked at the numbers?
Have you?
https://theblogbyjavier.com/2020/01/02/aviation-safety-evolu...
See the chart labeled "Evolution of accidents per million flights". Millions more flights, and a steady march towards zero deaths. We haven't had a commercial airliner crash in the US for over a decade.
The FAA's numbers look great.
Imagine what NASA would do in ten years with Pentagon's budget swapped.
NASA's budget is now $0 though, since the military budget all went to Medicare with nothing left to spare for anything else. Along with the FAA and every other government department.
https://en.wikipedia.org/wiki/Medicare_(United_States)
>In 2020, US federal government spending on Medicare was $776.2 billion.[3]
https://en.wikipedia.org/wiki/United_States_Department_of_De...
>Annual budget US$721.5 billion (FY2020)[2]
Hopefully, I won't need a citation that $776 > $721
> NASA's budget is now $0 though, since the military budget all went to Medicare with nothing left to spare for anything else. Along with the FAA and every other government department.
Meanwhile other first world nations pay about half that and get mostly better results...
Anyone think the NSA budget for "defense" is included in Pentagon, DoD, Homeland Security?
Anyone think the trillion dollars in weapons being sent to Ukraine for our proxy war with Russia is in any visible budget?
Medicare meanwhile cannot hide all the spending, it's right there, even the fraud in the system is visible in the spending. But defense spending hides twice maybe even three times the spending.
The previous president significantly cut the FAA's budget, maybe we're just seeing now how that affects us in the long term
There's a neat trick people do nowadays where people claim anything that's unsuccessful (usually government related) was either "sabotaged" or "set up to fail" as a way of absolving them of any failures. Literally impossible to criticize.
Finally, the FAA's budget was $23 billion this year and $28 billion last year due to pandemic / infra surpluses, nearly double what it was the last 4 years, where it remained flat (not cut as you say). So it's funny how 2 years of double funding can't get it back on track, but flat funding has caused it to fall apart in the same time period.
Presidents don't pass budgets.
FAA funding has increased, not decreased.
I can criticize my government all I want. You're turning this into a political party thing when it's a government service failure.
Seems to run in the family. The voters in Wyoming appear to agree.
Oh it's Trump fault Liz did what she did? Nope. Liz thought she'd make a name for herself, she did, and is reaping the results of her actions.
Liz Cheney is completely irrelevant here.
Post FAA's ACTUAL BUDGET over the last 8 years and let us judge.
I'm just contesting the idea that the President has no power over the US Federal budget.
(I'm of the general opinion that the FAA is fairly well funded, has been very effective at its mission, and could still do with some specific capital projects to modernize non-sexy stuff like NOTAM and METAR handling.)
Yes there is.
> The president recommends and Congress passes budgets for internal purposes
This is simply false.
> but they do not carry the force of law.
Yes, budget bills, like other bills, have the force of law once passed and either signed by the President or vetoed and the President’s veto overridden by both Houses (the President’s budget proposals do not have the force of law, just like other unpassed legislative proposals.)
> The numbers that matter are how much Congress appropriates
“The budget” is just the aggregate of tax and appropriations bills (sometimes, there is an annual package entitled a “comprehensive” or “omnibus” budget bill that covers some tax policy and most or all of the annual appropriations, but that’s rare, and even in that case there are typically continuous and multiyear appropriations and tax policies that are left untouched and outside of it, but which are considered part of the “budget” even though they are aren’t part of the “budget bill”; more commonly, this doesn’t occur.)
> if Congress appropriates $X dollars for something the executive branch is expected to spend it, not second guess Congress and say we really didn’t want to spend that money.
While this is true, and more than just a soft expectation, since Train v. City of New York and the Impoundment Control Act of 1974, it is not invariably the practice notwithstanding the expectation and the law (cf., the withholding of funds appropriated for Ukraine aid that was central to Trump’s first impeachment.)
First of all congress generally does not pass budget bills, they pass budget resolutions, and congressional resolutions are not submitted to the president for his signature and do not have the force of law.
As far as budget bills (like the Budget Act of 1974) are concerned, it is impossible for a previous congress to pass laws that govern the actions of succeeding congresses in a binding fashion because such laws have the character of congressional rules not statutes, and under the Constitution each house has complete authority over its own rules. So Congress can simply neglect to pass a budget resolution and the only remedy is for individual representatives to raise points of order. The executive branch cannot indict or prosecute anyone in Congress for neglecting to follow congressional rules or laws that have the character of congressional rules.
On the other hand, if you wish to refer to an appropriations bill as a "budget bill" you are merely disagreeing with me (and with congressional practice) over nomenclature. Personally, I have never heard of an appropriations bill referred to as a "budget bill", but I am sure someone does it.
The budget resolution is a planning framework for the appropriations bills that make up the budget. It is not a “budget”.
> and congressional resolutions are not submitted to the president for his signature, and congressional resolutions are not submitted to the president for his signature and do not have the force of law.
That's not true. All Congressional actions are resolutions (e.g. the USA PATRIOT Act was "House Resolution 3162" of the 107th Congress), and many end submitted to the President and with the force of law. Some are even primarily characterized as "resolutions" after they are signed and have the force of law (See, in terms of the budget space, the use of "continuing resolutions".)
There are types of resolutions that are nonbinding, and do not have the force of law, and the "budget resolution" (which, again, is not the same as the budget) is one of them, but this isn't generally true of "congressional resolutions".
> As far as budget bills (like the Budget Act of 1974) are concerned, it is impossible for a previous congress to pass laws that govern the actions of succeeding congresses in a binding fashion
The 1974 Budget Act is not (relevant to this discussion) a budget bill, it is a meta-budget bill, or maybe even a meta-meta-budget bill (that is, a bill on the process by which Congress will plan future budgets.) You are correct that it is not binding on future Congresses, but this has no bearing on the budget not being binding, merely a past Congress's plan on how to plan annually to arrive at a budget is not binding.
> So Congress can simply neglect to pass a budget resolution
…which isn’t a budget, but a plan for one…
> and the only remedy is for individual representatives to raise points of order.
Well, sure, but the budget resolution isn’t the budget. If Congress doesn’t pass a budget (or only passes part of one), then the remedy is “spending governed by the parts not passed stops” (that is, a government shutdown.) This is…rather noticeable.
> On the other hand, if you wish to refer to an appropriations bill as a “budget bill” you are merely disagreeing with me (and with congressional practice) over nomenclature. Personally, I have never heard of an appropriations bill referred to as a “budget bill”, but I am sure someone does it.
While it is common to refer to the aggregate of appropriations as a “budget”, its not usually used for individual appropriations bills as a “budget bill” unless they are a single consolidated bill, which is typically referred to as a “budget bill”. Budget bills are frequently introduced, but less frequently passed. But not having a single budget bill passed is not “not having a budget” or “not having a budget with the force of law”, its “not having a budget in which the annual appropriations were adopted in a single bill”. Which is…less significant.
Guess what the President's budget proposal says on the cover?
"Budget of the U.S. Government, Fiscal Year 2023"
And if we look inside do we find a list of congressional appropriations? No. Instead we find a list of appropriations requests and explanatory information for congressional appropriators. Typical chapter headings have language like this:
"The Budget requests $773 billion in discretionary funding for DOD, a $69 billion or 9.8-percent increase from the 2021 enacted level."
So I am sorry, I cannot take seriously the idea that appropriations bills are "budget bills" or specify a federal "budget" except in an informal sense, one contrary to that used by the federal government itself.
> All Congressional actions are resolutions.
I certainly agree in the technical sense, and since we are debating nomenclature here I should be more careful.
As far as Cheney is concerned, Trump had been out of office for months by the time she lost her leadership position. Anything he did was from the sidelines by that point.
A budget is a budget even if it is neither comprehensive (i.e., all functions in one bill) nor for a full fiscal year at a time nor uses the word “budget” in its title.
So a federal agency like the FAA generally prepares a budget recommendation that would be submitted to Congress as part of the president's budget, but then Congress would decide how much (if any) to appropriate for each activity, down to the level of individual programs in many cases. By the time Congress gets done with it becomes a list of appropriations, and the agency has no authority to transfer things around, spend money on what Congress did not authorize, or even purposely refuse to spend money (or spend less without a good reason) on what Congress has appropriated funding for.
Relevant quote:
"President Trump’s support for a plan to lop more than 30,000 Federal Aviation Administration workers from the federal payroll gives fresh momentum to an effort that stalled in Congress last year.
The proposal is included in Trump’s 2018 budget, which would cut funding for the Transportation Department by 13 percent."
The FAA's budget was not "significantly decreased" as the comment above claimed.
No, they propose, sign, use the veto threat to shape the legislative process on, and use various administrative powers and means to rearrange the actual spending resulting from them, but “passing” budgets is the name for a step in the middle of the process done by someone else.
Not sure what your point is; the President is by a very wide margin the single most powerful actor in the shaping of both nominal budgets and actual federal spending.
Especially when evaluating the FAA would need to include the absolutely stellar commercial aviation safety record they and the NTSB have created over the last few decades.
Does your tax burden stay the same, and they just make sure your dollars only go to the programs you support? Do they give you a list of every active program? Every single one? What happens with 800 page appropriations bills where you don't support one line-item? Can I opt not to fund the salaries of specific members of Congress? If my neighbor is a veteran, and they annoy me, can I disagree with their VA benefits?