Hacker News Reader

Comment by Laremere | Hacker News Reader

I disagree. I interned twice at a telco, and while I had access to a bunch of information in databases as a developer, I had no access to call records, and in general you didn't have access to it unless you needed it for work. The problem is as I understand it, the NSA currently has the view that if a corporation has access to it, then it's not private and they get free reign. It's not that I don't expect Google to have my gmail records, or my phone to have call records and geolocation. Those companies can use that data to improve their services to me. It's when the NSA decides that they have access to all of this information from all these companies is when I have a problem.

View on HN

I too interned at a telco and I was given free reign over this kind of data -- and more. As were some external, temporary contractors.

I wrote a little about it here: https://news.ycombinator.com/item?id=5969957

Different companies, different policy, I worked in customer care, and later in a quasi-engineering role. In both cases, and two different companies I had full access to customer CDR's.

Which two? I'd appreciate being able to avoid those companies.

That's kinda silly unless you can confirm that the other companies are not doing exactly the same thing.

It's really all of them - If your doing a job that involves either customer care, or resolving call based issues, you have the ability to: See call detail records, sometimes capture voice calls to and from that subscriber, capture signalling data in realtime, track a mobile user in call thru the system for the purposes of finding defective network elements.

There is a good reason why we had these abilities, they were essential to resolve and troubleshoot user reported issues.

For example, if a customer reported troubles or call quality issues calling 202-555-1212, I needed to be able to pull their CDR's so I could determine which outbound trunk group the call went out on, so I could report it to that carrier to see if they could resolve.

> There is a good reason why we had these abilities

Bullshit. Systems can be designed to allow troubleshooting and problem resolution without giving broad access to user data. Telecoms are just too lazy to do it. They have local monopolies, so they just don't need to care.

I've written tools for VoIP companies for diagnostics. The most helpful system is one that logs everything and has it available on-demand. The one I wrote literally recorded and indexed every call-related packet (except audio) and records were kept indefinitely. This is network-level traffic. It reduced time-to-resolution by at least an order of magnitude, since a ticket to the effect of "this weekend someone at the office called someone in Germany and it didnt work" could find out exactly what happened.

Apart from that, call detail records (CDRs) are available to many people in a telecom. And it doesn't matter if you trust your telco, because they probably hand off traffic to others in many cases. Even large companies like AT&T deal with very scummy third-parties to get the cheapest prices to place calls. (I know that first hand, because sometimes people sell call capacity they don't have. Instead of completing the call, they answer and play a fake recording of someone saying "wait a sec". Once, when dialing from my AT&T cell, I got the same exact recording I had PCAP'd from some third-rate provider.)

Unlike email, where you don't need to provide access to email records in most cases, CDRs are needed for all aspects of billing. While your complete profile may be relatively safe in a large company with internal controls, depending on your calling patterns, you may very well be leaking a lot of details to many third parties you've never heard of.

Edit: You could theoretically add all sorts of controls and access-limitations. There's really zero benefit to doing so. Plenty of companies still rely on techs being able to run tcpdump on a mirror port or server in question.

Pretty much exactly my point of view.

We had the problem with low cost carriers too - though it was usually a quality issue, some customers because of the nature of their business (e.g. lots of fax calling) we would push them on to generally high cost TDM trunks via the local LEC versus lower costs VoIP trunks with level3 or other carriers.

I wish I could convey to people that they need to trust that their carrier is not just sharing their info beyond what is required to make the network work and what is legally required of them by the federal/state/local government.

I wish I could also explain more about how the telephone network, works. Like, how its virtually impossible to record every single call made in the US in a blanket sort of way, and that every transiting carrier from here to there also has a slice of their information.

> You could theoretically add all sorts of controls and access-limitations. There's really zero benefit to doing so.

Wow. That, right there, is the problem. You don't have to care about other people's privacy, so you don't. You're not accountable, and neither are the companies you work for.

I hope there's a disgruntled insider at one of the companies that uses your tools, and they vacuum up all the data they can find and leak it. Maybe a public scandal could make you care, if it affected your bottom line.

Have you ever worked in a support role?

Hah. I don't mean to be condescending, but the security in telecom is laughable. I've tested nearly a dozen systems and networks, and in every case never had to spend more than a day getting root. (I'm sure there are decently designed ones, but that's not the common case.)

I had the CTO of one of the major switch vendors tell me that "buffer overflows aren't a problem, unless maybe the network is very fast". This is a company that markets an edge device with security being one of the main bullet points.

Not to mention, I'm willing to bet in many datacenters, you could literally just walk in and take disks out of machines. If they've got RAID and no alarming on RAID (not too uncommon) no one will even notice. Sure, some places have solid cages or require escorts. But a lot of colo space is relatively unmonitored and so long you look like you belong there and get past perimeter security (as easy as renting 1U in the same place, often) you're totally in.

And the sad part? To my knowledge, these attacks aren't that widespread. Why? Because there's even lower-hanging fruit that's more lucrative to hackers.

Anyways, to your point: How would this access control work? How are you going to verify a CSR actually got a call from a specific customer? How much effort are you going to put in verifying that data? This is an industry where tons of money is charged and moved around based purely on emails received, with zero verification. Also remember that full wiretap capabilities are a federal requirement (CALEA). So somewhere, they'll have a system that can turn on lawful intercept. That avenue is probably ripe for abuse.

You're right in your overall sentiment. The FCC and federal government should enact privacy laws and force companies to internalize this externality. Until they do, there's little reason to make it any more difficult for support to do their job.

Reminds me of a linux based edge router/SIP ALG product we used at a former employer if you put it on a 20 meg or faster symmetrical pipe, it would fall over, and eventually hardlock. Bugs in the drivers and a lack of hardware acceleration for the ethernet cards.

A popular ISP in one country shipped default DSL modem/routers with a SIP ALG on that didn't work. But not just the normal "totally screwing SIP up" but the alarm light would go on and the modem would be dead until power reset. On any SIP packet. This was over 6 years ago, but I think it was a ZyXEL.

the ZyXEL Modems were horrible in general. My preference was to use an Innoband 8012 and let the customer get their own router.

So, a typical scenario.

"I had an incoming call come in at 3pm, no, I dont know the number, and I couldn't hear them/poor quality audio, can you check and see what was wrong?"

"Everyone who calls me from johnstown says I have a really bad echo"

These require either the calling party or the called party, you need to be able to locate what network element they touched. At every company I had to sign a generic NDA to not disclose customer information without consent from the company. In the end though, you don't own this data, the company you contract with for service does - you'd need to change the contract with them to get control over it.

Adding the restrictions you propose would just make things more complex, not better. In the end, someone you don't want to have access will still have access - you need to trust the company you do business with to have good policy restrictions to protect their data, and your privacy.

You're just illustrating my point. Telecom systems are currently designed without privacy in mind at all. It would be possible to design these systems such that privacy was respected, but telecoms just don't care.

> you need to be able to locate what network element they touched

That doesn't require or justify you having blanket access to call metadata as a customer service technician. Instead, in a reasonable system, users would grant limited access to account data on a case-by-case basis. For example, a user could use a website or app to report a recent call as bad quality, which would grant access to a technician for that call only so they could investigate.

The number of people in the company with blanket access to all user data should be very, very low. Certainly nobody in customer service should have it.

The telecoms legally own this data, its theirs, and its needed for all sorts of internal functions beyond just customer care - but lets focus on customer care:

Without this access how would you:

Create Trouble Tickets on call quality when the customer cant give very precise information?

Give credits on miss dialed calls to overseas destinations?

Explain charges on a customers bill?

Open tickets on calls that cant complete - the example above "someone called Germany on Saturday, can you check and see why it didnt go thru?"

I mean, I'd be interested on what form you think these access systems should take.

There should be better ways to report call quality issues than calling up to wait on hold for an hour to tell someone "fix it for me". Problem reporting should be mostly if not entirely automated. Google Voice, for example, has a "report bad call quality" option in its list of recent calls. Reporting other kinds of issues like failed calls could happen the same way.

For situations where phone support is still required, automated systems can still enforce that a technician can only access data relevant to the user who is calling, and that access can be logged, audited, and reported to the user on their next bill.

The important thing is that users should give limited consent to access data for troubleshooting, and that access should be enforced by automated systems, not NDAs. Technicians do not need and should not have blanket access to user data.

Really, a lot of the problems you describe have been created by the telecoms for themselves. Telecoms rely on antiquated and terrible old equipment, and that causes the call quality issues. If they invested in updating their infrastructure, they could have consistent better quality calls and automated monitoring. But they have no incentive to update their infrastructure when they already have oligopolies or monopolies. My phone has a 10 Mbps connection to the internet and yet phone calls still sound like they're happening underwater; it's a joke! Billing complexity is an even more obvious example of a problem the telecoms have created for themselves.

Ironically, the carriers who have the most billing and call quality troubles are not the legacy incumbent local exchange carriers, its the newer VoIP based carriers.

In your 10mbps scenario its either a lack of QoS on your circuit (either from you or your ISP) or poor generally qualities of the end to end connection - like too much jitter or something.

Even if we had an automated ticketing system that let a customer pick which call was bad, they wouldn't use it. Also keep in mind, every customer along the path would need to do this. There might be 5 or more companies involved in completing a single call.

The end-user might report it to their local PBX guy. He's going to call his wholesaler and say "yeah sometime over the weekend, someone made a call from somewhere and something didn't work right". If the wholesaler is lucky and can comb through data, they might find the call in question. Sometimes the answer is "uh, you hung up" or "...yeah, so you got a busy tone". Most of the time, if you can actually find a problem, you forward it on to the offending carriers, and they repeat the search and forward it until it hits a real network. The answer to the end-user is mostly "Sorry about that, we've made some routing updates." and hopefully they won't experience the issue again. If the end-user repeatedly has issues, it's probably a question of their line not having proper QoS, or something else their local PBX guy needs to sort out.

The "antiquated and terrible old equipment" is actually the least likely to cause issues. As much as I hate telecom, the real switches do a solid job. Dedicated hardware on TDM leaves far less place for problems. VoIP is the real quality issue most of the time.

I swear - did we work at the same company?

You're echoing my experiences in such complete detail, its almost scary.

Reply on news.ycombinator.com