HNHacker News
TopNewBestAskShowJobs

avifreedman

184 karma · joined June 8, 2013

Co-Founder of Kentik, which helps the people that run the Internet to run their infrastrucutre.

Started first ISP in Philadelphia. Spend too much time following RSS + Hackernews. Working on cloud, storage, network visibility, and general Internet plumbing.

submissionscomments
avifreedman··on A Brief Guide to Philly Tech Scene
Don Feith is a great guy. He was a gentleman and took time to answer my questions about the business world (and the Unix-y one in particular) in the 80s and 90s. Hope his business is doing well, and if they're hiring I'd definitely give it a look.
avifreedman··on Gravitational (YC S15): Software On-Prem or in Cloud from Single Code Base
avi @ kentik is my email but I'll ping you.

So HW profiles can include IOPS and throughput testing, disk space, etc?

The 24x7x100% is more of a concern re: not restarting all VMs at once, or at least being able to specify subsets and rollout orders.

Will take to email to discuss further.

thanks

avifreedman··on Gravitational (YC S15): Software On-Prem or in Cloud from Single Code Base
Hi, looks interesting...

Some questions:

All based on what we do at Kentik, where we run a large distributed column store with high speed data ingest layers feeding it, and a relatively more easy core cluster of portal/high-level metadata nodes. Ingest nodes are 20 core low-disk and data nodes 36-core 24x2tb disk w/ ZFS, which we rely on for compression.

We use docker but on top of known equipment types of deterministic latency and performance. Machines are netbooted and much of the system is controlled by puppet push. Typical clusters range from 10 to low hundreds of nodes. We do on-prem but they are typically on top of bare metal and using our control stack.

So with that as background...

For an application that requires high speed native (in particular disk) performance and persistent disk to run, how is Gravitational requiring that at deployment time?

Is there an ability to add metrics to dashboards for on-prem 'operators' to access? Send those through logged mail gateways for SaaS companies to watch on-prem status remotely, at least at an aggregate level (dashboards plus perhaps some log excerpts)? Have part-time (customer-controlled) logged ssh gateways for supervised and/or logged remote access by SaaSco engineers?

How do you do versioning, deployment, and distributed rollout to running systems with 24x7x100% uptime expectation? Is there the ability to support partial upgrade of only some nodes of specific roles followed by automated and/or human checks? How about rollback?

Thanks, and good luck with the company!

avifreedman··on Ask HN: Do you use an old or 'unfashionable' programming language?
Me too, Perl most days, C most weeks.
avifreedman··on On choosing the Z80 over the 6502 (2014)
6809 for the win over both, by far!
avifreedman··on PostgreSQL Foreign Data Wrappers
Good advice...

In our case (@kentik), we're mainly focused on having a Dremel-like system accessible via FDW. Primarily we're using Postgres as the "API" to it, so if it dies as a result of an FDW bug it'd be a primary operational problem for us.

avifreedman··on I'm in my 40s, trying to apply for YC next year
Worst case it helps you focus on what/why and explaining your startup. Best case you get in. Worst case just continue. In fact, if you wouldn't start the company without getting into YC, you probably should re-think whether you're serious enough about it.
avifreedman··on Show HN: A terminal Space Invaders game
What did you do the recording in?
avifreedman··on Ask HN: What was Usenet's ultimate demise?
We'll loop detect to end the mutual admiration society, but thanks from you makes my day :) I just tweeted a pointer to "Doing Terrible Things" and suggested people read all of the "Falsehoods Programmers Believe". I want "Falsehoods as a Service" - outsourced paranoia.

And yes, distributed discussions that preserve free speech and allow eliminating crap is definitely moon-shot hard.

avifreedman··on Ask HN: What was Usenet's ultimate demise?
Sorry, ran out of bits. At least I didn't go signed.
avifreedman··on Ask HN: What was Usenet's ultimate demise?
(Background: I'm a Usenet user from the late 80's to early naughties, did outsourcing at netaxs as newsread.com, then ran readnews.com from 2004-1024).

Usenet is still around but mostly for binaries. The market's of pretty stable size, dominated by a few large wholesale players.

My take on what happened with text groups is that the S/N ratio just went to hell. In the 90s the problem was spam, but in the 2000s the problem was too many loudmouths who wanted to hear themselves talk drowning out the useful experts.

Like some of the other folks commenting, I've been pissed as hell at the PHPBB/vbulleting monstrosities. My original plan with readnews was to try to build a great web UI for discussion, but we got distracted by wholesale customers wanting service - and front-end is not my area of expertise.

For folks looking for something modern with promise, the news is good with discourse and a few others coming up. Would love to see something distributed, but if really distributed I suspect we'd see binaries and/or commercial spam and/or people with nothing interesting to say dominate - just like Usenet...

avifreedman··on What Makes Founders Succeed
To be successful, you need to put in the work to enable yourself to take maximum advantage of getting lucky.

And luck = right product, right time, right tech, right people, and of course just luck.

avifreedman··on Network Processing Powerhouse Kentik (CloudHelix) Raises $12.1M Series A
So interesting... flow (first NetFlow) has been around since the mid 90s but it's always been a volume problem relative to the (single) computers of the day.

The big issue we kept hearing from folks from the notes community is that the current 'cheat' is totake flow (now sFlow, IPFIX as well) and just instantly aggregate it to turn it into tsdb-like summaries.

The challenge is that you don't actually know in advance what you want to look at details for - whether you're investigating perf, availability, security, or just efficient ops.

So that's the current big problem - how to get and store and analyze thousands to tens of thousands to millions (for big networks) of flow records/second.

The 'old' problem from 1996 to 2006 of devices not being able to export has mostly gone away. There are bugs that happen every so often (like Juniper / sampled that ras called out and got resolved last year), but newer Juniper and Ciscos can do line-rate NetFlow/IPFIX to ~ 100-200k flows/sec per devices or sometimes per card. Which feeds back into point #1. Not that useful to have extra resolution if your systems are just doing top N on 10 fixed aggregation dimensions...

What I'd really like to see the silicon do is tcpstats, app decode and latency data. Both have applications for performance and security (even perf data can hint at botnets vs humans, for example, and of course URLs let you look for sql injection and app vs network problems).

The load balancer folks and some of the "Network Packet Brokers" have roadmap or the beginnings of this but it'd be better to have on Broadcom...

Or, an even more basic thing would be to have not per-packet sampling+sFlow like Broadcom and other chips do, but also a primitive for "flow sampling" to get all packets for 1/Nth of flows (say, 1/1000th) to get sampled such decodes up on a daemon running on say Cumulus or Arista.

avifreedman··on Network Processing Powerhouse Kentik (CloudHelix) Raises $12.1M Series A
Hi, Avi Freedman here, CEO of Kentik (formerly CloudHelix). Thanks for the mention, Ryan... And for being a human vulnerability scanner for netaxs back in the 90s :)

Re: data backend and analytics - happy to discuss... A poster we did earlier this year is @ http://avi.net/flocon-hydrasql.pdf or the marketed-up version is a white paper on the site - https://www.kentik.com/wp-content/uploads/2015/06/Kentik-KDE...

We've considered OSSing the backend (what sits behind postgres via FDWs) but it'd take require expanding the team working on it beyond the current ~ 1.5 FTEs. Maybe post-B...

Another note of potential interest: We applied to YC Winter 2014 and didn't get an interview. Same team, last minute sub, not everyone was yet full time, silly idea (though we did mention as an alt what we started Kentik to do). Lessons as are noted in many places: 1) Read the reqs, 2) Work on proposal, 3) Don't stop because you don't get {in,an interview}.

avifreedman··on I sent one of the best developers I know to a job interview, he was rejected
Please introduce him to us at CloudHelix (avi @)... Always looking to meet wizards as long as they are of reasonable ego:clue ratio :)
avifreedman··on Becoming more formidable
Great article...

Semi-related - who is the cartoonist? Can anyone (or Jason if following) xlate the signature?

avifreedman··on Osquery: Expose the operating system as a relational database
Cool. Would be interesting to see if that comes up as a common request; we have found the ability to do real-time and historic to be interesting for query-ability for system data.

Have you seen the other neat functionality about the Akamai query system? They allow (actually require) devs to create callbacks that get exposed as query tables to do debugging (vs letting devs on machines as first or second line of debugging).

It's an interesting paradigm though again the way they use it, like osquery, is only real-time and using the psql/OpenTSDB method allows for history as well as real-time.

osquery (and Akamai Query's) syntax is definitely simpler. We do plan to add a logical schema to map tags in OpenTSDB to logical columns to help with that.

avifreedman··on Osquery: Expose the operating system as a relational database
marpaia, Did you consider using OpenTSDB?

At CloudHelix, we did a Postgres FDW to OpenTSDB, which gives a time dimension as well.

That was an issue at Akamai - how to get historic as well as realtime with Akamai's Query system ([WARNING: PDF direct download] http://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd...)

Interesting stuff though! Maybe a FDW could connect Postgres to osquery, which could allow joining with local tables or other FDW-accessible data.

The FDW approach to OpenTSDB looks like:

select to_timestamp(atime::float), value, hstore(regexp_split_to_array(tags, ',')) as hs from chf_realtime where i_start_time >= now() - interval'1 min' and agg = 'sum' and metric = 'df.bytes.percentused' and tags = 'host=*,mount=/|/data|/ssd' ; to_timestamp | value | hs ------------------------+-------+--------------------------------------------------------- 2014-10-30 01:15:33+00 | 84 | "host"=>"XY4.iad1", "mount"=>"/", "fstype"=>"xfs" 2014-10-30 01:16:33+00 | 84 | "host"=>"XY4.iad1", "mount"=>"/", "fstype"=>"xfs" 2014-10-30 01:15:33+00 | 9 | "host"=>"XY.iad1", "mount"=>"/data", "fstype"=>"btrfs" 2014-10-30 01:16:33+00 | 9 | "host"=>"XY.iad1", "mount"=>"/data", "fstype"=>"btrfs" 2014-10-30 01:15:33+00 | 49 | "host"=>"XY.iad1", "mount"=>"/ssd", "fstype"=>"btrfs" 2014-10-30 01:16:33+00 | 49 | "host"=>"XY.iad1", "mount"=>"/ssd", "fstype"=>"btrfs" 2014-10-30 01:14:55+00 | 63 | "host"=>"XY.iad1", "mount"=>"/", "fstype"=>"xfs" 2014-10-30 01:15:55+00 | 63 | "host"=>"XY.iad1", "mount"=>"/", "fstype"=>"xfs" 2014-10-30 01:14:55+00 | 1 | "host"=>"XY.iad1", "mount"=>"/data", "fstype"=>"btrfs" 2014-10-30 01:15:55+00 | 21 | "host"=>"XY.iad1", "mount"=>"/ssd", "fstype"=>"xfs" 2014-10-30 01:14:55+00 | 21 | "host"=>"XY.iad1", "mount"=>"/ssd", "fstype"=>"xfs" 2014-10-30 01:15:50+00 | 63 | "host"=>"XY.iad1", "mount"=>"/", "fstype"=>"xfs" 2014-10-30 01:15:50+00 | 8 | "host"=>"XY.iad1", "mount"=>"/ssd", "fstype"=>"btrfs" 2014-10-30 01:14:56+00 | 89 | "host"=>"XY.iad1", "mount"=>"/", "fstype"=>"xfs" 2014-10-30 01:15:56+00 | 89 | "host"=>"XY.iad1", "mount"=>"/", "fstype"=>"xfs" 2014-10-30 01:14:56+00 | 55 | "host"=>"XY.iad1", "mount"=>"/ssd", "fstype"=>"xfs" 2014-10-30 01:15:56+00 | 55 | "host"=>"XY.iad1", "mount"=>"/ssd", "fstype"=>"xfs" (17 rows)

avifreedman··on Why old versions of 286 Xenix did not run on 386s
I set up a number of small companies in the Philadelphia area with Foxbase running on Xenix with terminals on each desk connected to a cheapish PC. Much cheaper than Netware and separate PCs!
avifreedman··on The Day the Internet Died – Courtesy of the Florida Internet Exchange
It wasn't the Florida Internet Exchange.

FLIX was Fat man Little man Internet eXchange. I was the fat man (I used to be more engineer sized). Vinny Bono was the little man. I (AS 4969 of old) never actually connected to it though... The rest can be found in archives etc as linked below.

I remember the moment I realized who it was - had a customer who called me and said "I am advertising a /23 to you and you are advertising a more specific /24 back to me, originated from AS 7007"

avifreedman··on Propose HN: Screenshot Saturday
CloudHelix is in beta with a portal and data platform for network visibility and control - top talkers, dDoS detection, and mitigation hooks.

http://avi.net/ch-device.png (device overview) http://avi.net/ch-ports.png (graph/table for one dimension) http://avi.net/ch-query.png (it's all SQL underneath)

We're focusing more towards networks but can do host data also (the screenshots are from a Usenet server). The UI is just a shim on top of postgres, which itself is a frontend to the clustered DBs running things on the backend.

Feel free to ping me (avi at cloudhelix.com) if you want to play around.

avifreedman··on Ask HN: CTO wants me to leave
Are you good at seeing the products that need to be created? Writing 'running specification' code (hacking things up)? Do you understand the product space well? Who owns product? Is this something you do or could do? How good is your relationship with your cofounder? There is a lot more that involves tech understanding to be done in a startup than architecture, coding, and managing architecture and coding.
avifreedman··on Motorola 6809 emulator in JavaScript
There was also the GIMIX, and of course later the Radio Shack Color Computer. A (ugh single 5.25" floppy) SWTPC 6809 system was my first "home" computer system. Flex was a great simple operating system.

Anyone remember Frank Hogg for CoCo ports of Flex?

FYI, '68 Micro Journal is available online or on CD, though I have a full print set and a few SWTPCs if anyone wants to do a 6809 get together and hackfest at some point.

avifreedman··on Ask HN: Freelancer? Seeking freelancer? (March 2014)
SEEKING FREELANCER - Local (SF Bay area) or remote.

PROJECT: Front-end assistance with graphing interface for real-time and historic network traffic statistics.

WHO WE ARE: CloudHelix is doing a modern network visibility and control platform on top of an in-house postgres-fronted clustered column store database (API is DBI or REST).

OUR CURRENT STACK: jquery, underscore, D3, node, postgres, c, go

We're looking for something a little snazzier than Highcharts or current graphite frontends (either developed or adapted is fine), capable of doing 10-20+ 1-second refresh graphs and ideally with some interactivity in a good-looking dashboard view. Happy to provide access to our beta portal and examples of what we like/dislike and pay for some scoping time.

Could consider hourly, or a first phase of discovery/mockup then agree on a project price.

avifreedman··on Tell HN: Server Status
Assuming the disk footprint is small...

Would recommend a new SSD-based ZFS box (Samsung 840 Pros have been great even for pretty write-intesive load), with raidz3 for protection and zfs send (and/or rsync from hourly/N-minute snapshot for data protection which should eliminate copying FS metadata corruption, as not sure if zfs send will).

Happy to provide and/or host such a box or two if helpful.

avifreedman··on Ask PG: Postmortem of the outage?
Gluster should by design I think avoid replicating filesystem metadata corruption (but would replicate internal metadata issues in files on top of the filesystem) but DRBD won't... At high volumes I still regularly break Gluster but it'd probably be OK for lower bandwidth/ops use. Not sure what the HN disk usage pattern is though.
avifreedman··on Netflix: Non-'A' Players Unworthy of Jobs
A friend of mine is a director at Netflix. He was explaining to me that compensation vs retention are different conversations there.

They want to pay people market+ for A players, but if you're not at the level expected, you'll be let go if you don't get (back) to that level.

So a conversation shouldn't (but sometimes does) go from "we shouldn't give her a $25k raise, {s}he hasn't been doing well enough" to firing her for not improving.

On the other hand, no one is supposed to be confused about where {s}he stands with respect to performance and improvement.

avifreedman··on “WarGames” Magazine Identified
The magic of the magazines from that era is that they were full of articles from people showing "look what I did with my computer!"

Dr. Dobbs from that era vs the current or even worse, last decade's, shows the difference.

Some of HN is now a feed of that kind of attitude and enthusiasm, which is why we all waste so much time here.

avifreedman··on “WarGames” Magazine Identified
For those talking about sequels, FYI... There is a WarGames sequel on Netflix but I made it 3 minutes in before abandoning. Can anyone recommend it?
avifreedman··on Going to Silicon Valley? Plan carefully
Also, if you're into the nerdly arts I recommend as part of "sightseeing" (all clustered pretty nearby):

Digital Guru Bookstore (and Madhuban has a great Indian buffet almost next door); Weird Stuff Warehouse (be sure to go in the back room); and there's a Fry's is close to those 2

Then slightly distant from those:

Unix Surplus (tell them Hacker News sent you and you might get a used hard disk or GBIC as a door prize, or ping me and I'll organize a time for a geek tour)

← PreviousPage 2 of 4Next →