184 karma · joined June 8, 2013
Started first ISP in Philadelphia. Spend too much time following RSS + Hackernews. Working on cloud, storage, network visibility, and general Internet plumbing.
So HW profiles can include IOPS and throughput testing, disk space, etc?
The 24x7x100% is more of a concern re: not restarting all VMs at once, or at least being able to specify subsets and rollout orders.
Will take to email to discuss further.
thanks
Some questions:
All based on what we do at Kentik, where we run a large distributed column store with high speed data ingest layers feeding it, and a relatively more easy core cluster of portal/high-level metadata nodes. Ingest nodes are 20 core low-disk and data nodes 36-core 24x2tb disk w/ ZFS, which we rely on for compression.
We use docker but on top of known equipment types of deterministic latency and performance. Machines are netbooted and much of the system is controlled by puppet push. Typical clusters range from 10 to low hundreds of nodes. We do on-prem but they are typically on top of bare metal and using our control stack.
So with that as background...
For an application that requires high speed native (in particular disk) performance and persistent disk to run, how is Gravitational requiring that at deployment time?
Is there an ability to add metrics to dashboards for on-prem 'operators' to access? Send those through logged mail gateways for SaaS companies to watch on-prem status remotely, at least at an aggregate level (dashboards plus perhaps some log excerpts)? Have part-time (customer-controlled) logged ssh gateways for supervised and/or logged remote access by SaaSco engineers?
How do you do versioning, deployment, and distributed rollout to running systems with 24x7x100% uptime expectation? Is there the ability to support partial upgrade of only some nodes of specific roles followed by automated and/or human checks? How about rollback?
Thanks, and good luck with the company!
In our case (@kentik), we're mainly focused on having a Dremel-like system accessible via FDW. Primarily we're using Postgres as the "API" to it, so if it dies as a result of an FDW bug it'd be a primary operational problem for us.
And yes, distributed discussions that preserve free speech and allow eliminating crap is definitely moon-shot hard.
Usenet is still around but mostly for binaries. The market's of pretty stable size, dominated by a few large wholesale players.
My take on what happened with text groups is that the S/N ratio just went to hell. In the 90s the problem was spam, but in the 2000s the problem was too many loudmouths who wanted to hear themselves talk drowning out the useful experts.
Like some of the other folks commenting, I've been pissed as hell at the PHPBB/vbulleting monstrosities. My original plan with readnews was to try to build a great web UI for discussion, but we got distracted by wholesale customers wanting service - and front-end is not my area of expertise.
For folks looking for something modern with promise, the news is good with discourse and a few others coming up. Would love to see something distributed, but if really distributed I suspect we'd see binaries and/or commercial spam and/or people with nothing interesting to say dominate - just like Usenet...
And luck = right product, right time, right tech, right people, and of course just luck.
The big issue we kept hearing from folks from the notes community is that the current 'cheat' is totake flow (now sFlow, IPFIX as well) and just instantly aggregate it to turn it into tsdb-like summaries.
The challenge is that you don't actually know in advance what you want to look at details for - whether you're investigating perf, availability, security, or just efficient ops.
So that's the current big problem - how to get and store and analyze thousands to tens of thousands to millions (for big networks) of flow records/second.
The 'old' problem from 1996 to 2006 of devices not being able to export has mostly gone away. There are bugs that happen every so often (like Juniper / sampled that ras called out and got resolved last year), but newer Juniper and Ciscos can do line-rate NetFlow/IPFIX to ~ 100-200k flows/sec per devices or sometimes per card. Which feeds back into point #1. Not that useful to have extra resolution if your systems are just doing top N on 10 fixed aggregation dimensions...
What I'd really like to see the silicon do is tcpstats, app decode and latency data. Both have applications for performance and security (even perf data can hint at botnets vs humans, for example, and of course URLs let you look for sql injection and app vs network problems).
The load balancer folks and some of the "Network Packet Brokers" have roadmap or the beginnings of this but it'd be better to have on Broadcom...
Or, an even more basic thing would be to have not per-packet sampling+sFlow like Broadcom and other chips do, but also a primitive for "flow sampling" to get all packets for 1/Nth of flows (say, 1/1000th) to get sampled such decodes up on a daemon running on say Cumulus or Arista.
Re: data backend and analytics - happy to discuss... A poster we did earlier this year is @ http://avi.net/flocon-hydrasql.pdf or the marketed-up version is a white paper on the site - https://www.kentik.com/wp-content/uploads/2015/06/Kentik-KDE...
We've considered OSSing the backend (what sits behind postgres via FDWs) but it'd take require expanding the team working on it beyond the current ~ 1.5 FTEs. Maybe post-B...
Another note of potential interest: We applied to YC Winter 2014 and didn't get an interview. Same team, last minute sub, not everyone was yet full time, silly idea (though we did mention as an alt what we started Kentik to do). Lessons as are noted in many places: 1) Read the reqs, 2) Work on proposal, 3) Don't stop because you don't get {in,an interview}.
Semi-related - who is the cartoonist? Can anyone (or Jason if following) xlate the signature?
Have you seen the other neat functionality about the Akamai query system? They allow (actually require) devs to create callbacks that get exposed as query tables to do debugging (vs letting devs on machines as first or second line of debugging).
It's an interesting paradigm though again the way they use it, like osquery, is only real-time and using the psql/OpenTSDB method allows for history as well as real-time.
osquery (and Akamai Query's) syntax is definitely simpler. We do plan to add a logical schema to map tags in OpenTSDB to logical columns to help with that.
At CloudHelix, we did a Postgres FDW to OpenTSDB, which gives a time dimension as well.
That was an issue at Akamai - how to get historic as well as realtime with Akamai's Query system ([WARNING: PDF direct download] http://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd...)
Interesting stuff though! Maybe a FDW could connect Postgres to osquery, which could allow joining with local tables or other FDW-accessible data.
The FDW approach to OpenTSDB looks like:
select to_timestamp(atime::float), value, hstore(regexp_split_to_array(tags, ',')) as hs from chf_realtime where i_start_time >= now() - interval'1 min' and agg = 'sum' and metric = 'df.bytes.percentused' and tags = 'host=*,mount=/|/data|/ssd' ; to_timestamp | value | hs ------------------------+-------+--------------------------------------------------------- 2014-10-30 01:15:33+00 | 84 | "host"=>"XY4.iad1", "mount"=>"/", "fstype"=>"xfs" 2014-10-30 01:16:33+00 | 84 | "host"=>"XY4.iad1", "mount"=>"/", "fstype"=>"xfs" 2014-10-30 01:15:33+00 | 9 | "host"=>"XY.iad1", "mount"=>"/data", "fstype"=>"btrfs" 2014-10-30 01:16:33+00 | 9 | "host"=>"XY.iad1", "mount"=>"/data", "fstype"=>"btrfs" 2014-10-30 01:15:33+00 | 49 | "host"=>"XY.iad1", "mount"=>"/ssd", "fstype"=>"btrfs" 2014-10-30 01:16:33+00 | 49 | "host"=>"XY.iad1", "mount"=>"/ssd", "fstype"=>"btrfs" 2014-10-30 01:14:55+00 | 63 | "host"=>"XY.iad1", "mount"=>"/", "fstype"=>"xfs" 2014-10-30 01:15:55+00 | 63 | "host"=>"XY.iad1", "mount"=>"/", "fstype"=>"xfs" 2014-10-30 01:14:55+00 | 1 | "host"=>"XY.iad1", "mount"=>"/data", "fstype"=>"btrfs" 2014-10-30 01:15:55+00 | 21 | "host"=>"XY.iad1", "mount"=>"/ssd", "fstype"=>"xfs" 2014-10-30 01:14:55+00 | 21 | "host"=>"XY.iad1", "mount"=>"/ssd", "fstype"=>"xfs" 2014-10-30 01:15:50+00 | 63 | "host"=>"XY.iad1", "mount"=>"/", "fstype"=>"xfs" 2014-10-30 01:15:50+00 | 8 | "host"=>"XY.iad1", "mount"=>"/ssd", "fstype"=>"btrfs" 2014-10-30 01:14:56+00 | 89 | "host"=>"XY.iad1", "mount"=>"/", "fstype"=>"xfs" 2014-10-30 01:15:56+00 | 89 | "host"=>"XY.iad1", "mount"=>"/", "fstype"=>"xfs" 2014-10-30 01:14:56+00 | 55 | "host"=>"XY.iad1", "mount"=>"/ssd", "fstype"=>"xfs" 2014-10-30 01:15:56+00 | 55 | "host"=>"XY.iad1", "mount"=>"/ssd", "fstype"=>"xfs" (17 rows)
FLIX was Fat man Little man Internet eXchange. I was the fat man (I used to be more engineer sized). Vinny Bono was the little man. I (AS 4969 of old) never actually connected to it though... The rest can be found in archives etc as linked below.
I remember the moment I realized who it was - had a customer who called me and said "I am advertising a /23 to you and you are advertising a more specific /24 back to me, originated from AS 7007"
http://avi.net/ch-device.png (device overview) http://avi.net/ch-ports.png (graph/table for one dimension) http://avi.net/ch-query.png (it's all SQL underneath)
We're focusing more towards networks but can do host data also (the screenshots are from a Usenet server). The UI is just a shim on top of postgres, which itself is a frontend to the clustered DBs running things on the backend.
Feel free to ping me (avi at cloudhelix.com) if you want to play around.
Anyone remember Frank Hogg for CoCo ports of Flex?
FYI, '68 Micro Journal is available online or on CD, though I have a full print set and a few SWTPCs if anyone wants to do a 6809 get together and hackfest at some point.
PROJECT: Front-end assistance with graphing interface for real-time and historic network traffic statistics.
WHO WE ARE: CloudHelix is doing a modern network visibility and control platform on top of an in-house postgres-fronted clustered column store database (API is DBI or REST).
OUR CURRENT STACK: jquery, underscore, D3, node, postgres, c, go
We're looking for something a little snazzier than Highcharts or current graphite frontends (either developed or adapted is fine), capable of doing 10-20+ 1-second refresh graphs and ideally with some interactivity in a good-looking dashboard view. Happy to provide access to our beta portal and examples of what we like/dislike and pay for some scoping time.
Could consider hourly, or a first phase of discovery/mockup then agree on a project price.
Would recommend a new SSD-based ZFS box (Samsung 840 Pros have been great even for pretty write-intesive load), with raidz3 for protection and zfs send (and/or rsync from hourly/N-minute snapshot for data protection which should eliminate copying FS metadata corruption, as not sure if zfs send will).
Happy to provide and/or host such a box or two if helpful.
They want to pay people market+ for A players, but if you're not at the level expected, you'll be let go if you don't get (back) to that level.
So a conversation shouldn't (but sometimes does) go from "we shouldn't give her a $25k raise, {s}he hasn't been doing well enough" to firing her for not improving.
On the other hand, no one is supposed to be confused about where {s}he stands with respect to performance and improvement.
Dr. Dobbs from that era vs the current or even worse, last decade's, shows the difference.
Some of HN is now a feed of that kind of attitude and enthusiasm, which is why we all waste so much time here.
Digital Guru Bookstore (and Madhuban has a great Indian buffet almost next door); Weird Stuff Warehouse (be sure to go in the back room); and there's a Fry's is close to those 2
Then slightly distant from those:
Unix Surplus (tell them Hacker News sent you and you might get a used hard disk or GBIC as a door prize, or ping me and I'll organize a time for a geek tour)