cliffcrosland

tl;dr don’t use an architecture from the on-premise era where storage and compute are coupled together.

Great project. Zig and Rust working together in harmony to make it easier to cross-compile Rust for different targets. We use this to great effect to build for x86 and ARM chips in AWS.

We're really liking the combination of using Rust, AWS Lambda functions, and ECS Fargate. Everything runs well in ARM, but make sure to use `musl` instead of `glibc` - we've found memory usage to be almost 10x lower with `musl` than `glibc` for our use case (indexing and searching logs). If anyone wants to talk about the tech, happy to answer questions!

Yeah I think this is where Datadog excelled and Splunk struggled. Design is clearly better at Datadog, especially the onboarding UX and workflow.

Maybe, but the Splunk query language is reasonably well liked by its users, at least in the security space. Much more approachable than SQL, which seems to be what all new tools these days are forcing users to use due to their dependence on Snowflake and Presto/Trino. In Splunk, you can type free text queries, and you can also add structure. Fairly flexible. We’ve been asked many times to make Scanner’s query lang more like Splunk’s.

I worked at Cisco following an acquisition. I tend to agree with this comment about the upper stacks :)

However, AppDynamics and Duo seem to be doing well at Cisco from what I can tell. I think observability and security tools are a good match for Cisco and bundle well with hardware. For this reason, I’ll bet Splunk does reasonably well under Cisco too.

We’re also giving this a shot. The annual Splunk bill at our last startup exploded from $10k to $1M when we reached 1TB of logs generated per day, which is actually an easy threshold to hit when you have decent traction and aren’t proactively reducing logs. So we built Scanner.dev to drop these costs by 10x.

Decoupling compute and storage is definitely the way to go. We’re using Lambda functions and ECS Fargate containers for compute that scales up and down rapidly, and S3 for storage. Getting ~1TB/sec log scan speeds, which feels fairly good. We keep sparse indices in S3 to narrow down regions of logs to scan. Eg. if you’re searching for an IP address that appears 10 times in a 25TB log set, the indices reduce the search space to around 300MB. Takes a few seconds to complete that query, whereas Athena and CloudWatch take like 20 minutes.

We’re also using Rust to maximize memory efficiency and speed - there are lots of great SIMD optimized string search and regex libraries on crates.io.

We’re early, so there are a lot of SIEM features like detection rules that we are still building. But Splunk/DataDog users might find it useful if costs are a problem and they use mostly log search:

https://scanner.dev

How do you think Splunk will fare as more companies move to public clouds? Seems unbeatable on-prem, which probably means that the company is a good match for Cisco.

Agreed. Costs plummet when you use S3 as the storage medium for these massive log data sets. I think S3 is much faster to query than most people realize. Just have to be smart about how you organize things.

Looks really cool! Great work!

Had a pricing question. Say we connected a log forwarder like Vector to send data to Clickhouse Cloud, once per second. If each write unit is $0.0125, and we execute 86,400 writes over the course of the day, would we end up spending $1080? Do you only really support large batch, low frequency writes?

We are using Rust to build a fast log event database. The workload includes a lot of asynchronous networking I/O and parallelized CPU-intensive work, like indexing and SIMD-accelerated string search. There are great libraries for what we need. The performance and low memory footprint are quite useful.

> But the biggest reason to RiiR is safety.

100% agree. We had a C++ service that made heavy use of libcurl. A particular release of libcurl introduced some memory safety problems that caused frequent segfaults for us. These memory safety bugs were eventually fixed in another release, but it scared us enough that we investigated rewriting the service in Rust.

After successful experimentation with some prototypes, we eventually rewrote the service in Rust, auditing the unsafe code in our dependencies (of which there was very little). No segfaults ever since.

Side benefit: Since Rust networking libraries tend to have strong async support, and since some of our C++ libraries performed synchronous networking operations, we saw a big improvement in performance. The number of threads needed dropped by 5x.

Reading an enumeration of architecture options can be helpful, but I've probably found more value in works like Ousterhout's "A Philosophy of Software Design" that cover fundamental design principles. Such principles seem to be applicable no matter the project.

I appreciate that Ousterhout tries to support these principles with some data from his own experience (building Tcl, RamCloud where Raft consensus was discovered, etc) and from code reviews of student projects in his design course, but he admits that there is a lack of data about things like design.

Sample principles from Ousterhout:

* When developing an interface, favor exposing a small number of functions each with deep functionality over exposing a large number of functions each with shallow functionality. For example, favor Unix file I/O interface (open, close, read, write, unlink) over Java file I/O interface (File, FileReader, BufferedReader, FileWriter, BufferedWriter, FileNotFoundException, ... etc.)

* When choosing between keeping your code specific and keeping your code general, favor "somewhat" general. For example, instead of exposing "deleteNextCharacter(text, pos), deleteNextWord(text, pos), deleteNextSentence(text, pos)", just expose "delete(text, pos, length)" and let the caller use it how they will.

To be honest, I feel bad for the engineering team at Equifax. The vulnerability that compromised their system was a bug in an open-source Java library, Apache Struts, and security researchers only noticed it a few days ago. It seems that the Equifax team had very little time to react and update their software. In some sense, I feel that more blame should be placed on the engineers who built the highly popular open-source software, not the Equifax team. Some large number of Fortune 100 companies also experienced the same vulnerability simply because they trusted a widely used library.

Makes me wary of trusting other big OS libraries, but since rebuilding every part of the stack from scratch is infeasible and unproductive, we don't have much choice but to use them.

Technical announcement:

Severe security vulnerability found in Apache Struts using lgtm.com (CVE-2017-9805):

https://lgtm.com/blog/apache_struts_CVE-2017-9805_announceme...

My father saw his biggest entrepreneurial success as a father of four kids in junior high and high school. I believe his schedule looked roughly like this:

- 6am work at home

- 8am work at office

- 6pm dinner with family at home, or cheer at kids' athletic events

- 8pm work at home

- 11pm or 12am, sleep

- In early days, he slept on couch near computer to take customer support calls at night.

It wasn't until I was older that I realized how much my family was prioritized.

That said, my mom wanted to be a full time caregiver and did an amazing job of it, so his caregiving responsibilities were not nearly as extensive as my mother's.

Strangely enough, Stoicism. I try to recognize what I can control and what I can't, and I try not to let things I can't control scare me into inaction.

Luminus - compilation of Clojure libraries for web dev. Postgres.

We use C++ heavily at Accompany. The CTO co-founder was one of the chief engineers at Google Analytics where most of the highly scaled systems were written in C++. Perhaps you could try investigating startups where the founders have a Google engineering pedigree?

Anecdotal data point: my father was an entrepreneur for 100% of my childhood. It was rare for him to miss dinner with us, and he attended easily more than 60% of our extracurricular events. He'd often work from home late after we went to sleep, sometimes sleeping next to the phone to take customer support calls 12-5am. He had freedom to spend time with family and worked sufficient hours to achieve successful outcomes for his businesses. I think this form of scheduling is effective. Another anecdote: apparently Sheryl Sandberg also eats dinner with her family every day but works from home late at night and early in the morning.

Thank you for sharing. Despite some scientific fumbles, I thought Interstellar did a great job portraying the gravitational lensing effects of black- and worm-holes. I just wish that the worm-hole travel scene would have been as short and strange as this proof-of-concept rendering. Will definitely pore over these materials later. Much appreciated.

An excellent point. The pointer is indeed distracting. I just removed it from my demo, and it's much better. Thanks

Your demo is excellent! If you don't mind, I might dig into the source later to compare our approaches. I'm thinking about doing a 3D visualization sometime, and I'm interested in seeing how other people solve the problem.

Great point about the warping issue in my demo. I use a solver to produce a quadratic polynomial that maps in-angles to out-angles. I feel like the coefficients produced by the solver don't fit the data very well for pixels far away from the black hole, resulting in zooming / warping. I'm going to dig into it a bit and see if I can produce a better polynomial.

UPDATE: Just made a quick change. The solver was only attempting to match data up to, but not through, the FOV limit, which meant that the polynomial did not fit data near the FOV limit very well. I updated it so that it now uses data up to and beyond the FOV. There is still a bit of warping for far away pixels, but there is substantially less than before. Thank you for the tip.

When I was a freshman in college, I was in a humanities class that focused on the intersection between humans and machines. We had an assignment to build and test out a "prosthesis", i.e. a technology that extends human capability, in Second Life.

I created a simple wristwatch accessory that was scripted to upload a copy of all of your chatbox text to an external service. Later, you could log in to this external service and search through a history of all of the conversations your character ever overheard.

Real-world versions of this technology appear inevitable as digital storage costs trend to zero. A rudimentary digital copy of the physical world is being created in services like Google Maps. The Google self-driving car records a 3D copy of its surroundings with accuracy at the centimeter level. Dropcam uploads video and audio data from within your home to cloud storage.

A world with fully recorded life experiences seems creepy at first blush, but I believe we'll discover a mechanism for trust that will allow everyone to safely record a digital copy of their lives that is inaccessible to third parties. Perhaps in the future we'll each own an open-source private cloud container of CPU and storage resources. Instead of processing your data on external servers, third-party services might provide code that runs in your own container under tight network permission restrictions. Such a system might be able to maintain the benefits of continuous software deployment while allowing consumers to keep their data under their control.

The assumption that startup success relies on enduring 80 hour work weeks is a commonly held one in Silicon Valley, yet there are some remarkable counter-examples. Dropcam, for example, made it an explicit part of their culture that employees work a focused 40 hour work week and have dinner with family and friends outside of the office. The CEO believes that startups who offer dinner at work are basically manipulating employees to work late into the night [1]. Did these sane, healthy work policies hold Dropcam back from massive success? Nope. It was acquired for half a billion dollars [2]. For the sake of our team members and employees, I think we would do well to emulate Dropcam.

[1] http://www.xconomy.com/san-francisco/2013/04/23/dropcam-ceos...

[2] http://techcrunch.com/2014/06/20/google-and-nest-acquire-dro...

The next job can absolutely be used as training for your next gig. Also, the next job can be your financial foundation as you start your next gig on the side. Running a startup should not put your family's livelihood at risk. My father is an entrepreneur and frequently had several small side projects going on at any given time while I was growing up. Right now, he's focused on building up one business, but another business he built up previously is still bringing in solid income. Several great companies started as side projects while the founders were occupied with some other "real" job: GitHub, Google (perhaps debatable, but it started while the founders were full-time students), Facebook, Basecamp, and the list goes on and on. I once read James Altucher explain that he stayed at his "real" job for two years while he built up his side business. By the time he quit to work on his startup full-time, the startup was already extremely healthy with several full-time employees and enough customers to profitably pay everyone a great salary. Of course, you'll have to talk to your new employer to work out the legality of creating side projects. But if you pursue building something up on the side, you'll be joining a long tradition of entrepreneurs who have done the same, some of whom have built companies that will be remembered for centuries.

I second what patio11 said about the low prevalence of modern frameworks. Even at the most advanced startups I've seen up close, more of the software is built using custom frameworks than using "cutting-edge" modern frameworks. Adopting each new modern framework has a cost, and if a startup's problems are largely orthogonal to the problems the framework solves, adopting the framework is unjustified. I can remember only one job interview where extensive Rails experience was a hard requirement, but it was probably because the company's founding team wanted help learning Rails. You should be able to find many jobs where experience with modern frameworks is not a hard requirement.

If I were in your situation, I'd do two things. First, I'd write a list of 10 ideas to make my career really fun. This list would probably describe companies to interview at and projects to pursue. Second, I'd write a list of 10 ideas to make my current job really fun. This might require some extreme creativity, but it might be better to make a possibly futile attempt to enjoy the current job while working on finding the next one than continue to waste away in misery.

Here are a few crazy ideas that come to mind. Your ideas will be better:

1. Make a bingo board where each square is a common annoyance you encounter in the code. Share combinations of the bingo board with your co-workers. Give a candy bar to whomever gets bingo first.

2. When faced with an annoying task, set a countdown timer and do everything you can to finish the task before the time runs out. Don't let the bomb explode.

3. Stand up and do a fist pump every time you make an especially clever commit.

4. Listen to music that is fun.

5. Periodically listen to comedy that makes you laugh.

6. Think of the people you admire or respect in your workplace. Try to meet with them and learn from them. Give back to them too - perhaps by giving them ideas that you think could possibly help them in their projects, or introduce them to people you know who could help them.

7. Bring in snacks or some particularly tasty treats to share with your co-workers. It might help build some emotional connection and friendship.

8. Bring a bag of M&Ms and eat one every time you save a file.

9. Try working for an hour using only one hand.

10. If you're in an office setting where talking out loud won't bother other people, try narrating everything you do during the work day like how YouTube stars narrate playing video games. An immensely talented engineer I knew at an internship would sing rap lyrics all morning and swear vigorously at his machine when things went badly wrong. It was ridiculously entertaining.

Best of luck making your career more enjoyable!

I think VR could be one of these bottom-up forces of transformation. There is no fundamental limit on virtual lands and goods. In VR, every person could have his/her own virtual palace without imposing burdens on other humans (except for maybe on the sysadmins running the VR servers). Even if your physical surroundings are modest, your virtual surroundings could be breathtaking.

With more of our lives occurring in the inexpensive digital world, perhaps we'll be less concerned with the expensive physical one and thus less concerned with working for money.

If you love what you do for work, it is probably sustainable to work for more than 4 hours per day. We all do things during each of the 16 hours of consciousness per day. Some of those things are more enjoyable than others. If you find yourself in a situation where you must work more than 4 hours per day, and it is not enjoyable, there appear to be two options:

1. Find work that you enjoy doing more than 4 hours per day. 2. Learn to love what you do for work.

In the healthy romantic relationships I've seen, partners fall in love, but they also choose to love one another. I wonder if loving work is similar. There is a lot of rhetoric encouraging people to follow their passions to find the jobs they love, but there is little rhetoric about choosing to love the job you have. It's important not to delude yourself that a horrible job is lovable, but it's also important to notice all of the positive aspects of a job and how you can use it to grow and give back.

On the other hand, studies also show that "radical collocation" can increase software development productivity. This study from the University of Michigan claims that productivity was doubled when developers worked together in specially designed "war rooms" that allowed for both spontaneous meetings and moments of solitude: http://www.researchgate.net/publication/220879336_How_does_r...

This quote summarizes the results well:

"Although the teammates were not looking forward to working in close quarters, over time they realized the benefits of having people at hand, both for coordination, problem solving and learning. They adapted to the distractions of radical collocation, both by removing themselves to nearby hotelling areas when they needed privacy, and by zoning out, made possible because of the distance between people in the larger rooms."

It feels productive to enter a flow-like state for several hours debugging a problem, but it's probably more productive if a teammate notices and provides the solution in minutes. Nevertheless, stretches of uninterrupted focus are often essential.

We have a C# asp.net MVC web app that we are transforming into a consumer of independent node.js and python web services. I must admit that I sometimes miss the static type refactoring tools visual studio offers when I'm wrestling with grep in node.js code, but my team really appreciates the open source tools and deployment options that come with the OSS territory.