sixdimensional

This, to me, is a general problem with a purely quantified, metrics driven approach to management, not just limited to NPS. I'm not demonizing these approaches, just saying that you have to balance humanity with data.

I like to look at this kind of thing as the one of the dark sides/dark patterns of using data for decisionmaking.

Qualitative measures are important, as is maintaining as much humanity as possible, to have a balanced and healthy culture. Being solely metrics/data driven can lead to cold, heartless, damaging culture (might be efficient or make profit, but very dehumanizing).

This seems like a case for an axiom I hear infrequently, but I think comes up a lot - things that seem like they should be simple and easy, but are in fact difficult.

To answer my own question, I suppose both roles (CIO and CTO) benefit from having coding experience, maybe CTO more than CIO assuming a CTO is supposed to be "closer to the technology engineering".

I am a little confused why we are talking about CTO and not CIO (chief information officer). I thought the latter is the role responsible for information systems.

Although for years we have been told that CIO means "career is over" and was a dead role. Which I never fully understood.

"Essentially the government has built a surveillance state by outsourcing it to private enterprise."

There's a rather innocuous sounding name for this - "public private partnership" [1]. If you've ever experienced this scenario first hand, you'd truly be surprised how much government is run in partnership with private enterprise.

[1] https://en.wikipedia.org/wiki/Public%E2%80%93private_partner...

I am following you here, but I also took this another way.

I think what this comment was about was a person who recognizes his own mortality but could put that against the backdrop of the bigger picture of our own shared human mortality.

I believe Shatner was recognizing the pure inhospitability of space to human life and the fact that we are propelling ourselves into it. If anything I think this was precisely a commentary on the very nature of the risk and adventure that space represents. Also a recognition of how much we thread the needle even just living on the Earth.

For if it were not that true exploration and adventure came at risk of life and limb, if the very risk of living did not include the risk of death, then what is it really? Thinking of the early explorers of the Earth, many who died doing so, and who risked it all in the name of exploration and adventure.

If I ever heard a more realistic characterization of what it truly means "to boldly go" where no one has gone before... I can't think of it.

I am eager to see further debriefing from Shatner, I think he will have much to say. In so far as they wanted to achieve sending a different kind of personality to space, to learn what human experience would be like, I think this was an incredible success.. putting all the commercial aspects aside.

For what it's worth, I actually do understand your sentiment. I am also a pragmatist and practitioner and find that academic theory sometimes has to "bend" for lack of a better word.

That said, I was brought up to find what is useful and good regardless of whether it comes from academia or practice. It's a beautiful thing to be able to find a balance of building an operational world view that borrows from both theory and practice.

It is my personal observation that much theory is based on observation of practice + theoretical projection, followed by more observation - i.e. the scientific method.

I loved Adam Savage's comment that sometimes science boils down to doing stuff and writing it down. I think as long as we find a good balance in learning from the past and building on it, while not letting the past limit our future endeavors for no reason, we can have the best of both worlds.

Actually, there is a entire field dedicated to this (disclaimer: I studied it at uni, plus CS and info systems) - organizational studies [1]. It has subdomains such as organizational structure/models, behavior, communication, etc. Much of it backed by theories, studies and more, ranging in disciplines.

Time and motion studies (for example) were part of the scientific management revolution for industrial management [2]. There have been both qualitative and quantitative studies of all kinds of things - organizational forms, people networks (things like, Dunbar's number[3]), power distribution (ex. work of Pfeffer), etc. I could go on.

I do think we are reliving an era of interest in management by data and metrics, much like that of the industrial revolution and scientific management. Nothing wrong with using science and quantitative measures to optimize, but any human who has been subject to purely management by quantitative objective will likely tell you it often becomes.. rather, inhumane. This is often what led to automation, I feel - to remove the human element that was crushed by industrial efficiency.

I suspect this is why the qualitative balance is important (and no less scientific - science can be logic not just metrics right?).

My two cents...

[1] https://en.wikipedia.org/wiki/Organization_studies

[2] https://en.wikipedia.org/wiki/Time_and_motion_study

[3] https://en.wikipedia.org/wiki/Dunbar%27s_number

I like this idea and I do wonder how it might fair on specialized niche search engines too, which likely would have much smaller indexes. I'm always looking for "simpler" architectures for small but custom web search, and I like what the author put together here!

I think you bring up a good point. I do believe that, especially at large scale organizations with many suborganizations, the situation you describe can occur even more frequently.

At the end of the day, we are all surrounded by different scales of organization, whether it be a small family unit, formal and informal networks, societal organizations and institutions, businesses, etc.

There has to be a balance, because many organizations want flexibility, "democracy", the ability to change direction quickly, autonomy in suborganizations, etc. Yet, this must be balanced with the need for some kind of order against pure chaos or harmful organization actors (for instance). This implies some kind of governance, some kind of "ground rules" if you were, for things to function and be reasonably effective/efficient.

I think this is where organizational culture truly matters. I do think culture can be encouraged, and norms and behaviors modeled, and sometimes even formally enforced, but ultimately the net governance that emerges at scale is a sum of the parts (i.e. the whole is greater). I do not know if, at the kind of organizational scale I'm talking about (many thousands of people or more), it is possible to govern everything with absolute precision and control.

Rather, at scale, one can only model the desired outcomes, norms and behaviors, and/or set policy. Utimately what the organization is - all its boundaries and suborganizations and networks all the way up and down the scale of an organization - is determined by the sum of those who participate and represent the organization. It does leave holes where those with personal agendas or sociopathic tendencies (for example) may take hold.

By all this, I mean to say that, I do not believe it is just in recesses where sociopaths grow most influential. I believe they grow wherever their influence has meaning and they have power/control.

The only thing that can protect organizations against such behavior ultimately is the collective will of the members of the organization, at all levels of scale. The culture, and members of the organization are the ultimate tool for protecting the organization against such threats.

Even with laws or policies, everything depends on enforcement. Enforcement also depends on culture, in addition to power and control. I believe though, that in the long run, everything depends on what culture allows, and only culture can stop sociopaths from being ultimately successful and protect organizations from such threats.

I'd go one step further. I have done research on "silos" and actually believe they form naturally, even virtually, and we should be looking how to take advantage of the natural tendencies and make them more effective rather than eradicate them because it sounds like a good thing to do.

I'm not saying a return to pure tribalism here, but I am saying maybe realistic acknowledgement of tendencies and how to make those more effective/efficient and less problematic might be a good idea.

In other words, turn a negative into a positive. Work with the tendencies with incremental improvement.

There's another way of thinking about silos - call them organizational boundaries - and observe a complete shift in thought on the topic.

Some might call this heretical thinking.

Yeah, the original comment is way off. I worked for a major electronics manufacturing company helping with demand driven MRP (DDMRP, MRP=material requirements planning) and obsolescence (DMSMS - diminishing manufacturing sources and material shortages).

I can definitively confirm that just-in-time is about trying to optimize the amount and strategic positioning of buffer needed to support the "right amount of production" to meet anticipated and actual demand, exactly as you describe.

Now, doing that can be rather difficult in the case of black swan events. But the goal is as you describe.

I went on vacation recently to a town with spotty electricity, no cell service for my carrier and barely functioning Internet. I didn't realize there would be no cell service or Internet on my vacation. I brought only books, binoculars, clothes, paper/pen, food and myself. I stayed in a motel/cabin kind of lodging and did a lot of mindless driving.

It was somewhat accidental forced unplugging/disconnecting.

Between work stress, COVID, and my own slight Asperger's, I struggle to get myself out of mental loops.

During this time away, my first few days, my dreams and every moment I closed my eyes was a flush of memories or thoughts of work. It almost felt like my brain was trying to process a backlog of stress and that process was "flushing" those thoughts (or brain chemistry) out.

I went hiking, read some, and mostly took it easy, walked around, ate and rested.

It took at least 3-4 days of a 8 day trip for my brain to barely start calming down. I had to keep repeating the "simpler activities" and just put out of mind literally everything else. I literally had to "escape" from my normal life entirely.

I don't say this to be run of the mill, I literally had this anecdotal experience but a few weeks ago.

I felt a little more like I had a handle on myself after this break, but only in a "just almost barely" kind of way. In other words, I had just started to come out of my loops and my fog, when my trip ended.

Just like the work had a lingering effect, so did taking the time off. I am back at work and finding myself going back to some of my loops, but my trip gave me a little break and opportunity to take some perspective.

Therefore, my non-scientific anectodal recommendation is to study your own rhythm and see how it changes when you disconnect or take a break, especially if it includes an aspect of "slightly forced disconnecting". If you can learn to become more self aware of the rhythms you experience or fall susceptible to, it might give you a chance to subtley shift or do something actively to tweak them in a way that helps you.

I think it helped me, at least temporarily.

You didn't connect to drill using JDBC? You can use any open source GUI with that, like DBeaver.

Somebody needs to tell the guy who made this YouTube video about this: Alugalug cat x the Kiffness https://youtu.be/S61ENc51Z1Q

Do you keep your laptop on your lap in this chair, or sit at a desk?

I found the posture in a similar zero gravity chair super comfy but, I don't think I could sit at a desk with one.

The zero gravity chair idea reminds me of this insane thing: https://arstechnica.com/information-technology/2013/06/seat-...

Not to mention, nothing on cloud egress fees.

That is probably one easy way to encourage competition. Eliminate the egress fees.

If it costs nothing abnormal to use services across clouds or on premises, then you can likely pick and choose best of breed anywhere as long as there is a decent network.

I'm super impressed by what Pallet[1] is doing in this space.

I think LA is talking with Pallet to implement their temporary housing, which would be amazing.

[1] https://www.palletshelter.com

I feel there is a "seesaw" effect here though - relatively speaking - where when the conditions are right (peace/stability), the needle tends to head towards one direction (decentralization), but when the conditions are different (conflict/instability), the needle often heads the other way (centralization).

There's also the thought that decentralization is seen as a threat by centralized players, therefore, even if it could be more effective, it may be prevented or delayed from taking hold.

I like your train of thought here, it's a good "noodle". If I might add some more to the noodling... :)

I see an interesting alignment between the trends we are seeing in technology and politics. Actually, my senior thesis back in college (2003) was that the politics of information technology are disruptive to organizations. Not necessarily disruptive in a "sided" sense, in that "disruption" can be considered good, or bad, healthy or damaging, or both.

There is a trend we see in information system/technology architectures towards "decentralized" and/or "federated" solutions. For example - "service mesh" [1] or "data mesh" [2], both higher level stack abstractions that came from lower level mesh networking concepts. And we even see "mesh" concepts in the form of knowledge graphs, social networks, peer-to-peer, etc.

If you read up on this you will often see the word "federated" and it is so interesting that federation is both something we apply to technology/systems as well as political. I am quite sure that it is part of the key to what you are thinking about. There's a reason that countries have been referred to as "federations", why we call it the U.S. "federal" government, etc. I believe, among other things, it largely revolves around how one implements governance and ensures a working "system" in terms of standardization/policy.

There is a desire to be autonomous yet interoperable, and this applies as much to "human" systems as well as purely technological systems. There is a link, after all, between how humans implement systems per Conway's Law [3] - organization's systems often are designed to mirror their communication structure.

Institutions are just one kind of organizational unit, and there too you come across all kinds of interesting issues that have perhaps, barely been explored - one such example is "institutional trust" [4].

In summary, my own latest "noodling" on this area is that actually, our technological and political systems are still very much evolving. When you take into context the possibility that it is truly becoming possible within, how about we say, the next 100 years - that humanity my be able to truly exist off the Earth, in space, for longer periods of time - then that is going to open up a whole world of questions about these very systems which we think we have completely figured out.

If there is one constant I think we can count on, it is probably change - but, ultimately survival (a key element of which is adaptability) will probably be the force that drives whatever the next 100 years look like - and that could be both good or bad overall for the development of new organizational structures.

Personally, I feel that all of this is interwoven together in quite an interesting way and all related, as information systems are the backbones of organizations.

[1] https://en.wikipedia.org/wiki/Service_mesh

[2] https://martinfowler.com/articles/data-monolith-to-mesh.html

[3] https://en.wikipedia.org/wiki/Conway%27s_law

[4] https://en.wikipedia.org/wiki/Institutional_trust_(social_sc...

Apache Zeppelin is one open source option - https://zeppelin.apache.org.

For what it's worth (sample size of 1 - me) - if you are at all competent and bad at setting boundaries, the dysfunction will absolutely frustrate the heck out of you.

Seeing what is broken, knowing that it can be fixed, and having even the slightest bit of motivation or knowledge of how to do so, and how it goes on forever and ever and doesn't get fixed becomes incredibly tiring.

I suppose, if you go into a dysfunctional company hoping to "coast", just be aware, if you are not good at managing boundaries and passing off the "frustration", it might be a lot worse than you realize.

Just imagine.. if you are spending 8 hours a day, 5 days a week, watching a slow motion wreck.. and powerless to do anything about it... sure, you get paid, but is it worth spending the time?

Here's an idea... what if search became a peer-to-peer standardized protocol that is part of the stack to complement DNS? E.g. instead of using DNS as the primary entry point, you use a different protocol at that level to do "distributed search". DNS would still play a role too, but if "search" was a core protocol, the entry point for most people would be different.

Similar to some of the concepts of "Linked Data", maybe - https://en.wikipedia.org/wiki/Linked_data.

The problem is getting to a standard, it would essentially need to be federated search so a standard would have to be established (de facto most likely).

Also, indexes and storage, distribution of processing load.. peer-to-peer search is already a thing, but it doesn't seem to be a core function of the Internet.

This is basically the same concept as making an "open" version of something that is "closed" in order to compete, I guess.

There could be a lot of reasons that are highly engine dependent, for this specific case.

A general answer, perhaps... not specific to the case you've specified.

Query optimization is a science with multiple dimensions [1]. I'd wager every problem in computer science plays a role somehow in query optimization.

Query performance is based on a combination of actions you take to optimize the design of your system to get the best performance (e.g. data modelling, index design, hardware, query style, and more), and the patterns the system can recognize based on your inputs and the data itself, with the resources it has available, to optimize your queries.

There are known patterns for optimization that are discovered over the years, many hard learned from practical experience. This is why older "popular" engines sometimes are more mature and more performant - they have optimizations built for the common use cases over long periods of time. That is not to say older engines are always better, just that they have often had more exposure to the variety of problems that occur.

The reason why the engine "can't figure it out" is that most engines, even the best ones, are quite complex - combinations of known rules as well as more fuzzy logic, where the engine uses a combination of information and heuristics to essentially explore a possible solution space, to try to find the optimal execution plan. Making the right decision, well, can be difficult and given the nature of these things, sometimes the optimizer makes the wrong decision (this is why "hints" exist, sometimes, you can force the optimizer to do what you see is obvious - but this is suboptimal for you).

In some cases, finding an optimal execution plan can actually be quite computationally expensive, and/or quite time consuming, or the engine in question may simply have no logic coded to handle the case. Optimization is all about finding the balance between finding the most performant query plan, but in the least amount of time, with the least computational and I/O impact to the overall system, that returns the right result. Optimizers are also highly depending on the capabilities of the engineering teams that build them.

It is not an easy problem, and it is an area which one could liken to almost machine learning/artificial intelligence, in one way. There are so many possible options, the problem space so big, with so many different ways to approach a given scenario, that it can be difficult for the "engine" to decide.

This is why known patterns were created, for example, dimensional data models for analytical queries vs. 3rd normal form. Dimensional data models enable certain optimizations, for example, star schemas [2]. If you take a combination of implementing known patterns, along with optimizers written by engineers that exploit those patterns, you can get to a world of better performance.

However, in a world that is, let's say.. more "open ended" - for example, the world of data in a "data lake", where data models are not optimized, data comes in unpredictable multiple shapes/sizes, then it often comes down to combinations of elegant/complex engines that can interpret the shapes of data, cardinality, and other characteristics, make use of much larger distributed compute and system performance, and in some cases - often brute force to arrive at the best query plan or performance possible.

There are so many levels of optimization.. for example, if you were to look at things like Trino [3], which started its genesis as PrestoDb in Facebook - you will see special CPU optimizations (e.g. SIMD instructions), vectorized/pipelined operations - there are storage engine optimizations, memory optimizations, etc. etc. It truly is a complex and fascinating problem.

Source: I was a technical product manager for a federated query engine.

[1] https://en.wikipedia.org/wiki/Query_optimization

[2] https://en.wikipedia.org/wiki/Star_schema

[3] https://trino.io/

I tried taking this for a while, but found that I started having really bad headaches. I was taking a well known brand from Amazon. I couldn't be 100% sure the headaches were due to the Lion's mane, but felt better after I stopped taking it.

I was pretty bummed, I was hoping for a little boost from this as I had read good things about it, including the NIH articles you posted.

I can also confirm, finding the supplements that are not just mostly the ground brown rice flour is also a little difficult.

You're absolutely right that what I'm saying is like data mesh.

The data mesh ideas aren't necessarily new - as this idea that data has been managed/owned by what some would call "federated teams" (optimistic view) and others might call "silos" (pessimistic view) isn't really new - especially in certain industries.

Certainly, the current energy and focus on building collaborative organizational service models, process, standards, etc. and standardizing this around the concept of data mesh is "fresh" and very much on the tip of many people's minds.

I worked at a company that had built a tool which was a data federation engine, and we dabbled with the concepts of "data fabric", "mesh", "network" etc. but we were more talking about tech/implementation vs. architecture and organizational structure. Still, we thought the right tech could enable this architecture/organizational structure and that it wasn't something totally figured out.

Thanks for sharing and mentioning it, and also Terminus. Certainly, I agree that knowledge graph is another approach and one that can also fit very well with the idea of linking data that is both common and specific to domains.

However, I've rarely seen graph databases that had the kind of versioning/time travel that a temporal database (such as relational ones) had built in natively (or anything like "diff").

I don't agree this can't map to SQL tables.. tables and relations (tuples) are just a form of data. People do collaborative data management using relational-based master data management systems already today and have been for many, many years.

That is not to say that graphs aren't a good match for storing common/reference datasets or linking data together. Given that graphs can be considered a superset of relational model, I don't see any reason that you can't do the same thing in graph databases, just that doing diffs and enabling the same kind of workflow might be more difficult in a graph data model - but definitely not impossible.

Lest we forget things like "Linked Data" [1], triple stores, etc. also that have been trying to get at linked data across the entire Internet. However, I never saw collaborative data management as part of that vision, it was mostly for querying.

[1] https://en.wikipedia.org/wiki/Linked_data

It is so true. I've done it, not out of ego, but out of passion and belief that it was the right thing, and I've paid the price physically and mentally for it.

I respect and honor this man for what he did. RIP Allan McDonald, and thank you for what you left behind.

This quote from the article helped me, right here, right now:

--- excerpt ---

"What we should remember about Al McDonald [is] he would often stress his laws of the seven R's," Maire says. "It was always, always do the right thing, for the right reason at the right time with the right people. [And] you will have no regrets for the rest of your life."

--- end excerpt ---

When people say that it is not just about about what you do, or why you do it, but who you do it with, it is so easy to gloss over that sometimes, it's about doing your part with the right people.

For me, this served as a reminder of high value.

In fact, in the social sciences, this is a known phenomenon called "normal accidents" [1].

[1] https://en.wikipedia.org/wiki/Normal_Accidents

See my post earlier in this thread [1].

Yes you need reference data for machine learning, but the world isn't only about machine learning. You might want reference data for human-interpreted analytics, table-driven logic (business rule engines, for example), etc.

[1] https://news.ycombinator.com/item?id=26371748.

I am not associated to Dolt, but I really like the idea of Dolt personally. I do see use cases, but not without challenges.

One of the main use cases you can see them targeting, and that I think makes a ton of sense, is providing tools for collecting, maintaining and publishing reference data sets using crowd sourcing.

For example, they are doing this with hospital charge codes (a.k.a. chargemaster data). Hospitals in the US are required to publish this data for transparency.. however, I have never seen a single aggregated national (or international) data set of all these charges. In fact, such a data set could be worth a lot of money to a lot of organizations for so many reasons. I used to work in health insurance, gathering data from all kinds of sources (government rules/regs, etc.) and it was a lot of hard work, scraping, structuring, maintaining, etc.

This reference data can be used for analytics, to power table-driven business logic, machine learning - to help identify cost inequalities, efficiencies, maybe even illicit price gouging, etc. There are so many reference data sets that have similar characteristics... and "data marketplaces" in a way are targeted at making "private" reference data sets available for sale - so then where is the "open" data marketplace? Well, here you go.. Dolt.

I have often realized that the more ways we can make things collaborative, the better off we will be.

Data is one of those things where, coming up with common, public reference datasets is difficult and there are lots of different perspectives ("branches"), sometimes your data set is missing something and it would be cool if someone could propose it ("pull request"), sometimes you want to compare the old and new version of a dataset ("diff") to see what is different.

One difficult thing about Dolt is, it will only be successful if people are actually willing to work together to cook up and maintain common data sets collaboratively, or if those doing so have an incentive to manage an "open data" project on Dolt as benevolent maintainers, for example. But, I could say then it has the same challenges as "open source" in general, so therefore it is not really that different.

Dolt could even be used as a foundation for a master data management registry - in the sense of you could pop it in as a "communal data fountain" if you will where anybody in your org, or on the Web, etc. could contribute - and you can have benevolent maintainers look over the general quality of the data. Dolt would be missing the data quality/fuzzy matching aspect that master data tools offer, but this is a start for sure.

For example, I work in a giant corporation right now. Each department prefers to maintain its own domain data and in some cases duplicates common data in that domain. Imagine using Dolt to make it possible for all these different domains to collaborate on a single copy of common data in a "Dolt" data set - now people can share data on a single copy and use pull requests, etc. to have an orderly debate on what that common data schema and data set should look like.

I think it's an idea that is very timely.

P.S. Dolt maintainers, if you read this and want to talk, I'm game! Awesome work :)