nerdjon

> it appears it answers my question

"Appears" to me is the key word here, how do you know that what it said was true?

It will very confidently "answer" your question in a way that is convincing but if you actually try to click through you may find that the answer is straight up wrong.

I feel like most of the time that I am looking up something that is not niche but also not a major thing it does this and at best useless at worst actually harmful since people won't click through for the real information.

The biggest problem being that these systems can be right enough times that you gradually start trusting it and stop checking. Which is what google wants, if you have to check its work in the first place why does it even exist.

Edit:

Yesterday I tried using one of these research tools because I was curious and it was low priority so figured why not. I was looking for an open source solution to a problem. I specifically mentioned one that I had seen but I was cautious because it seemed to have been abandoned so was looking for others. It confidently told me how wrong I was about thinking it was abandoned despite the last commit being in early 2024 (it even said 2024 in the report) and even before that was clearly slowing down.

Now thankfully in that case it actually told me in the report the 2024 part which clearly told me how bad the report was going to be, but that is clearly bad and tainted the entire research.

I am very curious realistically how can they reliably fix this.

So my understanding is that this is that the database/index that copilot used already crawled this file so of course it would not need to access the file to be able to tell the information in it.

But then, how do you fix that? Do you then tie audit reports to accessing parts of the database directly? Or are we instructing the LLM to do something like...

"If you are accessing knowledge pinky promise you are going to report it so we can add an audit log"

This really needs some communication from Microsoft on exactly what happened here and how it is being addressed since as of right now this should raise alarm bells for any company using Copilot and people have access to sensitive data that needs to be strictly monitored.

I am normally pretty quick to jump on how bad for privacy shoving some random LLM tool into a product is and the serious security risks especially in the terminal...

But looking at the marketing for Warp, this thing screams LLM everything. Nothing about this hints that things are processed locally. I can't imagine using a tool like this and not thinking that everything I type into it (and give it access too) is getting routed to a server somewhere.

What am I missing here about being upset that... it seems to be doing its job?

Unless I am missing that it is installing something so this happens in your normal terminal or something like that... to be blunt if you used this tool and this is what breaks your trust... how did you think it worked in the first place?

It is annoying, I would much rather pay for the content I am consuming since I want to support content being made.

But with prices going up and there now being so many services, I find it hard to justify more than a couple.

I pay for Apple TV+ and Disney+ with zero hesitation since I get a large amount of content from both of those that I actually enjoy. I added Hulu to Disney plus because why not.

Outside of that I just can't bring myself to subscribe to Netflix, whatever HBO is calling themselves now, paramount, etc etc anymore. It added up too quickly and there are alternative solutions.

It isn't even necessarily about the cost anymore; I have spent way too much on the hardware for my media server and I am nowhere near any concept of breaking even. But I no longer need to think about where something may be accessible just to be disappointed that I can't stream it.

Is there a name that these upscaled releases fall under to easily find them?

Star Trek in particular (I was watching voyager yesterday) the quality is always pretty depressing when shown on a larger TV. Been recently thinking about trying to find the best quality I can find but it is always a lot of trial and error. But if there is a common name and tag I could look for that would be great.

How we keep getting articles like this, that LLM's will flat out lie, and yet we keep pushing them and the general public keeps eating it up... is beyond me.

They even "lie" about their actions. My absolute favorite that I still see happen, is you ask one of these models to write a script. Something is wrong, so it says something along the lines of "let me just check the documentation real quick" proceeded by the next words a second later being something like "now I got it"... since you know... it didn't actually check anything but of course the predictive engine wants to "say" that.

I completely respect a personal choice of doing that, I mean I don't particularly want to engage with many of them either. Especially not when I can expect that I am going to likely be called a particular F word (I am a gay man).

My biggest issue is not the lack of contact, it is the demonizing. Using blanket terms like "if you voted for trump your racist, homophobic, sexist, etc" when I just simply don't think that is a valid blanket statement and is really just a "feel good" statement for us to justify not hearing why they might have done something.

I do think that we are actively pushing them to be more extreme with blanket statements like this and it isn't not actually helping.

We can keep calling them names all we want but the fact is they are still voters that are enabling what we all have to deal with. Either we acknowledge that or we just keep repeating the same pattern we have been repeating since at least the 70's. A little bit of progress followed by a regression.

While I can understand the side that you are coming from. One of the biggest failures I have seen from my friends is demonizing anyone that may have voted for tump and these people, and refusing to have a conversation. Immediately labeling them as racist for example (which I don't think is necessarily untrue for many of them, but when we know there are black people that voted for Trump that argument as a blanket statement gets harder to make).

I strongly believe that for many people just doing this is causing them to dig into their heels and instead of examining themselves they are pushed to being on the defensive trying to say they are not racist, homophobic, sexist, whatever. Which is not getting us anywhere and is just making both sides angrier.

There are the extremes, people that have the power that are pushing things like this. But then there are the manipulated. Those that are being told lies and being encouraged to vote a certain why because they simply are only seeing part of the picture. Maybe they don't have exposure to the world. Whatever.

While I do respect someone's right to protect their own mental health and not want to engage in a conversation with many of these people, these conversations do need to happen. I truly believe that the majority of people are nowhere near as vile as those in power right now are. So we need to understand why they are enabling them.

That being said...

It is a very fine line. Too much empathy can lead to them thinking that this is ok, there does need to be some force in a push back against what is happening right now. Pushing back on the misinformation that is causing many people to hold these views.

So yes we can try to understand where these views are coming from without giving them weight as being valid.

I feel like I have done a somewhat decent job of trying to keep some of that childlike wonder in my own life, even just not abandoning things that I love like video games, watching anime, the occasional cartoon.

I do wonder in practice how this really works for making friends, once you leave camp there is now a distance limitation.

But I have long wondered about the idea of recreating some of the experiences from childhood as adults. Things like camp, maybe going to a "day care", or other things to just not think about all of the stress of life for a little bit.

Sure we have things like going to Disney, amusement parks, adult oriented arcades like Dave and Busters. But as fun as those are they are not really replicating the childhood experience like this seems to do and other things that could be made similar.

Well... it is nice to get some good news on this front but I can't shake that this is likely short lived given the federal government right now...

There is less and less any reason for them to try to hide their true intentions and can just be more open with their blatant racism, sexism, homophobia, etc etc.

Side note: was quite surprised to see a reference to Cloud Atlas. While not surprised given the entire point of that book, it makes me wonder how much these people are actually reading these books and what that looks like.

I do wish that this was something talked about more, I know there have been studies supposedly showing there is not a correlation but that doesn't make any sense to me. I know there have been several times that I have been stoned enough that I would not want to drive.

Doesn't mean it should not be legal or anything, but I also don't think we should disregard it.

I mean... I am not drinking alcohol because I am thirsty...

Often I have both in front of me. It isnt a one or the other situation.

Sure, but if you are a non technical user what are you likely going to search for first.

A lot of non technical people know "winrar" and even if they don't if you search "rar file" on Google the first result is winrar.

I don't doubt the numbers.

Until very recently Windows could not natively unarchive .rar files and you needed to download WinRAR to be able to do this. I still find it not terribly uncommon to run into a random .rar file that previously would have meant I needed to install it, even if I only used it once.

> and the fact that nearly nobody downloads .zip files anymore

Citation needed? Why would people not be downloading .zip files anymore?

This is why research like this is important and needs to keep being published.

What we have seen the last few years is a conscious marketing effort to rebrand everything ML as AI and to use terms like "Reasoning", "Extended Thinking" and others that for many non technical people give the impression that it is doing far more than it is actually doing.

Many of us here can see his research and be like... well yeah we already knew this. But there is a very well funded effort to oversell what these systems can actually do and that is reaching the people that ultimately make the decisions at companies.

So the question is no longer will AI Agents be able to do most white collar work. They can probably fake it well enough to accomplish a few tasks and management will see that. But will the output actually be valuable long term vs short term gains.

I have now seen multiple articles about this and none of them talked about how much use it was actually getting today, which I have found disappointing.

Not sure why none that I have seen have been any better.

It is quite concerning because SaaS isn't going away, (putting aside the questionable ethical side of an application like this) an application like this just is not possible outside of SaaS. It needs a server to centrally store information like this. If as a society we decide that something like this is valuable, there just is not another way to do it.

Sure before there would be leaks that would break trust, but generally it seems most of the time the basics were taken care of and often those leaks were because of phishing employees or other means of getting the information vs really basic security issues. Not a hard rule obviously, but still.

Now we have seen time and time again that these vibe coded systems lack even the most basic security fundamentals. That will continue to erode trust.

I am a bit confused here as well, is there a difference between "Push" and the notifications on iOS?

I just checked my emails, I use Amazon Workmail and even that has push support. The only one for me that doesn't is my lesser used gmail.

To my knowledge I get alerts quickly with that email, I have never had an issue of not.

> Asking AI to tell reality from fiction is a bit much when the humans it gets its info from can’t, but this is at least not ridiculous.

I agree with that, but the problem is that it is being positioned as a reliable source of information. And is being treated as such. Google's disclaimer "AI responses may include mistakes. Learn more" only shows up if you click the button to show more of the response, is smaller text, a light gray, and clearly overshadowed by the button with lights rotating around it to do a deep dive.

The problem is just how easy it is to "lead on" one of these models. By simply stating a search like "why is rum healthy" implies that I already think it is healthy so of course it leads into that but that is why this is so broken. But "is rum healthy" actually provides a more factual answer:

> Rum is an alcoholic beverage that does not have any significant health benefits. While some studies have suggested potential benefits, such as improved blood circulation and reduced risk of heart disease, these findings are often based on limited evidence and have not been widely accepted by the medical community.

So I was curious, in normal google search (with the AI summary) I put in "why is rum healthy" I got this, and then it listed a bunch of benefits: "Rum, when consumed in moderation, may offer a few potential health benefits. These include possible positive effects on heart health due to its potential to increase HDL (good) cholesterol, and the presence of antioxidants in darker rums, which may be beneficial."

But if I just simply remove the "why" it clearly states "Rum is an alcoholic beverage that does not have any significant health benefits."

Man I love so much that we are pushing this technology that is clearly just "garbage in, garbage out".

Side Note: totally now going to tell my doctor I have been drinking more rum next time we do blood work if my good cholesterol is still a bit low. I am sure he is going to be thrilled. I wonder if I could buy rum with my HSA if I had a screenshot of this response... (\s if really necessary)

Under normal circumstances I could more easily give the benefit of the doubt on this, but the reality is we are not living in any amount of normalcy as far as this administration is concerned.

Of course removing this does not change the law, but this is still an official place for people to view these documents.

Could it have just been someone making an honest mistake? Yeah of course.

Has this administration already been deleting other important information from government websites and this could have been on purpose? Also yes.

Yeah I really don't believe that this is really the case, especially when we had a report recently saying clicks are down.

It has become shockingly common to see people sharing a screenshot of an AI response as evidence to back up their argument. I was once busy with something so I asked my partner if he could look up something for me, he confidently shared a screenshot of an AI response from Google. It of course was completely wrong and I had to do my own searching anyways (annoyingly needing to scroll past and ignore the AI response that kept trying to tell me the same wrong information).

We have to remember that google is incentivized to keep people on Google more. Their ability to just summarize stuff instead of getting people off of google as quickly as possible is a gold mine for them, of course they are going to push it as hard as possible.

Agreed, and the fact that there is not an easy "x" to close it is even worse.

If you want to do something cute and fun, whatever its your site. But if you actually want people to use your site make it easy to dismiss. We already have annoying ads and this is honestly worse than many ads.

Also, from the bio that I can barely see he writes about "UI Design" and... included this?

That is basically several paragraphs to just say "well you should just adapt to the new world instead of pushing against bad practices". There is barely any "why" actually said here.

We just had the article about how AI search is leading to less clicks, so where is that supposed "pipeline"?

Also completely ignores how you may not want your information to be misconstrued (lied basically) to the user with a helpful link telling them where the source is, but they may never click through. And worse if they know that the information being told to them is wrong, they may then think it was because your site was wrong and trust you less, all without ever clicking that link.

Slightly bothered that it seems like the article is conflating an app/website choosing to implement some form of verification vs a law requiring it.

I think that if a website or app wants to make this choice, they should be allowed too. Obviously we should expect that they have proper security, and we should make the choice on if we want to take that risk. But I think it is a perfectly valid choice by the developer, and users can choose whether or not they think it is worth it or use a competitor that doesn't (or a competitor is created that doesn't).

But the issue of laws requiring it I think is where things have gone too far. So much of this is being framed as "protect the children" but most of it really seems to be fueled from a puritanical "porn is bad" and needing to make it harder to get access too for adults. I really wish we could move past this as a society, stop vilifying it, being ashamed of sex, etc.

And likely throw in some tracking of what people are doing online since now you no longer have the anonymity.

Edit:

If you really truly are trying to "protect the children"... Maybe educate them instead of hiding things from them.

If we are purely talking about textbooks, I can see the value in a tool like this... assuming we still have yet to actually solve the problem of AI being able to tell the truth and could just lead to more issues if we can't even read a textbook without it and then learn the wrong thing.

Not to necessarily diss the work that was done on this, but the idea of actually wanting this for reading feels like it is a continuation of the lack of attention span that has seemed to get worse and worse. We already saw this with the oversimplification of television shows and movies. Many of them leaning more towards slapping you in the face with something instead of subtly.

I know way too many people that struggle to sit still for a half hour episode of some show now (like my partner, frustratingly) and have to be doing something else.

If you are struggling with absorbing the information you are reading that is likely a sign you should put down the book and come back to it later, obviously your mind wants to be doing something else. If it is a continued issue than practice reading something that you know you would like. Personally my "in" for my love of reading was reading video game books that expanded the lore and it grew from there, but I was already invested in the story so the book was easier to read.

Using this for a book feels more like a crutch than anything else. That is obviously before you get into whether or not the LLM is actually going to tell you the truth.

There is however one possible use case I could get into, but this is something that could be solved by just finding a video or something online. A refresher when it has been a long time between books coming out in a series.

While I do think the "without being asked" spin on this is likely wrong, I assume the "spicy" option is more than just saying "don't do X" and the system prompt was likely modified to encourage its creation even if the user did not specify it so choosing "Spicy" is in fact asking.

That does not change the troubling nature of creating fakes like this which to me is the real issue not anything about "Without being asked" since that implies it would have been fine if the user has specifically asked for this... which it still isn't.

No one should have images of them being made and circulated like this that they did not consent too.

It drives me absolutely mad when I see (largely liberal people) complain about how both parties are the same and just as bad.

And its like... how do you really believe that? Like yeah both parties have the same corruption but welcome to politics.

So at this point I am convinced it is willful ignorance on both sides (or ulterior motives when I see certain left leaning people STILL bring up Biden or Harris in relation to trump as if either of them matter anymore in the slightest given our current situation, at this point I don't care what Biden did or did not do). Seeing something that goes against their views of "this side is bad" and just trying to talk it away as some "abuse of government power" or something to justify why it should not have been a thing in the first place while ignoring its real benefits.

Same for me, buying my dream machine pro (and AP's) was one of my few tech purchases that I have zero regrets buying. It is still running strong after a few years and see no reason to change it anytime soon.

Have they been perfect? No, but this has allowed me to control my network how I actually want to control it.

This has lead me to now having multiple Ubiquiti components (with more planned), my most recent was switching away from Synology to the UNAS Pro and it has been great.

Really the only thing I ever bought from them that I really regretted was the tooless mini rack. Was really cool but I have non ubiquiti things that I need to mount and I doubt they are going to actually make a server I can run k8s anytime soon.

If whatever this SaaS can be so easily built with no technical knowledge... why would a company not just build it themselves with "vibe coding" and cut out our friend completely?

That is what is so broken about this whole vibe coding to riches thing. Ignoring the very real problems with the quality of the code and security issues, if you can vibe code it what is stopping someone else from doing the exact same thing.

It sure isn't like security is any better... clearly.