HNHacker News
TopNewBestAskShowJobs

upwardbound

3,388 karma · joined October 24, 2010

Researcher at Preamble, Inc., the first team ever to discover Prompt Injection. We provided a responsible disclosure to OpenAI on May 3, 2022. Detailed writeup here:

https://www.preamble.com/prompt-injection-a-critical-vulnerability-in-the-gpt-3-transformer-and-how-we-can-begin-to-solve-it

submissionscomments
upwardbound··on Prof. Ray Buettner, US Navy: Hardware Cutout Switches Are Needed in Weapons
Transcript:

   We need the hardware cutout so that you can't just control everything by software.  There has to be an independent system that will render the software side of the system inoperative or to a controlled minimum mode.  We need that cutout switch, and I think that's doable.  But good engineering and good testing will be required to see that happen.  If we're going to make them weapon systems, we need to invest the extra cost to make sure we can control - We can always cause a graceful or controlled failure when we need to.  And I think that just has to be part of good design.
- Prof. Ray Buettner, Naval Postgraduate School, 2020

Without this architecture, next-gen software-enabled weapons systems will be able to be hacked en masse, resulting in a hostile takeover of thousands or tens of thousands of US networked weapons simultaneously.

If you work at a defense contractor, please take Prof. Buettner's advice extremely seriously during the system architecture phase of every project. I propose that to fail to do so should be considered criminal negligence, as it would be intentional disregard of industry best practices ---- the best practices stated here by Prof. Buettner.

His bibliography: https://scholar.google.com/citations?user=3gSfhaIAAAAJ&hl=en

The Naval Postgraduate School website: https://nps.edu/

upwardbound··on What happens in a mind that can't 'see' mental images
I have higher than average ability to visualize mental images. It's not like a screen but more like wearing augmented reality glasses with very low brightness/opacity to the point where the images have only 1% solidity. Also, the detail level of the images is low, similar to an impressionist painting. However, properties of color, size, and 3D spatial location and orientation are all well-defined, so for example I can imagine a (very low opacity, very rough, impressionist style) picture of Mario or Luigi standing upon this line of text on my screen and being 1 inch tall. It's my understanding that this level of capability is higher than average, but less than talented artists like painters or sculptors. Despite not being good enough for a career in art, this mildly better than average ability level, combined with being able to code, allowed me to be quite successful as an augmented reality prototyper in the first half of my career.

It looks roughly like the detail level of picture "C" in this picture: https://history.siggraph.org/wp-content/uploads/2023/02/2014... But with much less brightness/opacity/solidity than that, maybe 20x less (i.e., only 5% of the brightness/opacity/solidity of the tie fighter image - but same level of lack of detail)

You can improve this ability through practice. If you spend 10 minutes every day concentrating on visualizing that triangle with more detail & specificity, like an art student would, you'll gradually improve.

upwardbound··on `find` + `mkdir` is Turing complete
The initial state of a Turing Machine tape is that beyond the finite-length input, all of the rest of the infinite tape is blank.

https://en.wikipedia.org/wiki/Turing_machine#:~:text=Cells%2...

Whereas the Rule 110 Cyclic Tags engine requires the infinite tape to contain infinite repetitions of structured patterns, even in order to simply run "while(true)". That's a key difference.

I also agree with zamadatix's sibling comment.

upwardbound··on `find` + `mkdir` is Turing complete
I don't know this field very well so I might be misunderstanding, but I think this is different than "infinite tape" in Turing Machines. As I understand it, the proof of universality for Rule 110 required that the program code which is called the "production rules" be repeated infinitely on the tape even for a finite size program. https://en.wikipedia.org/wiki/Rule_110#:~:text=An%20infinite...

If you had a halting problem oracle to tell you how much runtime is needed to run a certain program to completion, you could get away with having only a finite number of repetitions of the "production rules", and simply pretending that they're infinitely repeated. This would only work for programs that halt.

If I understand correctly, any program that loops forever, if implemented within Rule 110 Cyclic Tags, requires infinite repetition of the production rules. I think this is a difference of Rule 110 vs Turing Machine tape. If I understand correctly, a Turing Machine with finite, even quite small, tape can loop forever. But a Rule 110 program must have infinitely sized tape to be able to loop forever.

Basically (if I understand correctly), Rule 110 Cyclic Tags essentially "consume" tape symbols as basically a non-renewable resource, like an electrical computer server powered by the burning of coal. Infinite runtime (looping forever) requires infinite repetition of the tape symbols (both the "production rules" and the "clock pulses" - see the Wiki page above). I believe this is unlike Turing Machines, which can loop forever without "consuming" any non-renewable resource.

To clearly state this again: Running a simple "while(true)" loop in a Turing Machine only needs finite tape, but requires infinite tape in Rule 110.

upwardbound··on Metaphysical experiments probe our hidden assumptions about reality
Some of my favorite fun questions or speculations in the applied metaphysics realm, which I sincerely hope we or our descendants can someday test, are about whether one of the deepest axioms of modern astrophysics, the Cosmological Principle, is false.

https://en.wikipedia.org/wiki/Cosmological_principle

It says basically that the laws of physics are the same everywhere. If it's false, this would allow moonshot bets to escape this universe's limiting ending, its heat death - because we might be able to do things that are impossible here, but are possible in a different region of space, with different laws.

Reposting some of my previous comments about this:

For example, if we eventually have multiple star systems' worth of resources at our disposal, then I hope we put at least a trillion dollars per topic into long-shot research topics such as time travel, the multiverse, FTL travel, methods of creating new universes via an artificial Big Bang, applied theology (trying to find evidence of a God so that we can communicate with them), methods of achieving infinitely positive states of qualia such as something like nirvāṇa, and so on.

Each bet might be 1 in a million - but I'd much rather that we try every possible chance at escaping what we currently see as the inevitable, and not go gentle into that heat death night.

This goal, to break through the wall at the end of the universe, was what Asimov called for in his inspiring short story The Last Question.

https://users.ece.cmu.edu/~gamvrosi/thelastq.html

Keep in mind that anything is possible if the Cosmological Principle turns out to be false, and if we can travel far enough to escape our local zone.

For example, there might turn out to be particle types which don't exist at all in our zone, but which are force carriers for forces that enable time travel or other violations of causality. We might have evolved in a zone with zero of these particles simply because it's hard for life to evolve in a zone without causality. Our zone might be a life-friendly little bubble beyond which things get truly strange, but where with appropriately designed nanotechnology, we might achieve incredible things we've never before observed, including changing the past or even stepping entirely outside of the fabric of spacetime into someplace with more or fewer spatial dimensions.

https://en.wikipedia.org/wiki/Cosmological_principle

So, 1 in a million is my approximate a priori estimate of the odds of the Cosmological Principle being false in a useful way.

For a fun example of fiction that takes place in a setting where the Cosmological Principle is false, check out A Fire Upon the Deep by Vernor Vinge. The zone where Earth is situated, and where the laws of physics are the same as in our observed reality, is called the Slow Zone. There are three other types of zones with other laws, called the Beyond, the Transcend, and the Unthinking Depths.

I suppose that to approach a topic like this, we can reason in the classical-logic manner of the ancient Greeks and others who sought to understand their universe without yet having the instruments to make necessary observations. What I mean is, without our civilization having seen evidence either way yet, the question of what is a good prior probability is only able to be informed through non-empirical means, such as the philosophy of science (e.g. https://en.wikipedia.org/wiki/Sunrise_problem ), philosophy of knowledge (e.g. https://en.wikipedia.org/wiki/Anthropic_principle ), and theology (e.g. https://en.wikipedia.org/wiki/Best_of_all_possible_worlds ).

Originally I posted these here: https://news.ycombinator.com/item?id=38217412

upwardbound··on Secure Boot is broken on 200 models from 5 big device makers
Do those charge pumps use external capacitors? If so, you could disconnect/float those capacitors, or if that would damage or glitch the chip, you could replace the real capacitors with some circuit like a voltage regulator + diode that would be designed to provide the charge pump output rail with a voltage that's high enough not to glitch the chip but low enough to be unsuccessful in writing to the flash. Would one of those ideas work, and allow the retrofit you envision to be designed with existing flash memory silicon + a few additional components?
upwardbound··on Give Me the Green Light Part 1: Hacking Traffic Control Systems
We do have that now, as of 2022! The new Justice Department policy now instructs prosecutors not to prosecute security researchers who acted in good faith for the public benefit and who avoided any harm to individuals or the public.

https://www.justice.gov/opa/pr/department-justice-announces-...

upwardbound··on CrowdStrike Update: Windows Bluescreen and Boot Loops
Or the opposite: the enemy (or a third-party enemy who wasn't previously a combatant in the battle) hijacks your entire naval USV/UUV fleet & air force drone fleet using an advanced cyberattack, and suddenly your enemy's military force has almost doubled while yours is down to almost zero, and these hijacked machines are within your own lines.
upwardbound··on Google Chrome has an API accesible only from *.google.com
> GMeet can be counted as a part of the browser itself

That's the part that is arguably an antitrust violation.

See e.g. https://www.zdnet.com/article/microsofts-browser-bundling-ba...

upwardbound··on Google Chrome has an API accesible only from *.google.com
Other extensions, not other websites. This functionality is a feature of the Google Meet website that other video conferencing websites cannot offer.
upwardbound··on OpenAI was hacked year-old breach wasn't reported to the public
Would you be open to collaborating to investigate whether we can further widen this bugged behavior, demonstrate that it's serious, and get it fixed? You could reach me at strangecompanyventure@gmail.com

It sounds like the bug affecting your account is uncommon, making your account special, and as an AI security researcher I can help investigate the extent of the issue and I have contacts that can help call attention to it. Thank you for discovering this and trying to escalate this.

upwardbound··on OpenAI was hacked year-old breach wasn't reported to the public
The email address they have listed there is defunct, and they haven't bothered to update this security.txt page. When you try emailing disclosure@openai.com, you get an auto-reply saying:

    Hello and thank you for reaching out to OpenAI. Our vulnerability disclosure program has migrated to OpenAI's bug bounty program, and this mailbox is no longer monitored. Please use the "submit report" functionality available through our bug bounty platform to inform us of security concerns, or reach out to support@openai.com for any non-security-related inquiries.

    Thank you for your help in securing OpenAI!

    Bug Bounty Program: https://bugcrowd.com/openai
upwardbound··on OpenAI was hacked year-old breach wasn't reported to the public
OpenAI seems to have, unfortunately, outsourced the triaging of bug bounty reports to people who don't seem to understand security well enough to recognize issues. As an example, I've been trying to get OpenAI to fix the fact that "eval()" is used incorrectly in one of their Cookbooks in a place where the correct function would be "json.loads()".

https://cookbook.openai.com/examples/how_to_call_functions_w...

https://news.ycombinator.com/item?id=40474451#40474452

The bug bounty report was closed with a message saying:

    Upon reviewing your report and consulting with the OpenAI team, we have determined that this feature is operating as intended. This means it does not constitute a valid sandbox escape. The Code Interpreter environment is securely sandboxed to support code writing and execution, including shell operations. Any code execution within this environment falls outside the scope of our program ... As you have not demonstrated a valid sandbox escape or RCE, we're closing this submission as Not Applicable.
This shows a fundamental misunderstanding of basic coding, as the eval() I pointed them to is completely unrelated to the Code Interpreter environment. So, the report is incorrectly considered "Not Applicable", without any real further ways to try to get them to fix it. I tried contacting the Cookbook authors directly, but never heard back.
upwardbound··on Debugging hardware is hard
Is it possible that the autocal is in the process of shifting the phase the clock it controls to match the reference clock, and that the user is supposed to wait until that's done before running phase-sensitive operations? I'm unfamiliar with the chip in question, just making guesses or shots in the dark.
upwardbound··on CriticGPT: Finding GPT-4's mistakes with GPT-4
You make a good point so I should clarify that Iterated Amplification is an idea that was first proposed as a technique for AGI safety, but happens to also be applicable to LLM safety. I study AGI safety which is why I recognized the technique.
upwardbound··on CriticGPT: Finding GPT-4's mistakes with GPT-4
Thank you for providing the reference numbers, this is helpful! I'll update my GP comment.
upwardbound··on CriticGPT: Finding GPT-4's mistakes with GPT-4
For those new to the field of AGI safety: this is an implementation of Paul Christiano's alignment procedure proposal called Iterated Amplification from 6 years ago. https://www.alignmentforum.org/s/EmDuGeRw749sD3GKd

According to his website he previously ran the language model alignment team at OpenAI. https://paulfchristiano.com/

It's wonderful to see his idea coming to fruition! I'm honestly a bit skeptical of the idea myself (it's like proposing to stabilize the stack of "turtles all the way down" by adding more turtles - as is insightfully pointed out in this other comment https://news.ycombinator.com/item?id=40817017) but every innovative idea is worth a try, in a field as time-critical and urgent as AGI safety.

For a good summary of technical approaches to AGI safety, start with the Future of Life Institute AI Alignment Podcast, especially these two episodes which serve as an overview of the field:

- https://futureoflife.org/podcast/an-overview-of-technical-ai...

- https://futureoflife.org/podcast/an-overview-of-technical-ai...

In both of those episodes, Cristiano's publication series on Iterated Amplification is link #3 in the list of recommended reading.

upwardbound··on NASA indefinitely delays return of Starliner to review propulsion data
> We could have a society where the average person works ten hours a week ... and owns a home.

I don't think 10 hours a week on average is enough hours to grow the food to feed everyone, or keep any other part of the economy running. There's probably truth in some of the other parts of what you're saying but this, no way. Maybe 30 hours might be believable.

upwardbound··on MLow: Meta's low bitrate audio codec
https://www.amnesty.org/en/documents/ASA16/5933/2022/en/
upwardbound··on 'Operation Endgame' Hits Malware Delivery Platforms
If folks are interested in this kind of story (law enforcement vs cybercriminals), one of the best examples is the book Crack99:

https://icdt.osu.edu/crack99-takedown-100-million-chinese-so...

https://www.amazon.com/CRACK99-Takedown-Million-Chinese-Soft...

upwardbound··on How Waymo outlasted the competition and made robo-taxis a real business
It doesn't make the tech look bad, but to me it makes the safety driver & the other executive look callous and uncaring.

> They didn’t go back to check on the other driver or to see if anyone else had been hurt

They should have made sure the driver was okay.

upwardbound··on Unsafe OpenAI example code includes eval() of untrusted text generated by GPT-4o
Never eval() AI-generated text without a sandbox. This should one of the most obvious principles of AGI safety, especially to a company working towards building superintelligence.

Yet, inexplicably, the OpenAI tutorial linked above includes this un-sandboxed command, with no explanation for this egregious lack of any safety precautions: `tool_query_string = eval(tool_calls[0].function.arguments)['query']`

Though the content of function.arguments is usually reasonable in typical scenarios, it is unvalidated, arbitrary text. There are multiple documented instances of it containing text that causes parsing errors:

https://community.openai.com/t/malformed-function-calling-ar...

https://community.openai.com/t/malformed-json-in-gpt4-1106-f...

https://community.openai.com/t/ai-assistant-malformed-functi...

Anyone running the code in the OpenAI cookbook, containing eval(), could have their machine compromised by a model that gets hijacked via prompt injection or whose behavior goes astray for other reasons.

The fact that OpenAI recommends this code to developers speaks to a serious lack of care about AGI safety.

upwardbound··on What's the difference between a motor and an engine? (2013)
> An engine produces power. Locomotion can be derived from power but isn't required to.

By this definition, is a fuel cell an engine?

https://www.energy.gov/eere/fuelcells/fuel-cells#:~:text=A%2....

upwardbound··on DB-19: Resurrecting an Obsolete Connector (2016)
Yes you definitely can. Obviously you should buy the pins instead of CNCing pins, but for the complex geometry of the main connector body you can definitely CNC it. Example of the pins being sold on Amazon: https://www.amazon.com/JRready-Terminal-Connector-205089-1-c...
upwardbound··on Player-Driven Emergence in LLM-Driven Game Narrative
Agree. This point is a big concern and is explored in the stories (the protagonist creates a "mad scientist" character that can use the LLM to invent new equipment, and then uses a lot of trial and error to find prompts that generate insanely overpowered equipment). It sounds fun to play though! Balance doesn't have to be .. well .. balanced if there are other ways to maintain fun, such as leaning heavily into the creativity & self-actualization angle. As an example, if I could have any superpower in real life, I'd love to be able to fly, even though that's actually a very weak power compared to things like teleporting that are mostly a superset of flying, but not as fun and whimsical.

Encouraging "living your dream" rather than being the most powerful might be easier to achieve in a game with coop or open-world objectives rather than one focused on PvP, but if "living your dream" could be compelling made into the primary player objective, it would pair very well with the custom powers mechanic envisioned in the story.

As an example of how this is explored in the story, one of the protagonist's friends who loves surfing has a character whose superpower is "the ability to surf anything". It's an extremely weak power but the character really loves it because they get to surf the most extreme things imaginable, just for the fun of it, including in outlandish scenarios such as surfing along the blast wave of an explosion, or even the beam of a laser - silly stuff that is fun to explore & try.

I think it wouldn't have to be VR to be fun - imagine if it was something like a Roblox-based game played on laptops. Even then, the "surf anything" power and other outlandish powers would be incredibly fun to use. Imagine if you could do almost anything! You don't have to be the most powerful player in the world to have fun. Maybe you just really want to pilot a mecha-Godzilla, or something - even though an Iron Man or Thor level super could smash it in seconds. It's still awesome!

upwardbound··on Player-Driven Emergence in LLM-Driven Game Narrative
Something related is compellingly explored in an interesting trilogy of LitRPG comedy books called the Vaudevillain series, consisting of Top Hat Express, Black Tie Villainy, and Cane Whirling Lunacy. In the series, there is a VR video game called “World of Supers” where players can type in any superpower they can dream of into a game-master LLM, and the LLM will then create a character for them which has that power, but reworked a little bit to fit into game balance. For example, the player could type “I can stop time and still move around and take actions for up to 90 seconds” and the game-master LLM may approve this power but alter 90 seconds to 10 seconds, and give the player a major weakness, such as “However, stopping time in the vicinity of a gamma ray source will result in immediate loss of 75% of HP due to the interaction of the time stop with the radiation’s effects. This HP loss can be mitigated if you are encased in a significant amount of lead, such as a lead vehicle or heavy lead armor.” It's a pretty neat concept for how LLMs could allow custom player abilities while maintaining game balance. That way, player agency extends to actual powers/abilities, not just dialogue.
upwardbound··on Avoid blundering: 80% of a winning strategy
> I computed a simple “error score” that includes mistakes while giving blunders more weight: [number of mistakes] × 2*[number of blunders]

This doesn't give blunders more weight..

Multiplication is commutative and associative, so the author's formula is also the same as this one: 2*[number of mistakes] × [number of blunders]

To give more weight to blunders, you could use an exponent, which is a common trick in baseball statistics (SABRmetrics), like this: [number of mistakes] × [number of blunders ^ 1.1]

See e.g. some of the formula concepts invented by Bill James https://en.wikipedia.org/wiki/Bill_James#Innovations

upwardbound··on NASA and Boeing Are (Finally) Putting Astronauts on Starliner
> we (my company) generally don't like to bid on T&M contracts and prefer fixed price, I like the challenge of finding efficiencies.

As a taxpayer, thank you!

upwardbound··on NASA and Boeing Are (Finally) Putting Astronauts on Starliner
Yes, from an economic standpoint that's the primary reason. Boeing primarily does spaceflight contracts on a "cost-plus" basis (i.e. their profit is a fixed %, such as 10%, of their their expenses), where they make more money if they take longer and use more expensive components. SpaceX primarily or maybe even entirely does their contracts on a "fixed price" basis, where they make more money by doing things faster and with cheaper components.

In a rationally-run corporation (of which real ones are merely an approximation), these profit incentives will trickle down into things like the culture of the team and the decision-making criteria used and decision processes used.

The cost-plus contract was popularized by the US military in WW2 because it allows a way for the military to sign a contract without the scope of work being known - bypassing most or all of the contract negotiation phase, and saving months or a year. The military can say "just make it happen, I don't care how, just do it, and I'll pay you your costs + 10%". https://quizlet.com/11795259/251-flash-cards/

However, in WW2, the cost-plus defense contractors still worked very fast, despite the lack of a profit incentive for fast work, because they were motivated by a survival incentive (if they didn't work fast enough, they'd lose the war and possibly be killed).

upwardbound··on Many AI safety orgs have tried to criminalize currently-existing open-source AI
Hahaha true... :)
← PreviousPage 2 of 16Next →