The code needed create, train, and perform inference on a Transformer is quite short. How short depends on how you count the `import` statements in https://github.com/openai/gpt-2/blob/master/src/model.py and https://github.com/openai/gpt-2/blob/master/src/sample.py etc.
Spreadsheets performing linear regression etc., — Do they learn? Sure!
If you accept that Transformers adapt and learn then you must accept that a spreadsheet also does, because someone implemented GPT-2 in Excel: https://github.com/ianand/spreadsheets-are-all-you-need
Do polymorphic computer viruses adapt? Border Gateway Protocol? Exponential backoff? Autocomplete? And that's aside from any algorithmic search results or "social" feeds, which are nothing but that.
What differentiates a “traditional computer program” from a single matmul? How about a bunch of matmuls? How about a bunch of matmuls with some non-linear arithmetic thrown in between them? And if we are adding in sampling from a distribution but set temperature to zero and use random seeds so that it is fully deterministic?
What I said is thst a traditional computer program (say Excel) is not intelligent in itself, as it cannot, itself, without further input from a human, adapt.
Excel is as capable of adapting to new input as any AI, and the input need not come from a human specifically: the act of adapting to input is an automatic consequence of being able to perform linear regression.
In fact, this is the real value of a spreadsheet over a piece of gridded paper where the sums are done by hand.
If "adapting to input" is your test for intelligence, however, it is a test which a thermostat will pass.
I said 'sensory input', as in, "any primary, real world data". Can I just download excel, and without feeding it machine learning algorithms, feed it a bunch of videos or text and have it learn history?
>If "adapting to input" is your test for intelligence, however, it is a test which a thermostat will pass.
Not just adapting to specific input the system is designed for, adapt it to ANY input. A thermostats function will never change on its own without modification, just like excels, or whatever other traditional program.
It has some built-in.
Linear regression is a machine learning technique.
Alternatively, neural networks are also just doing explicitly what they are designed to do… sure there is a larger computational graph with lots of operations, but it’s all deterministic… backprop is not really much different on a procedural level than the simple fitting algorithm that I outlined above, in as much as it is just a specific well-defined algorithm or sequence of steps designed to compute some parameters from data.
Adapting and learning means it chose to concentrate on packing the world into retrievable storage.
When do we not adapt and learn? When we ignore our inputs and do nothing with it (don’t store it, don’t retrieve it).
In the example you gave, those classical programs cannot concentrate, it’s a one and done.
In the example I mentioned, the program clearly is taking inputs (load), storing them, querying for previous values that satisfy certain conditions (the value at a certain timestamp for each of the last 7 days), running computation (computing a mean), and operationalizing the result (pre-emptively scaling) to achieve a goal (avoiding having insufficient capacity) that affects the world (whoever interacting with the system will have different experiences because of the actions taken by the system). That seems like it satisfies your criterion to me.
Not sure I follow your usage of discontinuous: clearly the load inputs are continuous variables (albeit discretely sampled in time and magnitude, but I don’t see the significance of either of those sampling frequencies, each of which can be chosen to be effectively arbitrarily small). Also don’t understand the relevance of continuity at all since you didn’t reference it in the post I replied to.
> A few null or unknown inputs that you didn’t even consider will break it.
Null handling should be assumed to be a trivial problem here (ie just use the last 7 valid values for the time step of interest), and I’m not sure what “unknown” values would be here, can you give an example? I think it’s safe to assume that the inputs have already been normalized to some meaningful scale as an implementation detail of the load sensors. Even if the telemetry scheme changes because the instrumentation changes or the infrastructure changes, so long as the instrumentation produces values at the same scale and the actuators can still respond to the outputs, the core kernel of sense/compute/actuate can still be left unchanged.
I’m assuming that you are citing the failure to respond to null or unknown conditions as a lack of adaptation capabilities, especially as you are referencing its inability to last 100 years - though good luck finding any piece of software that can last 100 years, especially more robust machine learning algorithms that have to deal with problems like data distribution drift, hardware changes, etc etc - same goes with wetware (besides turtles?). I’m strawmanning you a little bit on the appeal to time though, since I assume your main point was to just emphasize that “it won’t deal with changing conditions/new scenarios well” which I think is somewhat fair. On the other hand, I think it is safe to say that what I described does at least respond to the world and in turn affect it in a kind of cybernetic feedback loop, which includes adapting to the shifting conditions to meet some desired state. So maybe it would be helpful if you could define more precisely what you mean by adaptation? Not trying to be snarky, just trying to genuinely push you to try to make a clear definition of what you mean by adaptation.
A calculator is discontinuous graph/program because there is no point to plot when you divide by 0 (undefined). This is true for your classical program. Over time it won't know how to adapt its f(x) function to handle something as crazy as dividing by zero (it can never do it, but unless you put in an error output for that input, the program will never do so on its own).
The belief is these AI programs will be able to make that adaptation. All the changes you bring up that your program can apparently handle are still under your conceptual control (you can map in your mind if this happens, then that happens). You can prove this to yourself when you read your own statement "inputs have already been normalized" - sure, in a world you control.
I'm suggesting a world where your telemetry monitoring function can change into a Reddit function if need be, so yeah, kinda batshit. Why would it need to do that? Remember, we can't imagine why, that's the whole point.
That's how a program could possibly live a hundred years in a world that is constantly changing. Your program can only exist in a very static and predefined world.
There are some that will tell you that you are an ever-changing function. I try to stay out of that simulation :)
https://en.wikipedia.org/wiki/Matchbox_Educable_Noughts_and_...
I wouldn't say that's more "intelligent" than, I don't know, a workplace scheduling system.
Likewise, any symbolic mathematical system that accumulates theorems that speed up future work as it solves current tasks, seems like a high intelligence type of activity.
Deep learning is “just” structured arithmetic.
I think different kinds of intelligence can look quite different, and they will all be “structured” or “tropic” at their implementation levels.
—
Stepping away from the means, I see at least four “intelligence” dimensions:
1. Span: The span of novel situations for which it can create successful responses.
2. Efficiency: The efficiency of problem solving.
I.e. When vast lookup tables, exhaustive combination searches, and indiscriminate logging of past experience can be matched instead by more efficient Boolean logic, arithmetic, pattern recognition and logging, we consider the latter more intelligent.
3. Abstraction: The degree to which solving previous different novel situations improves the success or efficiency in solving new problems. I.e. generalizable, composable learning.
4. Social education: Ability to communicate and absorb learned information from other entities.
Plants, and I expect all surviving life forms, are very high in intelligence types 1 and 2.
Adaptive nervous systems and especially brains excel at 3.
Many animals, but most profoundly humans (whose languages for communicating are themselves actively adapted for compounding effects), excel at 4.
Today’s humans are effectively more intelligent than 10,000 year ago humans, not because of 1-3, but because of 4. Learning as a child to read/write, do arithmetic, understand zero and negative numbers, and countless other information processing activities and patterns, from others, profoundly impacts our intellectual abilities.
Deep learning, as with the human species, non-trivially spans, and continues to improve, on all 4 types of intelligence.