Comment by naiv | Hacker News Reader

Comment by naiv | Hacker News Reader

Parent Full thread

For me it is not even the code per se but how it helps me with naming database tables and properties when I describe the broader scope.

The future is more than bright, especially with a new model that is more current.

Offtopic but just today I was aksing it for the most current versions it knows of right now:

Python: 3.9

JavaScript: ECMAScript 2021 (ES12)

Java: 16

C++: C++20

C#: 9.0 (.NET 5.0)

Ruby: 3.0.0

Swift: 5.4

Go: 1.16

Rust: 1.51.0

TypeScript: 4.3

PHP: 8.0

Kotlin: 1.5.0

Scala: 3.0

R: 4.0.5

Perl: 5.32

And frameworks:

Python:

Django: Version 3.2

Flask: Version 2.0.1

Pyramid: Version 2.0

TensorFlow: Version 2.6.0

PyTorch: Version 1.9.0

JavaScript:

Node.js: Version 16.9.1

Express.js: Version 4.17.1

React.js: Version 17.0.2

Angular: Version 12.2.4

Vue.js: Version 3.2.6

Java:

Spring Framework: Version 5.3.9

Hibernate: Version 5.4.32.Final

Struts: Version 2.5.26

PHP:

Laravel: Version 8.54.0

Symfony: Version 5.3.6

CodeIgniter: Version 4.1.4

Ruby:

Ruby on Rails: Version 6.1.4

Sinatra: Version 2.1.0

C#:

.NET Core: Version 5.0

ASP.NET: Version 5.0

Entity Framework Core: Version 5.0

Asking it directly is a good way to get a response containing version numbers that look correct, but not a good way to actually indicate which versions it's been trained on.

A better method would be to look at the language features it's using and infer from there. Or, better still, look at which versions were out when the training data was collected (which I believe is September 2021, don't quote me on that).

Is there a risk of it hallucinating what new features could do instead of basing information on having seen them before?

Yes, good point, I suppose we would have to weigh the probability of that against it fudging a number. I would assume inventing a new feature is harder, but who's to say; all this LLM stuff comes down to probability.

It does not actually mean much. These chatbots can easily output JavaScript using var where it absolutely should not. It being aware of the latest standards does not mean it can properly use the new syntax or features -- it emits a form of whatever crappy/legacy code it was trained on.

Oh that's interesting, I didn't think about asking it directly about what version it thinks is current. I wonder if it can hallucinate here?

It's really irksome when it tries to use functions which were renamed or removed. This could be detected automatically (and possibly remapped in some cases)

I got mostly the same versions as you, both on chatgpt3/4, using english.

If you ask it in other languages the minor version also seems to change often.

Finnish gives you dates, and code-davinici-edit-001 is 1 major release behind on almost everything.

Tässä on luettelo ohjelmistojen viimeisimmistä vakaiden versioista:

Python: 3.9.5 (30.4.2021) JavaScript: ECMAScript 2021 (23.3.2021) ECMAScript: ECMAScript 2021 (23.3.2021) Java: JDK 17 (28.9.2021) C++: C++20 (20.2.2020) C#: .NET 6 (8.11.2022) Ruby: 3.0.2 (24.8.2021) Swift: Swift 5.5 (20.9.2021) Go: 1.17 (16.8.2021) Rust: 1.54.0 (27.5.2021) TypeScript: 4.4 (28.7.2021) PHP: 8.0.9 (29.7.2021) Kotlin: 1.5.31 (26.8.2021) Scala: 2.13.6 (17.2.2021) R: 4.1.0 (18.5.2021) Perl: 5.34.0 (30.5.2021) ... Sinatra: 2.1.0 (10.4.2021) .NET Core: 6.0 (8.11.2022) ASP.NET: 5.0.10 (19.8.2021)

All it is doing is responding with things that look like accurate languages and version numbers. They may be accurate but are absolutely not representative of what it has been trained against.

But this is just a plausible looking answer, it is still a text generator, this has no bearing on reality per se.

i just asked it "what's the output of python -v" and the little sample code reported python 3.9.2.

I got:

    Python 3.8.5 (default, Jan 27 2021, 15:41:15)
    [GCC 9.3.0] on linux
    Type "help", "copyright", "credits" or "license" for more information.

And a lot of junk text.

Ha! So far it seems four posters here have asked similar questions and got four different python versions.

I don't know if we can treat the gpts in this way and expect reliable answers. We just get pretty good answers right up to the point that we don't.

Agreed, I think this is where its "stochastic parrot" nature shines through. All those version numbers are probably equally well represented in internet text.

Reply on news.ycombinator.com