The "LLMs perspective" section is hiding at the end of this notion is a literal goldmine
14 karma · joined March 29, 2021
They could make so many people so happy if they just gave us a damn good app.
I also can't see how this benefits them at all - their app isn't tied into monetisation, it's free to download (replacing their own chargeable reader!), and it doesn't work at all without a sensor (the paid component).
The only reasons I can think of are: - legal restrictions on them as a healthcare provider? - preventing users from jailbreaking it to work longer (honestly people only do this if they legitimately can't afford a new one) - they want user data for monetisable purposes
Thoughts?
I created an HTML page that used vis-network to created a force-directed nodegraph. I'd then just open it up and wait for it to settle.
The initial code is here, you should be able to dump it into an LLM to explain: https://github.com/HebeHH/skyrim-alchemy/blob/master/HTMLGra...
I later used d3 to do pretty much the same thing, but with a much larger graph (still only 100,000 nodes). That was pretty fragile though, so I added an `export to svg` button so you could load the graph, wait for it to settle, and then download the full thing. This kept good quality for zooming in and out.
However my nodegraphs were both incredibly messy, with many many connections going everywhere. That meant that I couldn't find a library that could work out how to lay it out properly first time, and needed the force-directed nature to spread them out. For your case of 1 billion nodes, force-directed may not be the way to go.
But I've ended up in several headed discussions on this, in the context of SQL vs NoSQL. The one argument I've been slightly persuaded by is: schema-on-read is significantly more flexible than schema-on-write. In most of my actual programming applications I still use SQL, because in coding I think it's better to prioritize planning and structure over flexibility.
If there's one area where flexibility is necessary, it's real life. When I first start researching something new, I don't usually have enough knowledge to actually structure my schema effectively, and the usefulness degrades. Of course, you can update the schema. This isn't always great. If your schema needs to change constantly, it wastes a lot of time. Depending on complexity, it can also just be a massive cost. Eg: when I have the kitchen set up all nicely, but then we get a new blender with five different attachments and now I need to find an empty shelf for them. Took about an hour to reshuffle everything satisfactorily.
All that said, I'd still say that schema-on-write is better than schema-on-read. Some structure is typically always better than no structure.
However, I've recently been reading a book that I think gives an interesting different insight to this problem - Designing Data-Intensive Applications, by Martin Kleppmann. I've always considered the main categorization of databases to be schema-on-write vs schema-on-read, but this gives a completely different method: databases are either document-based, relational, or graph. Relational databases we're all generally familiar with, while document-based is similar to today's NoSQL.
Graph databases have fallen out of favour, but I actually think that they might be the best at representing the human information gathering process. They have a structure that's provided on write, but isn't always consistent across entries - because it's flexible, and can be added to very easily. This lets us expand the schema as we gather more information and our view of the world changes, without needing to rearrange our past knowledge. I also feel like a graphlike structure better represents how we think.
Honestly, the main useful point I got from the book is that schema-on-write doesn't need to mean lack of flexibility. That is the case with most of the RDSs we use today, so it's what I've come to expect. But that shows a lack of imagination on my part, rather than any inherent restriction.
But you're right, the speed is definitely the biggest problem. I'll definitely take that idea seriously if I do anything else with this.
Fanwikis are my next target - probably a bigger amount of interest, and I'm hopeful that there will be a similar amount of consistency. I'll also need to work out what the interesting relationships to trace would be for fandoms - or see if I can parameterize that, since I suspect it varies a lot.
I now understand a bit better why Wikipedia isn't that popular as a base layer - it's built by humans, for humans. That can make it a bit tricky to parse for computers. There's no consistency to naming, labeling, or even organisation. There are some cultural standards, but they're not binding and tend to vary between sections. For instance, children are labeled as 'children' for most people, but 'issue' for a lot of monarchs, and 'offspring' for mythological characters.
What do you find your customers care about the most? Security, cost savings, UI? It seems like a pretty old-tech industry that you're digitalizing, so I'm curious about what the pull is from their side