Very cool - I had no idea this existed. I wonder if this could be useful for using ramdisks when working with temp tables.
767 karma · joined August 31, 2011
vaughanhackernews at gmail
If your internal representation and API start to differ then it adds complexity fast. Its far better to have as close to a 1-1 mapping for your backend and frontend data models as possible.
I see the problem here being that too many manual optimizations need to be done when implementing a schema.
You start with a logical schema (ERD diagram) and then implement it via a physical schema with denormalizations added for efficiency (usually because the relational model has scaling limits with number of joins, or because db migrations are too difficult, or handling unstructured data without introducing a ton of new tables). The db should do the denormalization automatically, allowing the user to interface with their logical schema directly.
Another reason is we can't use SQL in the browser - we have to go through many caching layers and API layers which complicate things.
Real-time queries should be the default. Some kind of smart query invalidation (similar to incremental view maintenance) as a result of DML statements.
2. Can run in a web browser / mobile phone
Your client-side cache is usually a messy denormalization of your SQL database. For offline/local-first apps you essentially need to run your entire backend API in your browser. So you should be running an SQL database in the browser. For this to be seamless you need your database targetting JS during design stage.
3. Syncing
To support offline/local/mobile use cases, your database needs to be able to sync changes efficiently.
4. Logical-first data schema
You should be able to give your db a logical schema instead of a physical one, and the database can take care of the necessary denormalizations for performance reasons automatically.
5. Lower-level api for query planner
Instead of having SQL as your only interface, there should be a way to interact with the query tree and the query plan directly. People are always creating new query languages hacked over SQL (like Google's Logica).
6. Graph-based queries
Support a query language like Cypher. Provide better solutions for tree-based data structures - the queries get too unwieldy today.
Although the gallery feature of CC is a bit buggy. A better flow might be to sync an album to Dropbox - but syncing plugins are always unreliable I find.
I just sync one album of photos I want to share to CC. LRC stores every photo I have ever took on a 10TB WD MyBook.
I'm not sure how well CC syncs changes back to LRC though which could be a problem.
For me this is the perfect combo/workflow and I'm happy to pay the subscription. LRCC has some bugs and the gallery view could be better though.
The best thing is that I manage the photo files and folder structure locally with LRC, and can back them up easily, and maintain a good archive without having to sync everything.
I found Apple Photos way of taking full control over your photos super painful. The app is terrible and some tasks are so inefficient. Changing your primary photo library to another disk for example will take days - unless you have an SSD. This is absurd. The library is one folder called `Photos.photolibrary`, but the app seems to need to read every single file. And so many other issues too.
Same here.
I used to use 2x Seagate portable 5TB. Would mirror one to the other every week.
Now I use a Western Digital MyBook 20TB in RAID 1, and backup this weekly to my portable externals using Carbon Copy Cloner for macOS (ideally I would have an identical second one for this).
And run Backblaze for offsite backup.
I avoided a NAS because I prefer the speed of USB3 versus network - super quick to view RAW and video, and for ease of relocation.
I also have a 1TB portable SSD for my Apple Photos syncing iPhone content without filling up my MacBook disk.
The WD MyBook's are a bit dodgy though, there are so many horror stories. They encrypt all data, so if your RAID config is corrupted...which is supposedly stored on the hard disk, then you are screwed. Also, when you tell it to sleep when inactive, whenever you open a file dialog, the laptop freezes for 5/10s while it wakes up. There is also a bug in the utility application that says "can't access drive" which makes you think your drive is broken but it's actually just a software bug. Not running the utility software fixes it. I also think it can cause some unable to sleep issues for macOS. Sometimes it is super loud too, and its quite noisy.
Schema design is a process of discovery and code-first approaches become too sticky to change.
I think we need to head back to the ease of MS Access for data modeling and migrations should be automated.
Schema design always results in endless "are we gonna need this?" questions, and usually everything is postponed down the road because it adds needless upfront complexity.
All data are naturally triplets, and with the relational model we are forced to group the triplet attributes into tables. The difficulty arises because these groupings are often incorrect, and then difficult to change. There is always premature optimization, because the SQL query statements become insane with too much normalization.
A bigger problem is how sticky schemas are due to how difficult they are to change, and how difficult it is to update your code to support a schema modification.
I think these two problems need more attention.
We've gone too far down the code-first / code-generation ORM approach which makes tinkering difficult.
I think all database design should happen visually. MS Access style. You should design your schema alongside your data (as you suggested by using spreadsheets). But instead of then transferring that to code, it should be a living schema that connects to your IDE code refactoring. The more declarative queries are and the closer they are to what needs to be finally rendered the more chance of reliable refactoring. The more imperative code modifying the queried data, the more complex and difficult to refactor things become. A lot of the cruft in a codebase is when schema changes are additive because people are too lazy or too risk averse to refactor.
E.g. Think about a User having one address, and then needing to support multiple addresses. There's quick and dirty way by adding more columns or adding to a JSON column, or we add a new Address entity and keep the existing User address columns, or we delete those columns and migrate data to the new Address table - which is the cleanest way to represent this schema change. I think there are few who would do the last option though, and this is the problem. Hiding this behind an API also causes more problems down the line because our frontend data model starts to separate from our actual DB data model.
We need better tools, and less code.
I'm not sure you need a full SQL implementation on the frontend though, as the data is not going to get all that large to need the optimizations it affords, but it would be nice to be able to use the same queries on your browser DB as your backend DB.
There is a moment when you step back and realize that all the data-wrangling code we write in apps is essentially what an SQL query planner does. And our frontend is just one big materialized view that we need to update.
I often think that if we had a database that runs in the browser, and we could subscribe to queries, and missing local data would be fetched on-demand, frontend development would be a lot easier. Its of course much more complicated than that.
I think the reality though is that backend scaling requirements always end up dwarfing frontend productivity concerns. And there is also a huge amount of glue between backend data sources, preventing the creation of a clean database on the backend to sync with, meaning we are creating API gateways and GraphQL federation layers, and all we can hope for is a good client-side caching layer.
If you look deeper though you will find many projects doing all these kinds of things already. Mobile developers are very familiar with offline techniques and SQLite on the client, and it doesn't feel like anything special to them. For the web/desktop, maybe no one has packaged it in the right way yet, or maybe we are still digesting GraphQL, SSR, and serverless, and then there will be another shift with offline-first, reactive SQL in the frontend.
It's funny though that if you think long enough about these concerns, you always seem to end up wanting some Datalog thing.
Run `sudo watch -d -t -n1 thermal levels`.
When levels are above 100, your laptop will throttle. Once they return to 100, throttling will cease. There is a back-off algorithm at play too, so its not good enough that your CPU returns to normal temperature, it must stay at normal temperature for a period of time before things stop lagging.
Also, disable turbo boost.
Throttling occurs when these values go over 100. Throttling is not directly linked to temperature. There is some kind back-off algorithm that means that even if you trip this thermal throttling threshold, it will take some time before throttling is disabled. Even if you CPU returns to a normal temperature.
Also, disable turbo boost.
First, disable turbo boost.
Second, point a desk fan at your MacBook.
Third, eliminate processes that peg the CPU.
---
How to know when it is throttling:
Download Intel Power Gadget and watch the Frequency graph. If blue line doesn't match the red line then you are throttling.
Run `sudo watch -d -t -n1 thermal levels`. When any reading goes over 100, it is in throttling mode.
Once it trips over 100 you start throttling. Throttling doesn't end though until this value returns below 100. This number is not directly linked to temperature either. Even though your CPU may be at a safe temperature like 80 degrees which normally shouldn't throttle, your `thermal levels` will not decrease immediately. The best way to get them to decrease is by sleeping your MacBook.
Maybe there is a hard-coded delayed back-off algorithm that is a function of the time the CPU spends above a certain temperature threshold - I'm not sure.
I have also read that its potentially the voltage-regulator module that doesn't have a temp sensor that this may be linked to.
---
You can also do a [cooling mod][1] if you really want to but its probably going to void and warranty and its risky.
[1]: https://forums.macrumors.com/threads/2018-macbook-pro-coolin...
The scrolling is a lot less smooth on the canvas than the DOM.
Text selection seems faster but the canvas version is not editable...
Maybe they are working out the kinks though...
I'd love a search indexer that runs locally and incrementally, so that I can search without building/publishing.
You can mix guides/tutorials with generated API docs.
For example, I really like having simple API docs in the `readme.md` file, but then also in-depth docs elsewhere.
MDX + remark really let you do anything you want. It's the ultimate documentation stack.
Wish someone would do this for Logitech Harmony remote controls.
1. The software was terrible but hardware was awesome.
2. The entire line is discontinued.
JS/Node is full of unmaintained projects that people inevitably need to migrate away from.
Of all the new choices in bundlers today, it’s also inevitable that some won’t work out, yet they will drive hype and adoption, which could have been spent on already established tools such as webpack.
Yes, innovation is good and no one should tell someone what to do in their free time.
But it’s always going to be harder to build on top of old rather than to write something new, but what is new eventually becomes old...and the cycle repeats.
It’s usually slow css loaders, or too-inclusive patterns for loaders ending up processing node_modules, and not good enough disk caching.
There is a plugin called smc to monitor loader execution time.
Dll plugin was the best speed up to avoid recompiling things that don’t change, but now module federation is suppose to be a better solution. Disk caching in v5 will also do great things.
Thing is, it’s so complicated to setup, and debugging it involves so much config tweaking ans waiting.
I think we are a few years out from when we have nice speedy builds in webpack with good abstractions.
But then it’s a question of whether people move away from webpack because we don’t need all the plugins and transpiration and want native compile speeds.
One thing for certain though is the hype cycle will continue on...
Of course it would prob be better if this was a toggle.
I think webpack module federation may improve this situation too. I used webpack all plugin previously to speed up vendor dep compilation.
Thing about webpack though is it’s so complex how all this works that I always have to revisit it every few months to jog my memory.
We need simpler abstractions on top. Next is nice but still, if you need to dive deeper it’s painful.
I think in like 20 years time we will probably be able to get rid of all of this tool chain and the new kids will never know the pains we went through.
Check out: https://wiki.postgresql.org/wiki/Incremental_View_Maintenanc...
Just as document dbs can be derived/denormalized from SQL dbs, relational dbs can be derived from a graph.
Conceptually, data is a graph.
I always find the decision between 1-M and M-M is so sticky with RDBMS, and with a graph, it can be whatever you want it to be.
I shouldn’t have to re-adjust my mental model because of some stylistic/level-of-detail decisions by the author.
These diagrams would be interactive too.
Many systems are very similar in what they do. There are many times I am asking myself: “but how is this different from <well known architecture pattern>”.
Think about categories like append-only logs, search indexing, web requests, git trees, etc.
I’d love to be able to diff against two diagrams to the same problem and see how the approaches differ.
Maybe you could speed up your queries by using a materialized view for `FROM products WHERE id > 1234`. You could then maybe derive more materialized views for other attributes depending on the clustering of your watched queries...e.g. gender might split the dataset in half, but this is not guaranteed to improve perf and indexes might actually be faster, place there are space concerns and additional indexes.
If you want to run less queries you could combine queries like [Hasura does][1].
To avoid running queries with no new results you would need to watch DB writes, and map DML statements to query subs...but pointless in a batch setting.
[1]: https://github.com/hasura/graphql-engine/blob/master/archite...