816 karma · joined April 7, 2016
(although this is specific to S3 rather than generic HTTP)
(shameless plug - made mostly by me)
I think for 2 reasons.
Firstly, they don’t account for complexity you need to solve your problem. And secondly, they seem to suggest that splitting up the same functionality over more functions, so each has a lower complexity, is “better” - this isn’t always the case, at least for me. Often I find following the flow/logic of loads of small functions more difficult than through lines in the same function.
The examples I’ve seen in articles such as this are always fairly clear cut “well yes of course these small re-written functions are clearer”, but they don’t really seem to consider the wider context of complex requirements?
I don’t really have an alternative metric to suggest though… I wish I did!
I wonder if SQLite didn't store the schema for a table as a "CREATE TABLE..." string, but instead something a bit more... structured, then such things could be added in a more backwards compatible way?
(Full disclosure: I'm the main developer of stream-unzip)
… but I do wish they would implement being able to store objects at arbitrary keys. Some combinations of keys are not possible to store at the same time. And I guess to me it seems… odd… and maybe even frustrating?, that they say it was a “blunder” on AWS’s part to implement such a thing. [1]
For example, as far as I know, in minio you can’t have objects at both of the below keys:
red
red/blue
while in AWS S3, this is fine
(In fact, that project was one of the sources of inspiration for this one. I've now added that fact to the README)
I can understand the AWS auth code... while there seem to be so many layers in boto3/botocore. There's a lot of code to add a few HTTP headers to each request. (It was a few years ago, but I posted about not using boto3 https://charemza.name/blog/posts/aws/python/you-might-not-ne...)
I might be suffering a little bit from not-invented-here syndrome admittedly... I don't know. But one of the things I really like about S3 is how much of its API is quite simple: it's just HTTP (I briefly wrote about _that_ as well https://charemza.name/blog/posts/http/s3/the-core-s3-api-is-...). My instinct is that in many cases, it's better to use a simple API directly rather than (what I see as) a more complex code base on top of it.
But I think there is a, or could be, a place for this:
- I don't think (but very happy to be corrected) that the things you mention can query SQLite files directly. So, there would have to be both a conversion of data, and then the query language used would not be SQLite's flavour of SQL. So, if you're using, or pro, using SQLite and wanting to keep the moving parts and languages down, then I _think_ there is a good argument for sqlite-s3-query to be used (with the caveat that it's a _very_ early version!)
- If you want, "light" dependency on AWS. There are a number of S3-compatible services that this could work on. Or even, the code could be tweaked to just use HTTP range headers, remove the dependency on versionId, which I think is only feature that makes this quite S3-specific. (Although, this would allow the case of errors/corruption if the database is replaced half way through a query...)
I have now given them credit.
I think Zip64 _is_ supported by Python, and doesn't really involve anything at the zlib level(?)
(The reason I'm asking: a recent answer at https://stackoverflow.com/a/67579486/1319998 mentions Deflate64 and Windows)
- Getting something "out there" + "working" as soon as possible. Even if _really_ rough, buggy, ugly, can't scale, only works in 1 case, needs a code change to work in any other etc...
- Keep it working with every change
- Every change is small. Potentially, teeny-tiny
Examples:
- Forking a project from GitHub. Have something working straight away! I've done a few project that literally have no lines of code in common with the original, except perhaps the license [so not appropriate for all projects]
- Releasing a website with whatever it is really early. Even if unstyled, no documentation, but does a tiny sliver of what you want it to do eventually. Then iterate, releasing the new version every single change.
- I think some people suggest avoiding tests in early stages of projects, but for me... some high-level tests on the "public facing" behaviour really helps give me the feeling that it's working as changes are made, and I'm never stepping backwards.
Things to avoid:
- Long running feature branches where lots of things are broken
- Refactoring without some sort of externally-facing feature in mind
- Low level tests of implementation details
- Working too much on "behind the scenes" stuff. You probably don't need a really fancy CI system that supports 100 developers deploying every day... [But: enough of a system for _you_ to release often is good]
"What do we want?"
"Now!"
"When do we want it?"
"Fewer race conditions!"
http://projections.charemza.name/
... click and drag a Mercator map to rotate the world "before" the Mercator projection is applied, to see a whole range of "equivalent" projections. None of them look like the world to me, but they're just as valid.
The issue is what happens on exceptions to avoid tasks hanging around forever. Exceptions propagate “forward” in the pipeline fairly automatically. However, getting the ones “behind” the exception to notice: that was tricky. The best I came up with was the bit of state wrapped up in buffered_pipeline that notices the exceptions and cancels the tasks.
While yes, running a coroutine in a new task is in the standard library, that is only a component of what this library does, which is:
- given an async iterable,
- iterating over it in a new task, into a buffer,
- and then returning an iterable that yields values from this buffer as it's filled.
I don't _think_ this logic is in the standard library, at least not at https://docs.python.org/3/library/asyncio-task.html#asyncio.... from what I can see.
I was slightly torn on the usage of the word "parallelise", in that yes, CPU-wise, no tasks will progress in parallel.
However, the point of the library is to allow certain parts of a pipeline to progress at the same time in the real-world. In the example, they are calls to asyncio.sleep, but in real cases, they could be HTTP requests. As in, the bytes of the request/response will be going across the wire at the same time, so the total wall-clock time of (certain) pipelines will go down.
So, I claim "parallelise" _is_ appropriate.
So far, I like it: a variety of modules, and each I have taken so far has sparked potential dissertation ideas :-)
https://www.ed.ac.uk/bayes/about-us/our-work/education/data-...
I was hoping to get across exactly that meaning of leaky: that HTTP shows information from, and allows control of, parts of the otherwise-abstracted-away lower-levels, e.g. TCP/IP. Will see if I can make that clearer.
> Hash tables provide constant time O(1) access for single values, while trees provide logarithmic time O(log n) access. For single value lookups, this means hash tables are faster,
> for small n, the hash table is better, but for large n, the cost of that rare scan dominates, and the tree is better
The Big O notation describes how the number of operations scales as the size of the input tends to infinity: the asymptotic complexity. This means that an O(1) algorithm can include some fixed number of operations, even enough to add _years_ onto the runtime, and still be classed as O(1).
Essentially I don’t think it’s really enough to consider the asymptotic complexity of algorithms to compare practical runtimes. You need to know what the smaller-order / constant terms are.
If the MacOS filesystem was not case-insensitive, i.e. it was _less_ liberal with what it accepted, the reported issue wouldn’t have happened.
There isn’t really the term “jaywalking” here. It’s just “crossing the road”. I’m not sure on exactly who has what legal responsibility, but it certainly feels like pedestrians should look out for cars when crossing, and drivers should look out for pedestrians.
I couldn’t help but smile.