Also, for the record, that's largely how I think of databricks ideally. A big distributed database/storage layer into which I can write queries in my chosen language...
So i...admit equal confusion...
3,921 karma · joined October 18, 2013
I have experienced some of the joys and horrors of SAS, Common Lisp, SQL, R, python and C.
Also, for the record, that's largely how I think of databricks ideally. A big distributed database/storage layer into which I can write queries in my chosen language...
So i...admit equal confusion...
But if you believe that, then 9/10 women found shampoo/moisturiser to improve their shine/visibly fade wrinkles!
/that is to say, self-reported testaments and improvements have historically been the chosen method of snake-oil because placebo and self-dellusion are so powerful. There does not appear to be robust evidence that it works at all yet.
I know you're (presumably) not the author of the website, but if you're going to produce a method based on the theory of faster/easier reading, it helps to not have a landing page that has up front in big massive fonts a sentence that my mind parses as:
"Did you know that we humans store learned words and so just a few letters your brain reads faster are enough to recognise whole words. than your eye?"
How can one take it seriously when the very first example on the webpage fails on reading cognition!
we are basically in no risk of running down the population within any of our lifetime...
... on the proviso that everything doesn't go pear- shaped.
And with a recognition that those risks of things that might go pear-shaped are probably highly correlated with the carrying capacity of the planet with regards to human population numbers.
Perhaps an ironically appropriate test for hiring for a job at many offices/IBM (and exemplifying some of the problems with these bad faith test techniques).
while csv is less than optimal, it's always worked out in practice for me (because we still have to validate anything of significant size and nothing finance or city/nation sized should generally be edited by hand anyway, and errors are everywhere independent of format).
honestly, my bigger nightmare has been people trying to put things into something like xml...
edit: not that there aren't huge problems with the csv/excel/database software and interaction: generally I just end up having to assume everything is text and defining type information my end afterwards on successful parsing
but the use of percentile to percentile (or quartiles, but those are just grouped percentiles) to give the impression of a particular kind of effect (lower groups overestimating and higher groups underestimating) is a flaw i think. it's a common one, in my experience, when dealing with percentages.
if you think about it, percentages/percentiles have to be bounded at 0 or 100. for a dunning kruger effect not to appear, participants at both ends of the ability spectrum would have to be eerily accurate in their self-assessments. if they aren't, there's just more space on one side of the measurement scale for each group to make an error (if you score 1 in ability, there's ~99 percentiles available for you to make an overestimate and only 1 to be accurate, ditto for those with high ability. those in the middle of the ability group have equal chance on either side and so appear statistically more accurate even with a purely random distribution of guesses). so if there is any measure of central tendancy towards the middle percentile in the estimations of peoples abilities at all (and i would argue there is a priori reason to believe there would be, as the alternative would require those at both ends of the distribution to be getting increasingly accurate, which would be really weird), then practically any real world graph of percentile performance to percentile estimation will show a dunning kruger effect (with lower ends overestimating and higher ends underestimating). the article does a good job of showing this by plotting just random estimations and observing one appears.
But my wife has worked at a company that did offsite kinda vacation/camp things, and gotta say, it ranks up pretty high there on the corporate dystopia cringe scale for us.
I work at a semi distributed organisation atm (several offices in different cities) and we generally used to have once a year offsite. while it was nice to see some faces in person, the experience of leaving my family to do so, and to have to go to locations and accommodation chosen by work, was again just a downer. I know other people looked forward to it, so horses for courses, but I also know there were people working long hours and liking the travel because they had nothing else going on in their lives.
I think the strengths of centralisation stand independent a lot of these practices, that make a lot of us shudder on some level.
I also did some work experience at tidbinbilla tracking station when I was younger, as well as the radio telescope array out near Narrabri. the NASA aligned tidbinbilla one had a company cafeteria, which at the time I put down to some American cultural import thing, because it seemed weird as hell to me. the Narrabri facility isn't exactly in the middle of the city either, and is the only place I've worked with a bus that came round to pick you up from the town each morning (I've heard some tech companies do this in Silicon Valley?) and even they expected you to make your own lunch (there wasn't really an option to go get food from a shop). on the upside, they did have a volleyball court, so swings and roundabouts...
- CPU $1100 AUD
- GPU $3000 AUD
- RAM $1000 AUD
- Motherboard $279 AUD
- NVMe Drive $300 AUD
- SSDs $150 - 300 each AUD
- SATA drives $100 - 300 each AUD
- Powersupply $240 AUD
- Case $250 AUD
- CPU cooler $170 AUD
That prices the full thing as new at about $6000 to $7000 AUD, which is vaguely in line with about what I recall my budget being, though I've had it for a year or so now.
That would put it at roughly 4k - 5k USD by my napkin math, but as a general rule US consumer tech prices recieve a bit of a discount compared to us over here, so you could probably come in on the lower end if you could physically get everything (i don't know what part supplies are like at the moment in the US).
Taking the AU Apple store price, accepting that their CPU probably outperforms mine in some multithreaded performance, and going for the top GPU option, I'd price the Apple M1 Ultra rough equivalent between $7000 to $10,000 AUD depending on where you want to cut it.
So like for like i'd say 4k-5k USD for mine vs 5k - 7k USD for the M1 in my locality.
See: https://browser.geekbench.com/v5/cpu/compare/13330272?baseli...
I think you'd expect the M1 to outperform a bit on the multicore because it has more cores, but the comparison suggests a lot of the current benchmark score is being driven by exceptionally high scores in a handful of benchmark sub-tests.
On single-core I'd describe them as very very comparable at the moment.
There's another issue that was made apparent when I benchmarked my setup which suggests that a significant portion of people running 5950's are leaving a portion of performance on the table (i presume that's some mix of not setting things up right or not having the right mix of hardware).
I'd say this is an issue that doesn't happen with Macs, but...I'm aware that their past laptops have run really really hot (and dropped performance below their apparent specs due to insufficient cooling?), although everything I've seen shows the M1 family to be a a relatively great performer in this regard (to the point where I recommended my wife pick up the new M1 Macbook Air).
And additional observation is that they're claiming ownership and exclusive rights to things over things which:
a) as broken and dystopian as our intellectual property laws are, it's not immediately apparent you can claim exclusive ownership of. Can you claim property rights on a pin number? on your customer's name? on your customer's phone number? is data even ownable?
b) as above, even if you could, it's not apparent that the airline is the one with the greatest claim to that ownership. Does the airline own my name if I fly with them?
c) the issue with screen scraping may just be scale, automation and commercial value, and it's not apparent you can just wilfully ban competitors from that because you say so. Indeed, is it a violation if an individual uses the information within without screen scraping? cause a lot of those exclusions and terms would seem like they restrict the use of the website and the information for its actual intended purpose on an individual level, screen scraping or not.
d) what are we doing when we read a website but biological screen scraping?
I think giant programs that have to interact with the real world are nightmares regardless, myself. And small programs written in a value-passing functional style by competent programmers don't really have very many issues with managing state, in my experience.
The short answer is Common Lisp isn't really "functional": it doesn't manage side-effects. You can just go imperative/side-effect crazy if you want. Just "setf all the things" or use CLOS for objects everywhere (i almost never actually use CLOS or complex objects, so maybe that's why I'm a bit unsure of what there is to manage).
The long answer is, i guess, a combination of several features and conventions:
- A number of internal functions offer two versions: a destructive and a non-destructive version. (remove) and (delete) are examples of two functions that do the same thing in the abstract, but one is allowed to mutate it's target list, primarily for efficiency reasons.
- Regardless of the above, you tend to program in a universal style of accepting and doing things with return values from functions: so you explicitly "manage state" in the sense that you might assign return-values to variables, but it would be extremely weird to actually rely on a destructive side-effect of a function itself without treating it's result like a returned value of a function. To use a function for it's destructive side-effects isn't really done, and even if it IS, you'd treat it in a style that would be about treating it AS THOUGH it were just a returned value from a function. Almost universally you only use the destructive version for some computational efficiency reason, never because you expect to predict the outcome of what would happen to the underlying memory that you're mutating. That's left up to the compiler to give you the correct answer (and it may or may not mutate the underlying thing at all). You just accept that the function gives you the correct result.
- The various name-spaces tend to separate out clashes and management on that front: values go in one name-space, functions/macros go in another, and packages and things are another demarcation (i recall there's technically more name-spaces, but those are the main ones).
- There's no official "global variables" in the "normal" sense of the word. There are dynamic and static variables, which might be analogously thought of as global variables/values, but those tend to only be used for their specific use cases, and convention is to mark them as special exceptions. Variables with values in them are explicitly declared in functions, and you're responsible for choosing to manage state within functions, but due to the abstraction of functional boundaries, you'd generally never think of what mutations are actually going on inside a function unless you're authoring it.
- garbage collection and things dropping out of scope generally just take care of most things in practice.
- If you REALLY got into some sort of nitty-gritty bit-twiddling state nigthmare, you'd fall back on macros/testing/formal proof to help you. But most people are not doing that.
- Multi-threading isn't part of the spec IIRC, so that depends on the compiler extension and the problem. Race-conditions and ambiguities in that respect aren't really handled by the common lisp spec.
- These days with the advent of test driven development, there are testing frameworks that can be used for complex programs, but I'm hesitant to say anything in that regard as CL was really written before that became a cultural thing.
Of course, with all of these, there's the assumption that you can generally write in some functional/non-destructive style if you so choose. So if that solves the problem or makes your problem simpler, that's generally what you'd do. And if it doesn't, then it's not apparent that functional-ness of your problem is really the main problem at hand...
I hope that helps, feel free to ask me a few more questons if it helps, but also take note that it's been a few years since I've done anything with Common Lisp :\
Edit: I should say, if you're working with things like "inifite structures" or "delayed evaluation", you'd generally have to manage that explicitly yourself somehow, as CL is generally eager in it's evaluation style, but that's not most problems in my experience.
His "enlightenment" moment didn't result in him walking away from his father's world (or the money, or the emptyness of living a life to play tennis). in many ways its the opposite lesson of the article.
It helps to remember of course that the whole anecdote and narrative is likely just more commercial promotional fiction. everyone likes a redemption story, especially when it's somehow a redemption story that doesn't result in recognising the emptiness of fans and fame and money and professional sport but a kind of self- congratulatory reawakening that involves no actual change. This allows the commercial engine of a professional tennis players life to keep turning and draw in new fans and retaining old ones... presumably the whole actual point of publishing his "autobiography".
I'm guessing of course agassi didn't write his biography either. I'm sure his pr team/managers just hired a ghost writer.
I've got nothing personally against agassi, just for he record, don't even know or care much about him. I'm sure his father probably was overbearing: that's the harsh reality of professional sports preparation. But it does well to read these things with a critical eye and recognise the reality behind them rather than the image. At the end of the day, he's still a guy that hit balls back and forth for money just like he was brought up to.
one of my constant bugbears is even finding other data people who can and will take into consideration the context and uncertainty around the data itself, for in our societies and with our current levels of education and numeracy, there seems to be an bordering mystical connection between data, numbers, and authority. even beauty products, junk science, and conspiracies will often try to convince you with numbers.
and the struggle of how, as a data scientist and statistician, to use numbers in policy with largely innumerate policy makers and populations is an ongoing friction.
for an example on the inherent politics of numbers from my own country, observe the following from our official statistics office:
https://www.abs.gov.au/ausstats/abs@.nsf/mf/1370.0
note the date of last publication. this work was subsequently cancelled and will no longer be updated. I'm not here to say whether that was right or wrong, but merely to draw awareness to this tension.
meanwhile, while I was working in official stats, we were always extremely frustrated by the conspiracy theories around the likes of unemployment statistics or cpi. while all stats are inherently political, the amount of bad or invalid critiques of these statistics, as well as the implications that we were trying to generate then in some kind of conspiratorial bad faith was always disappointing.
I took part in a Facebook recruitment competition (this was many years ago when it was less obvious how evil they were going to be).
They gave us some data and some notional analytics task and your goal was to maximise a certain prediction metric. The catch was that they had errors/repetitions in the data such that you could exploit it to maximise the prediction score by methods largely unconnected to the notional analytics. This allowed some to get scores miles above accuracy reported in the literature, to the point where it wasn't physically possible to do that well. I'm still not convinced it wasn't an error on their part or that it wasn't an inside job.
I discovered this and basically made the claim that some of the other scores others were getting weren't possible, and in the process of chasing down what the "cheaters" were doing, they came clean with the "oh, it's part of the job to assume/find such problems in the data and exploit it". So then everyone changed their models to include this little hack and the competition continued on the remainder of the data after wasting most of our time.
but it left a really bad taste in my mouth. first of all, the metric is now meaningless in this case, so no, it's not an analysts job to exploit errors to maximise a meaningless metric. secondly, just do things in good faith: if you give me a data set and tell me it's a certain thing, I'm going to believe you. if it's dodgy or you're interested in OTHER things, then tell me so we can stop fucking around, because this isn't real life and I've got no idea what contrived thing it is you're looking for in this finite and time restricted exercise. if you want me to find some contrived thing, then at least tell me that's part of it.
gotchas where they're fucking with you or dishonest about what they're actually doing in the process are the worst. especially if they're actually incompetent, made a mistake and they're too proud to admit that fact.
they've been so successful in their advertising they've just managed to get people to call them "tech".
tech would probably be untouched, and who knows, might even flourish a bit once people stop wasting their time and paycheques to make the world a worse place.
Firstly, I'd urge everyone to deny the initial premise. The right or desire to be free of or avoid advertising has nothing inherently to do with the act of paying money or being able to amass or claim property rights over things. That's not why i hate junk mail. That's not why I don't want to be proselytised to. That's not why i shouldn't have to view propoganda. Of course, the addition of private property rights issues is one more barrier that advertising would have to overcome in order to be deemed acceptable, but on the contrary, i think a better premise is to start with WHY or WHEN advertising should be allowed, given that it is almost universally loathed or objected to. I wouldn't want to live in a world where people are subjected to advertising because they haven't payed for things or own things or content. That's dystopian.
At the very least, I think people should have the ability to say no, and for that desire to be respected and not interferred with. I think there's good grounds for advertising, or at least many types of advertising, to not be allowed in many public areas or via many public resources. I have no problems with companies or organisations not delivering material or allowing access to general consumer goods as a consequence of saying no to advertising, but that's different to users saying "no i don't want ads and I don't want to be tracked" and having the companies go "but I'm gonna do it anyway and you shouldn't be ad blocking".
Finally, I always come back to what the web is, or at least what it was, in that I deny the underlying unspoken premise that the web is there to run business models or pay money for content to be put on it. If businesses can make it on the web, good for them. But if they can't, the web is more important than business.
To be clear, the internet and web and content came first, then the businesses arrived and co-opted it for their purposes. Now the narrative is trying to be rewritten so that you can't possibly let people consume material on the web for free without people paying for it via advertising.
So let me be clear: I pay for my computer. I pay for the bandwidth of my ISP. This allows me access to a public network of computers where I can make requests to other computers on the network to send me information. People can also pay to put content up on that network to be requested by others, but no one forces them to do so, AND, no one forces them to answer my computer's request for content. When i use an ad-blocker, the main activity is for me to determine client-side that there are certain computers I don't want to talk to, and certain content I don't want to see. In an ideal world I wouldn't be tracked also, but that's just an additional abuse or manipulation of the system.
Several years after this whole system had been operating just fine, companies arrived and started trying to make money, and today we have a feigned victim-complex retcon narrative about how people not paying for content should view ads.
On the contrary, if companies do not want me viewing their content: don't put it up on a public open network that's publicly searchable, publicly addressable, and that responds to my requests for that content. Put it behind a paywall, put it behind authorisation, put it on your own intranet or network.
Keep the open web as the web. Keep it open. Keep it in good faith and respect user's explicit preferences. The hosting computer decides what content it wants to make available and who it wants to talk to. And my computer decides who it wants to talk to and what content it wants to see.
And in an ideal world, advertising should be delegated to the bare minimum of places because the starting assumption should be that most users, rationally, justifiably, and understandably, don't want it.
In many ways this is what trade magazines are/were, but the absence of a general market for such generalised interest is so absent its the kind of witness evidence that is deafening in its silence.
in fact, I'll go so far as to say that's exactly what most proper ad-blockers are: user implemented acceptable ads. if the user wants to view ads, they have it in their power to whitelist a site or domain.
perhaps a business could be made where you provide a list of acceptable ads and people can pay money to view them. on the one hand, I think that's a more reasonable business model than the one you've suggested, as at least it aligns the user's interest with their own explicit actions.
of course, there is the little problem that such a business model would probably never work, because fundamentally, most people don't want ads. they tolerate them because they don't know they could have a choice otherwise...
once you understand not desiring one, you can understand not wanting the rest, since they're all fundamentally the same thing.
I feel like I grew up on forums and I'm still followed around by the psychological archetypes and ghosts of the long-term friends and acquaintances and their personalities many years later.
I've been in communities with theologians, philosophers, republicans, democrats, economists, poor, rich, nutjobs, communists, programmers, non-programmers, truck drivers, radicals and straight down the middle types. People from Europe, Asia, South Africa, South America, NZ, Australia, North America. Singletons, children, members with families, and even a couple of oldies.
Modern social media feels so absolutely non-diverse and universally dumb, consumerist, artificial and corporate focused in comparison I don't generally want a part of it.
These days I'm largely stuck on a private discord server without about 20-30 other people that I've known anonymously for the last 5-10 years because there are no more open or interesting communities available on the general web that don't get the doors knocked down by spammers, hackers or hostile actors.
honestly, as someone who's had to put together official stats, I'm almost always disappointed to hear the conspiracy theories and methodology misunderstandings in the general and tech community. back in my day when the government wanted to do actual conspiracy they just cut the funding for measuring/recording something. but instead we get "CPI is rigged", "do you know what the REAL unemployment rate is" government-manipulation ignorant memes.
First, they have to determine the conceptual basket of goods to track. Ok, that's comparable to your 1000 "highest volume" items methodology (albeit yours is a bit simplified too, we'll ignore the problem that good or volume is complicated for everyone). Generally stats bodies do this by looking at what consumers actually spend things on. This seems emminently sensible to me, and no offense intended, superior to a top 1000 volume method. Fuel, presumably, gets included in both.
Then they need to assign a weight for every good to determine how much its mixture of quantity + price movements contribute to the CPI figure. I put it to you this is necessary. Every item shouldn't be equally weighted even in your methodology (the distribution of volume of relative goods purcahsed/consumed is guaranteed to not be uniform across all 1000 goods, and it makes no sense for no. 1 to be given an equal weight to no. 999). Stats bodies do this too. This is what is meant by weight adjustment in this context. And both you and stat agencies should be doing it.
At some point, your "top 1000" goods is going to change and you're going to have to figure out what is in a new basket at a new time. So too does the CPI basket of goods. You both need to do this. Note that your methodology does not actually fix the fuel-budget-substitution issue. Price movements in your method that invoke a strong enough substitution or consumption effect result in fuel dropping out of your top 1000 as well.
By keeping your basket constant for a given period of time and just tracking the good's prices, you track expenditure somewhat consistently for a given period. That's a good thing. There's no reference to changes in a consumer's budget. But...CPI calculating stats agencies do this too for a set basket of goods just like you do. They fix the basket and track it over time, adjusting, like yours, only for re-inclusion the basket of goods under consideration.
Lastly, comes the hedonic quality problem. Changes in quality of consumption/goods over time. Some substitutions seem just because of taste or culture (see for example, relative consumption of antimacassars over time, or certain culture's preference for certain foods or items). Others are obviously qualitative in nature (i.e. computer processing power), and most things are a complex mix of tech/economy/culture/price/quality. This is the part that captures and tries to control for your qualitative substitution problem (it's not without its controversies, but I argue that your method needs it as well because it suffers similar problems, and indeed all ones which try to standardise subjective consumption of a national economy into a single figure). Relevant to 'fuel' is its price, but also substitution to other new fuels that turn up (hydrogen, LPG, lithium-ion), but also technology efficiency gains that have changed the consumption to utility ratio. Again, stats agencies try to control for this too.
Aside from the definition of the basket of goods (a weakness and methodology shared by your "fix"), there is generally no explicit reference to consumer's budget or assumed spending patterns when tracking within a given basket. There is nothing inherently equivalent to "assumed spare money or choosing to spend or not". (I suppose I'm happy to be proven wrong on this for a given countries methodology). it's not just a simplified survey of median households or expenditure.
Now, you can (and many people justifiably do) take issue with ideas and quantification of hedonic quality problems: but it's a universal problem, your method doesn't fix it, and I think you're generally misconstruing how CPIs are calculated around the world. The problems you mention are already attempted to be tackled and controlled for in methodology, and the official methodology is already generally superior to your suggested replacement.
Edit: it's also not like stats/economics agencies are doing this WITHOUT an accesible measures of nominal wages (that is wages and resource flows without inflation taken into account) available to them.
edit 2: in the real world, further considerations like geography, seasonality, population, etc is additionally controlled and adjusted for. And many stat agencies, in my experience, offer additional breakdowns of other price series, such as geographical and categorical indexes, or various additional weightings and combinations, so if you take issue with headline CPI you can investigate other more appropriate measures for your specific use cases.
if the system was unsustainable, CPI would adjust just as the prices and products adjusted in the system. which is presumably almost exactly what you'd want out of a measure like CPI.
CPI's job is not to determine whether a situation is profitable or sustainable.
for instance: CPI. how is it calculated?
I haven't downvoted this post, but let's use it as an example.
here's how CPI appears to be calculated: https://www.bls.gov/opub/hom/cpi/calculation.htm
by my reading, the basic methodology is to obtain a basket of goods and consumer items and then track the price change in those items. yes, it's more complicated than that in the actual methodology in reality, there's likely some debatable things like hedonic adjustment and a real question over what the index should track, but the general idea seems sound.
this post seems to suggest that CPI is based on some kind of longitudinal household survey methodology weighted for how much individual households consume over time, and that statisticians and economists are simpletons fooled by basic household budget substitution effects as prices and consumption patterns shift.
CPI becomes "a load of crap". But the error seems to be in the original post's incorrect misunderstanding of how CPI is actually calculated. and the actual method is somewhat closer to his suggested base-item method.
in my country (Australia), there's some irony that i think our biggest problem with "foreign influence" isn't Russian or Chinese troll accounts, but genuine American accounts, media and American social media companies talking absolute crap and spreading the general phenomenon and quality of American politics worldwide. anonymity and Russian trolls aren't the problem when your mainstream spread so much FUD worldwide and largely serves the same purpose as those trolls but in a "legitimate" form. American media has far more reach, both in absolute power, influence and damage, than any subtle espionage agent or internet troll, and its personalities and commenters are happy to use their real names because their medium of influence is "legitimate" and they're commercially/socially rewarded for doing so. It seems, given the state of things, that the obsession with "Russian trolls and foreign actors" is prima facie absurd, and the limited influence they actually have compared to the elimination of sane discussion or valid analysis and criticism that will similarly be removed if forced to link back to real identification is something that should be considered in any cost benefit, as well as their relative effect compared to the bullshit consumerism/ partisanship/culture-war/ racist/religious/lobbyists/violent material that's seen as somehow "legitimate".
Once you find a place you like, it's not like most people are just anonymous singletons coming in and ordering a coffee and taking up a table for a whole day. You keep the place busy during slow periods, other people see the place has custom, and depending on what's there you'll invite friends or other people over or drop by at other times for meals.
On the weekends or during particular busy times, sure. But usually you need lunch or dinner during those times as well, or you'll invite people for those times.
The successful cafes around here got quite skilled at appealing to the different groups and swinging demands throughout the day. The after-work crowd is different from the workday crowd is different from the tourists and the weekenders is different from the locals, and you need to keep all of them happy to keep the place ticking over and remain profitable. It's not just a substitution between all types of customers...
If you remove the semi-american notion of being 'stink-eyed' or hurried out of the cafe, then I become more confused about how such a space has sufficient demand to keep surviving.
I admit the thought of being charged by the minute irks me in a particular way...