oddthink

It's great for basic data-analysis tasks, where you just want to slurp in a few CSV files, join them together, filter out some rows, and spit out the results.

Sure, you can do the same in R or python, but the whole process is very quick and easy in q.

From the parent, "not-quite-as-good-as-haskell-but-more-industrially-acceptable language". Not intending to be obscure.

I just started at a Microsoft shop a few months ago (first time for everything), and I actually quite like F#, now that I've had a little while to play with it. It's a good contender for the NQAGAHBMIAL throne.

Having used CMUCL/SBCL for a while, I always found ocaml's "well, you can compile your code, but then you can't run it in the REPL" to be off-putting. (For my purposes the CLR JIT compiler for F# is good enough.)

I'm slowly moving my elaborate data-cleaning system from R to python. I never got the hang of R packages and ended up rolling my own "import" scheme, but now it all seems like I've passed some critical complexity point.

I'm still doing all of my model estimation in R, and data.table + ggplot2 is still my go-to solution for interactive exploration and plotting, but I'm putting more and more of the process-csv, write-hdf5, remove out-of-bounds values, backfill, stratified sampling/bootstrap, etc., logic into python.

I may end up moving more of the estimation itself into python, now that statsmodels and sklearn are more mature, but R does that pretty well.

I've not tried Julia, but I have experimented with Java for some of the data-processing, but it's never seemed worth it, over python. Basic CSV processing is faster in python than in Java (although the python version uses much more CPU), so I haven't felt the need. (I'm probably limited more by disk bandwidth than CPU, but I've not done the tests required to prove that.)

Julia seems nice, but it seems intent on copying all the questionable design choices of matlab, rather than using ideas from the numpy/kdb+/apl worlds.

I would be shocked if those survey results were statistically different from "the same." The spread from the top (YouTube) to the bottom (Facebook) is 0.11 on a 1-5 scale, with a sample size of 395.

If I assume a std deviation of 1 (which seems low), that's 2-sigma, total, top-to-bottom.

Nope, aside from marching band road trips. Ithaca was too far away, for a PA kid. New Jersey was as far as I managed.

Thanks, that's fantastic.

It's funny, I found it to be very emotionally evocative. I loved my IIci so, back in the day. The tinkering with the background images, recording everyone on the dorm hall making funny noises, replacing the system sounds, making the shutdown sound be HAL, etc.

It definitely helped that campus was pretty much all-Mac, so we all could tweak the same silly noises, and all try to find the joke that sounded silliest when told by the Talking Moose.

I don't have the same connection to my Mac these days. I wonder if it's just me, if the college freshmen have the same experience in 2013, or if it's somehow just an artifact of that particular era in computing.

I use the ipython notebook sometimes, but I almost always use something similar. (In my case, I often use emacs org-mode files to make basic literate program-style documents.)

I keep these things in a dated "Notebook" file, like "Notebook/2013-07", just to hold all my one-offs.

Why do I keep them? Well, often it takes a while for me to set up the particular environment I need. If there's a problem with the calculation for X, I need to load the right data, set up the context for X, run through the individual bits until I see what the issue is.

I save the ipython notebook where I go through all that, and then if I need to go back to the calculation, for example to demonstrate to someone else my example that something has gone wrong, then I can just load up the notebook. If something is supposed to have been fixed, I can load up the notebook and repeat the calculation and see if it works now.

Unlike emacs org-mode, I can easily save the graphs as part of this. ("Look, if I plot the distribution, there's this big extra lump over there. Where's that coming from?")

If I go back to a file, it typically takes me a bit of thought to remember exactly what I was doing there. Once I figure it out again, I add sentence or two of markdown.

It's a useful thing to have around.

Sure. At MS, I used it while working on mortgage prepayment and default models. Before we got the K license, I had been using SAS and R. There, I had been dumping aggregated (or sometimes not) CSV files from the database (Sybase) and running my model fits based on those.

I had a bit of exposure to A+ then, trying to debug bits of the interest rate model subsystem. I hated it. We were mostly using A+ as the wire protocol at that point, and I had endless problems getting the APL fonts set up, and so on. Non-ASCII was a huge black mark.

Once we got the license and expanded the team working on the models, we decided to switch over to using kdb+ for the basic data store. That switch was like going from night to day. I could operate over the entire dataset, rather than just on a subset. I could query things on the fly with real aggregations. Things that were a huge pain (and slow) using SQL were suddenly fast and easy, like "calculate the average default rate grouped by FICO in 25-point buckets, for loans issued after 2005" or "backfill missing data from the first non-missing observation."

The first example can be done in SQL, but it takes a whole lot more typing and is (IMHO) much more error-prone. The last, I'm sure you can do it, but it's a nuisance.

I first got comfortable with kdb+ as a database query language. I read "q for mortals" by Jeff Borror, and we actually had Jeff around for a while for questions.

After that, it we started actually using q to estimate the models. It was great at the data aggregation and was a pretty natural fit. It didn't have anything built-in, but so we wrote a few functions for multinomial logistic regression, and that was good enough. That part didn't have a huge advantage over using something like R. However, the interaction with the large data store was far superior, better enough that it was worth it to write our own routines to do the estimation.

We even used q to run the models in production. That was more of a reach, but it worked pretty well.

So, I liked kdb+'s ability to handle large data sets in a speedy way. It makes a fantastic query language. I liked q's conciseness. It doesn't matter much for actual in-production code, but if you're doing research, every character counts. It seems shallow, but I think it's a real effect. Writing a half-line of q to do a paragraph of SQL is a huge win. I liked q's array operations, but by then I'd already been using Numeric and numpy and such for a decade, so those weren't really new. It was nice to be able to write them easily, as opposed to something like "np.concatenate((foo,bar))", but it wasn't really novel.

Basically, it was kdb+ that sold me on q.

I definitely miss having the Morgan Stanley site license for K at my disposal. I can't quite justify the money to get a license, just for me, so I've been slowly assembling a poor-man's substitute built on numpy, pandas, and hdf5.

K itself isn't as consistent as I'd like (odd corners like the auto-casting arrays, especially dict arrays to tables, and skipping NaNs in sums), but it's a great place to start.

J really needs a table or dictionary data type, or at least the tools to make one yourself.

Or, you know, you could just use Notepad as Notepad. Or a real text editor, or something like Simplenote.

Snark aside, this would be a more interesting post if the author gave some reason why this was a good idea. As far as I can tell, it's strictly worse than a text editor. I mean, it is an interesting hack, but the post doesn't frame it at all.

I suppose I don't find it surprising, because I've used a lot of R, and R can't even optimize the simple case. There, it's for good reason, since you can extract the environment of any function, so once a closure escapes, you have a handle onto its entire containing frame.

I think I've used scheme implementations that have the same "feature", with less of an excuse.

It's a similar "failure of GC" if you create an object with two fields, like "obj={a: giant_value, b: 5}" then had your closure return obj.b. I bet the "a" slot would not be GC'd, even though there's no way to reach it.

If your mental model is that you're holding onto the lexical frame, it's a pretty much the same behavior.

But you're still clearly retaining a reference to the lexical frame, so I'd think the "zeroth-order" assumption would be that you'd be keeping everything in that frame.

It's nice that JavaScript will do enough tree-shaking that it will eliminate the single-closure case, but that's an optimization, not something you should be counting on.

Similarly, I bet if you had a call to "eval" in there, you'd keep the entire frame, but I don't understand the exact semantics of JavaScript's eval enough to really say.

I suppose this boils down to whether your mental model is more "frame-like" or more "upvalue-like". If you think of keeping around the containing frame, it's not at all surprising that you retain the variable. If you think of the closure as a discrete set of upvalues, then it's a surprise.

You might not like it, but if they're not doing it under false pretenses, it isn't phishing. Now, if they pretended to be gmail and asked for your password, that would be phishing.

Note that I'm not saying it's a good practice, but you need to come up with another name for it than "phishing", because that one's already taken.

I'm sure there are many places where other pointer types would be better, but, like I said, I only tend to see shared pointers, usually typecast to something like FooPtr and used indiscriminatly. It's an uphill battle to even use something like a pointer to const.

I've never seen refcounting overhead show up in callgrind, so I think the choice to uniformly use the most general version is OK.

It's hard to talk about philosophy. I liked my philosophy classes in college (one overview, mostly with a Hume/Kant/Wittgenstein/modern analytic bent, and a philosophy of quantum mechanics class), and I like reading Peter Singer, but the older stuff just gets very woolly-headed seeming. It feels like they always try to hide some equivocation under fancy text, starting with "we define X as Y", then jumping through a natural-language use of X to get Z, then assert "Z is Y".

For example, I may just be a heathen, but Nietzsche's whole style seemed to be boldly asserting things, making fun of anyone who thinks otherwise, yet rarely bothering to make an actual argument.

I like the idea that everyone should take a semester's worth philosophy course somewhere along the way, but I don't know if it would do most people any good. An actual rhetoric course, with a focus on logical argument, may do better.

In every C++ project I've worked on, the vast majority of allocations go immediately into a shared_ptr. This article seems to assert that most C and C++ programs stick to malloc/free or auto_ptr-style semantics. This seems to be a contradiction, so I'm confused. I can see it being true for C, but definitely not for C++.

Am I misunderstanding the thrust of this article?

Edit: deleted comment about cycles, since they are discussed a bit at the end.

Interesting. I've heard good things about the Windows Azure program, but I don't think any cloud-based option will work for finance. People generally want to keep their servers and data in-house, for trade secret protection and regulatory purposes.

Do you know whether numerical analysis has improved with .NET? I know Morgan Stanley was evaluating it a few (4? yow) years ago, but there were two main problems:

- There were very few numerical libraries, and those that existed either had a high overhead (compared to, say, calling NAG from C++), or weren't very robust (didn't degrade well in edge cases).

- A windows license for a server machine with > 32 GB of RAM cost an arm and a leg, so server farms got very expensive very quickly. (Oddly, the price of the license depended on the RAM of the machine involved.)

It looks like NAG now has a .NET version, so hopefully things have improved. For a while, it seemed like math in .NET was strictly amateur hour, but I'm sure it's better now. I don't have the strength of will to research if the licensing cost has improved at all.

I think most places enforce the opposite. There's a strain of very-little-space style out there, but I'm not sure where it comes from. Maybe Pike, for all I know: http://www.literateprogramming.com/pikestyle.pdf is certainly rife with it.

Emacs. Aquamacs was just too weird. scratch wasn't in lisp-interaction mode, I had to change too many things to get normal behavior (opening multiple frames vs just buffers, etc.), and so on.

For me, it's not compatible enough. At work we have a few batch scripts that set up the path and set some environment variables, so we can launch compiles/tests/etc with a short command. These work under cmd.exe, but not under powershell. I keep meaning to experiment with powershell, but never quite enough to overcome that initial barrier. (I'm 95% on Linux, so it's not as if improving my windows workflow would make a big difference.)

I've been on LinkedIn for a while, without engaging with it that much. In the early days, if someone sent you an invite to connect, there was an option "I don't know this person," with the implication that they would be somehow censured for spamming someone they didn't know.

That option is long gone, but I don't know when it disappeared. Does anyone know when this happened? If anything, that's a sign of when it jumped the proverbial shark.

I don't think either of those books is very good, so try to borrow before you buy. The best of a bad bunch is still bad.

I never look at Fabozzi's book these days, but I still break out the Guide to MBS for its appendix on mortgage amortization formulas. That's a pretty weak recommendation.

Not really. Lakhbir Hayre's Guide to Mortgage-Backed Securities isn't bad. Fabozzi's MBS book is okay, but a bit of a cut-and-paste job. I don't know of anything that deals with the post-2007 environment. Otherwise, just a combination of statistics (GLM, GAM, NLS, etc.) books, time series books (e.g. Durbin & Koopman, and econometrics books (Greene, etc.)

#3. Definitely #3. I mostly model mortgage prepayments and defaults, the housing market, and do econometrics-style forecasts. That's not in any of those books.

Usenet also didn't have a good way to deal with spam, at least not one that I knew about. It steadily became harder to find the signal in the noise.

It still had the best reader experience. I don't know why modern email clients, for example, don't look more like trn.

Ok, maybe I'm missing the point of this, but aren't phone numbers, by their nature, mostly public information anyway? There's this big fat book that maps name to phone number, and I'm sure there are a few online versions as well.

If you're not trying to make your Twitter handle anonymous, then why not tweet your phone number?

Aside from the performance differences, data.table makes it very easy to do interactive manipulation, at the cost of making it hard to program. Pandas currently goes in the opposite direction.

I'd rather have R/data.table at the prompt and python/pandas in my script, but if you have to err on one side, the python/pandas "low magic" is the side to err on. Pandas does have its own strange corners, though. For example, it seems like it tries hard to stick similar-typed columns into contiguous matrices, which leads to some unexpected casting, and I have no idea what the supposed benefit is over just keeping distinct columns.

One thing that was conspicuous for its absence was the "Law of Demeter." Is this not considered a good practice any more?

It's trickier to use, because blindly applying it can lead to immense classes, but I find it a useful thing to keep in mind.