robertelder

I assume you're just joking around, but to you and the parent comment, I'd be happy to hear any good arguments for avoiding 'useless' cat. Note that I did mentioned 'useless cat' in the article, and there is already a comment thread in this article that contains my opinions on it.

Using head when there's lots of data make sense, but I really don't see any advantage to avoiding useless cat. Useless cat is way faster to type and make additions to. I sort of get the feeling that 'useless cat' is really just a fun copypasta kinda like when people like post "I'd just like to interject for a moment. What you're referring to as Linux, is in fact, GNU/Linux, or as I've recently taken to calling it..."

hah, I knew someone would point that out (which is why I talked about it in the article).

I actually prefer useless cat because when you're prototyping a pipeline it's very awkward to use non-useless cat. You'll probably start off with something like this to observe the content of the file:

    cat something.txt

Using this doesn't work in bash:

    <something.txt

Then, continuing with useless cat to build on it you do

    cat something.txt | grep stuff

Which you can type easily from using 'up' in your terminal. But if you use non-useless cat you have to re-type the entire thing or move the cursor around:

    grep stuff < something.txt

With useless cat, you can keep adding things and check the result:

    cat something.txt | grep stuff | sed 's/"//g'

Or if you need to insert another filter before the last stage like this, you can just press "up" and insert it:

    cat something.txt | grep -v negmatch | grep stuff

I don't think there is any easily-typed equivalent workflow with non-useless cat.

The most significant use case for all things command-line IMHO is automation. Also, I would change that from "command line vs in Python or R" to "command line and Python or R". Build a pipeline like I've discussed in the article, then pipe it into Python or R.

Hi, (I wrote the article). A few people commented noting that I included "Data Science" in the title, but the content doesn't include any statistics or machine learning which is closer to the core definition of 'data science'. I still think the title is appropriate since any kind of low-fidelity data science task you do on some had-hoc data (log files, heaps of text, web pages) is going to start with setting up a processing pipeline that involves these commands. I could have re-named it "An intro to text processing" or "An intro to data processing", but then the people who need to see this content won't associate the title with something they're interested in, so they never benefit from it. The list of commands was chosen specifically with the question "What Linux commands would someone answering data science/business intelligence questions use?" in mind. These commands are also among the list of ones that are usually already installed on every system.

'a relatively new kind of fraud calling "SIM swapping"'

Cool. You've got a fairly impressive following on that tool. I wish more people took an interest in studying these sorts of algorithms. Most people assume that some genius figured out the optimal way to do things years ago, but more often than not if you spent a few weeks focusing on a specific paper/algorithm, you'll come up with ideas for improvement (although it might not always be in the way you initially expected). Sometimes the improvement is just figuring out a better way to explain it and that in itself can lead other people to create breakthroughs.

I think there are still a few improvements that could be done the the Myers algorithms. One specifically, is that I'm pretty sure it's tail recursive. You won't get any big O improvements off of that, but less stack space is still an improvement. Plus better cache locality. I'd work on some demos, but too many things to do these days...

I spent a bit of time learning the Myers Diff algorithm, (I made this page: https://blog.robertelder.org/diff-algorithm/)

Like someone else said, I think the key is that you're not getting the shortest edit script, you're just getting a edit script.

Interestingly, even git's implementation of diff (which uses a variant of the Myers Diff algorithm) doesn't actually find the shortest edit script by default. It has a hard-coded heuristic that just quits early if it finds a 'good enough' solution:

https://github.com/git/git/blob/master/xdiff/xdiffi.c#L143

Same this for unix diff (unless you specify the commandline flag to find shortest):

https://github.com/Distrotech/diffutils/blob/distrotech-diff...

Having said that, if you're still confident that you've made an improvement, you can port the function to python and try it out as a test case in my unit test file that I wrote when I developed my variant of the Myers Diff algorithm:

https://github.com/RobertElderSoftware/roberteldersoftwaredi...

On that topic, I'm pretty sure my variant of the Myers Diff algorithm is actually slightly faster/uses less memory than the original (I'd be happy to be proved wrong). At least, I've had my blog post up for a few years and no one has corrected me, but then again, I think the original paper has a few mistakes/obvious flaws and it still has hundreds of citations and no one has written much about improvements.

I find this city fascinating for some reason. It's something about the fact that it's so harsh, but humans are still able to live there. Here's a YouTube video of it that makes it look almost artistic:

https://www.youtube.com/watch?v=ks9E9XQp_2k

I wonder if OP plans to submit the report for course credit too? It would probably get a pretty good mark.

Here is the instructional video referenced on how to reset the bulb:

https://www.youtube.com/watch?v=1BB6wj6RyKo

It reminds me of the 'cinco' product line:

https://www.youtube.com/watch?v=TSwqnR327fk

https://www.youtube.com/watch?v=ZG8ZKwaC1jY

https://www.youtube.com/watch?v=Y5BZkaWZAAA

I haven't yet. It was actually quite a bit if work to get these videos recorded and document/test the process. I suspect there is tons of low-hanging fruit left in terms of improvements that could be made. Hopefully the steps I documented will be reproducible enough that others can make improvements and create even more impressive demos.

Nope, I didn't do any black level/gamma adjustments. I just used the first values that would produce what looked like a fairly high-quality result with ffmpeg and dcraw. There is probably tons of low hanging fruit to improve these results from someone who really knows what the're doing in terms of image processing.

Hi Everyone,

I do have it already linked in the article, but I should point out that the work I did is based upon the work of (at least) several other key individuals:

A lot of the low-level sensor interaction code is found in:

https://github.com/Hermann-SW/fork-raspiraw

Also, check out his web site with lots of other high speed camera hacking stuff:

https://stamm-wilbrandt.de/en/Raspberry_camera.html

The original fork of raspiraw is (I think) found here:

https://github.com/6by9/raspiraw

And the code for converting RAW images into standard formats uses dcraw:

https://en.wikipedia.org/wiki/Dcraw

I didn't personally write much code to put this together. I would say that my contribution here was mainly to pull together some impressive work done by others that wasn't getting much attention and market it well. Hopefully some of you get ideas and push this even further.

Usually, I would try to wait until evening when the sunlight shines directly into my apartment. Firlming under a 60 watt lamp was still quite dark. I didn't do any experiments with post-processing to reduce flicker, but I know there some advanced video editors do provide such features.

I've watched this series before too, and I can attest to the fact that it re-ignited my childhood dream of building by own hydroelectric power system.

I know this is definitely true for specific 'ham' radio frequencies, but I've never found a clear answer as to whether it's also illegal to use encryption for commercial purposes over other frequencies like 2.4ghz on your own home-brew devices (assuming that you got them tested and licensed). Using a wireless router for commercial purposes obviously isn't illegal, but I'm not sure if 802.11 is some kind of exception?

Might as well save you a click I guess:

[32] But I must explain to you how all this mistaken idea of denouncing of a pleasure and praising pain was born and I will give you a complete account of the system, and expound the actual teachings of the great explorer of the truth, the master-builder of human happiness. No one rejects, dislikes, or avoids pleasure itself, because it is pleasure, but because those who do not know how to pursue pleasure rationally encounter consequences that are extremely painful. Nor again is there anyone who loves or pursues or desires to obtain pain of itself, because it is pain, but occasionally circumstances occur in which toil and pain can procure him some great pleasure. To take a trivial example, which of us ever undertakes laborious physical exercise, except to obtain some advantage from it? But who has any right to find fault with a man who chooses to enjoy a pleasure that has no annoying consequences, or one who avoids a pain that produces no resultant pleasure?

[33] On the other hand, we denounce with righteous indignation and dislike men who are so beguiled and demoralized by the charms of pleasure of the moment, so blinded by desire, that they cannot foresee the pain and trouble that are bound to ensue; and equal blame belongs to those who fail in their duty through weakness of will, which is the same as saying through shrinking from toil and pain. These cases are perfectly simple and easy to distinguish. In a free hour, when our power of choice is untrammeled and when nothing prevents our being able to do what we like best, every pleasure is to be welcomed and every pain avoided. But in certain circumstances and owing to the claims of duty or the obligations of business it will frequently occur that pleasures have to be repudiated and annoyances accepted. The wise man therefore always holds in these matters to this principle of selection: he rejects pleasures to secure other greater pleasures, or else he endures pains to avoid worse.

I think the famous 'lorem ipsum' text (the English translation of it) is very instructive on this topic (see Wikipedia for full translation):

https://en.wikipedia.org/wiki/Lorem_ipsum

I had been considering setting up a Partreon for a while now, and seeing this headline was what pushed me to convert my existing account to a creator one just now:

https://www.patreon.com/RobertElderSoftware

If you've ever seen my blog posts show up here on hn and you'd like to see more of them, then this is one way to make that happen more often :)

Location: Waterloo, Ontario Canada

Remote: Yes, but I can meet in person if you're in the Kitchener/Waterloo area.

Willing to relocate: No.

Technologies: HTML, CSS, Java, C++, C, Python, XML, JSON, apache, tomcat, Mysql, Postgres, Javascript, react.js, embedded systems, compilers, servers, OS dev, Linux, Unix, bash/shell, AWS, PHP, raspberry pi, arduino, ESP8266, git

Anti-Req Technologies (technologies that I don't do): Microsoft products, .NET, C#, node.js, npm, nosql, Mac OS, mobile apps.

Résumé/CV: http://blog.robertelder.org/

Email: robert@robertelder.org

I do contract-based consulting/software development for startups in the Kitchener/Waterloo area. I am looking for non-exclusive part-time engagements. I have previously taught courses in operating systems, algorithms, and web development part time at Conestoga College ~2015-2018. I frequently write blog posts and submit them to a neat site I found called 'Hacker News': https://news.ycombinator.com/from?site=robertelder.org

Most companies generally protect their logo with very strict intellectual property devices. I suspect that a very small number of recipients would have a very big problem with someone else incorporating their logo into third party marketing material. More specifically, they might get the impression that these shirts are already produced somewhere laying in a warehouse to be sold to anyone who wants them without their permission. They might not like that.

If you like that, there is yet another seldom seen feature of c99, the use of '' in an array declarator:

    echo 'int my_func(int i, int arr[*]);' > main.c
    gcc -c -std=c99 main.c
    -- * observe that there are no errors * -- 

See 6.7.5.2 Array declarators for details http://port70.net/~nsz/c/c99/n1256.html

Edit: I can't figure out how to escapse an asterisk, so the above might not read correctly but I tried.