Hacker News Reader

Comment by PeCaN | Hacker News Reader

I find it massively useful in practice, even for mundane tasks.

For example, the other day I was trying to search the columns of a very large SQL Server table¹ but also needed information about the column type.

  …\dbo.site> ls Columns | select Name, DataType, {$_.Properties['Length'].Value} | where {$_.Name -like '*int*'}

  Name            DataType $_.Properties['Length'].Value
  ----            -------- -----------------------------
  internal_end    datetime                             8
  internal_start  datetime                             8
  interval_code   varchar                              5
  prod_intv_start int                                  4
  prod_intv_stop  int                                  4
  working_int     decimal                              5

Quick, easy. In a *nix shell that would be very tricky.

¹ SQL Server installs a Powershell extension so that you can essentially treat databases like a fancy object filesystem. It's wicked cool actually.

View on HN

How would that be hard in a *nix shell?

    mysql > SELECT  column_name, data_type, numeric_precision FROM information_schema.columns WHERE table_schema = ? AND table_name = ? AND column_name LIKE "%int%"

Imagine for a second that the DB can't do it and you need to filter the STDOUT output.

Okay, I've imagined it. It does seem terrible.

Fortunately, that's not a world any of us live in.

OK, but now imagine it's not a database, because there are programs that do act this way.

And those programs usually emit proper CSV. You could use regex to match on what you want, but AWK was literally built for this. It's fantastic, it works great, and if you don't use that tool that was built to solve your problem, that's your own lookout.

I don't think awk is actually capable of parsing CSVs in all cases.

It doesn't parse escapes, IIRC, but those are rarely emitted.

Er, it's a world all of us live in. Say, for example, ‘ps’ or ‘ls -1l’.

Then you end up using awk, which is rather like a half-assed version of powershell to work around shortcomings with the unix everything-is-a-string design.

Parent comment was talking about parsing the raw output from a database in stdout. That's what I was responding to, and I'm not wrong.

AWK isn't a workaround. It's a tool for processing tabular data, in a textual format. The fact that it can solve so many problems is actually a strength of the EIAS design.

That doesn't sound hard at all. AWK, anyone?

Writing awk to do it is definitely more work than calling obj.Property though.

  $mysql_cmd|awk -F\| '$2 == "what you want the second row to match" {print $0}'

Not that much more work. And you can do better if you can coax mysql to emit proper DSV.

That kind of stuff adds up fast in an interactive shell.

...Which is why you write a script to do it if it comes up often.

Extendable tooling!

The reason I chose that particular example was not for the database integration (which is very cool) but for the fact that it's actually something I did.

Point is, imagine trying to grep anything tabular in bash. Hypothetically, the output of ls -la for files with execute permission that have been modified in the last day. Or, more realistically, anything involving ps.

Imagine trying? Sir/Ma'am, I have done so.

When boxed in-between the fearsome foursome--`bash`, `sed`, `awk`, and `emacs`--every problem involving tabular data (2D) is trivial. Hm, I'll give a shout-out to `sort` and `uniq`, too! When you go 3D, you might need to bring in `join`.

But anyway, you'd use

    find ~/bin /usr/bin -mtime 1 -executable -type f -ls

..to provide `ls -la` output for every executable, ordinary files (not directories, which are often +x) modified in the last 24 hours, under the directories /usr/bin or the bin/ under your home directory.

Hey, don't forget cut and xargs!

    *gulp* I've heard people say that, but I haven't delved into them!!  :-S

cut doesn't match awk, but it's handy for the little stuff. And xargs is the most usefull thing ever.

And paste for tasks requiring the opposite of cut, then there's tr, the lightweight sed. And jq for structured queries on JSON. But, all these are built in to poweshell, essentially.

No they're not. PS can't process text all that well...

Most users do not find awk trivial. Most people basically use it for {print $colnum} when simple columns are available.

PowerShell's integrated query syntax that you can patch into various things is a lot more consistent and reasonable. It is difficult to imagine volunteering people into a world of awk, sed and cut with good intentions in one's heart.

I didn't call awk trivial, I meant to say that the prudent application of powerful tools makes quick work of otherwise intractable problems.

Forgive my being frank, please, but I think that given the context of this conversation, it's appropriate: Most users don't HAVE non-trivial problems!

There aren't features in sed nor awk, nor regular expressions in general that weren't put there for some specific reason in the past.

When you wield your shell, well, you wield a tool honed through the ages by people who think like you! Do so with pride! :)

I have no time for tool worship, language worship, and other sentiment like that. These are tools, not people. I'll use the best I can buy, borrow, or make until I see another better one comes along.

A corporation has no incentive to give you (let you keep forever) the best possible tools.

If that is true, then it is equally true of the authors of shells.

But of course, it's not true. Many great and lasting tools have been founded out of corporate work in a bid to improve mindshare and ease recruiting by increasing the esteem of corporate engineering efforts. Many of the things we think of as "open source" and "free" were built subsidized either directly or indirectly by capitalistic interest.

But I'm not concerned about keeping it. Ossifying my process so that I can give myself the luxury of time of from learning is not something I'm prepared to do, because I think it will make me soft and less capable as an engineer. I regularly rotate my editors and learn new languages because I don't want to become like the sad specters of the previous generation of software engineers I see now, starched shirts and ties and a fanatical devotion to the dated technologies they stomach in order to collect huge consulting fees from corporations.

I've switched off Fish to Powershell as my primary shell for awhile. It's interesting.

I think this speaks to the difference in mindset I've seen a lot in this thread. It basically comes down to people who have gained and honed experience with these tools vs. people that haven't. Yes, for someone with a lot of experience with these tools something like Powershell is redundant... for them.

Some people just want to get shit done because this is just a footnote in a task that's a small part of a project. Utilizing gained proficiency in several tools is cool but if you have a choice between doing that and just taking advantage of a simple tool that streamlines things and lets you move on to your next task quickly the correct choice is obvious.

It could be that these tools are better and may be more valuable in some situations but asking someone to spend time gaining proficiency or mastery when another path is available is just premature optimization.

Pretty much all databases will support "describe tablename" to do exactly that.

> Quick, easy. In a *nix shell that would be very tricky.

You'd probably need to make an explicit SQL query since you wouldn't have the database integration.

Well, yeah, but I mean the concept (grepping in tabular data).

If, hypothetically, bash had database integration, ls Columns would probably return something like

  foo	varchar	30	NOT NULL
  bar	decimal	 8
  baz	int	 4

Which would be a royal PITA to search for anything in—if I were to search for ‘int’ I'd get all int columns and any columns with int in the name….

I could've used an example involving process lists, but I didn't want to trigger PTSD in anyone who's had to grep the output of ‘ps’ for anything non-trivial. (Plus, the database thing was something I actually did a few days ago.)

But thats why there are tools like awk.

awk is rather similar in many ways to powershell, actually, but untyped. I agree most of what powershell does you can do with awk, but it's often less pleasant than powershell (in my experience).

ps aggregates /proc info for human viewing. Why grep the formatted output rather than the source data?

Maybe on Linux. OpenBSD doesn't even have procfs, FreeBSD deprecated it, DragonFlyBSD doesn't mount it by default.

/proc is a race condition waiting to happen, so I can't really say I recommend using that.

Reply on news.ycombinator.com