schizoidboy

I'm no expert, but I did a bit of research into this since I live in California, and here's data that supports the parent's assertion: https://radicaloptimism.org/wiki/In_The_News#California_Drou...

It seems to me that the patterns fit within the historical norms for over 100 years.

Similar tool: https://github.com/myplaceonline/posixcube

Is it fair to say that a major theme in The Great Gatsby is the decadence of rich people? If so, I don't understand why so many people seem to like the book, non-ironically, for the parties and the glamour. Am I missing something?

Agreed. I referenced this discussion in that issue and hopefully soon I'll get around to building it (would be wonderful if you could help). Thanks for the pointers; they'll save me some time learning Ansible.

Yeah, I'll add some of the most common parameter expansion/substitution, string functions, array functions, etc. as APIs.

I had a thought yesterday for handling pipeline failures by using set -e in a sub-shell. I still need to test this, but something like:

  (set -e; command1 | grep foo | grep bar | sed ...) || cube_check_return

I want to see a fully fleshed out example for comparison. When I find some time, I plan to take this use case as an example, implement it in all major automation frameworks, and publish the comparison.

> On the other hand, after a few years playing with bash and posix shell -- I'm fairly certain I'd prefer bash, to posix shell

What are some bashisms that you find particularly useful? I wonder if I can just reproduce them in POSIX. So far, I haven't found POSIX too bad, although I don't have much experience making huge and complicated shell scripts. posixcube does support some optional Bashisms already like tab auto-completion and `caller`.

> And shell script is a fickle language -- I'd at least consider using "trap" and/or "set -e"

I didn't like `set -e` because it disallows using "benign" error codes. For example, one of the most commonly used APIs in posixcube is cube_set_file_contents. You pass the file you want to update and the source file. If the file is updated, it returns 0; otherwise, it returns 1. This is commonly used like this:

  if cube_set_file_contents "/etc/npt.conf" "templates/ntp.conf"; then
    cube_service restart ntpd
  fi

I hadn't considered `set -u` and it's part of POSIX. My initial reaction is that it's a nice feature, so thanks for pointing it out, and I might set it.

pipefail is a Bashism, and I'm torn on this. Handling errors in a pipeline is one of the worst parts of POSIX shell scripts. However, using something like pipefail means exiting with an error without a stack. I think it's always useful to have a stack to quickly jump to where the error is in which script. So I still prefer this guideline:

  There are non-standard mechanisms like pipefail and PIPESTATUS, but the standardized
  approach is to run each command separately and check the status. For example:
  cube_app_result1="$(command1 || cube_check_return)" || cube_check_return
  cube_app_result2="$(printf '%s' "${cube_app_result1}" | command2 || cube_check_return)" || cube_check_return

> I agree that ansible is far from perfect - but I'm not convinced posixcube is a great alternative right now.

Please see my reply to another comment in this thread, asking how a simple firewall whitelist configuration would be done. I think this would be a useful example to debate around.

> From a brief look at the code, it does look nicely structured and commented, though!

That's very nice to hear, thanks! This was my first time doing anything hardcore in shell scripting.

> I've seen people here on HN recommending some makefiles for deployment (along with simple shell scripts) -- and I've played a bit with using m4 for templates rather than here-documents -- and it looks like that might work fine. Does mean your base-installs/bootstraps/images will need m4 (or "compile" the templates on the workstation you run your automation from).

I really like that posixcube has 0 requirements, but I'm not philosophically opposed to requirements, so I'm certainly curious what a make-based system would look like. Does make have imperative programming like shell scripts?

> But right now, I feel GNU guix is the right approach: have a small, minimal, vm/container host (right now I'm playing with Ubuntu and LXD/LXC - Xen and KVM also works nicely), and have all images be managed by something like guix.

I'm not aware of that, I'll check it out.

Thanks for the comments.

Let's say I want to whitelist a set of IP addresses in the firewall configuration. Let's assume I know the commands to do this using firewall-cmd. Below is how I would do this with posixcube. Can you give an example with a declarative system of your choice for comparison?

  # Loop through the list of servers passed in
  for cubevar_app_server in ${cubevar_app_servers}; do
    # Get the IP address of the server
    cubevar_app_server_ip="$(dig +short ${cubevar_app_server} || cube_check_return)" || cube_check_return
    
    # Skip ourselves by comparing to our hostname
    if [ "${cubevar_app_server}" != "$(cube_hostname)" ]; then
      if ! firewall-cmd -q --zone=trusted --query-source=${cubevar_app_server_ip} ; then
        cube_echo "Adding ${cubevar_app_server}'s IP ${cubevar_app_server_ip} to trusted whitelist"
        firewall-cmd --zone=trusted --add-source=${cubevar_app_server_ip} || cube_check_return
        firewall-cmd --permanent --zone=trusted --add-source=${cubevar_app_server_ip} || cube_check_return
      fi
    fi
  done

This would be run as follows (where the -o parameter is expanded from ~/.ssh/known_hosts):

  posixcube.sh -h $TARGETSERVER -o cubevar_app_servers=*.mydomain.com -c firewall_whitelist

I agree the ambiguous return code is a major limitation of SSH. I think I'll just avoid any retry logic and treat every failure the same: fail immediately and let the user to decide how to retry. Since posixcube does most of its execution over a single SSH connection (packaging all commands/cubes into a cube_exec.sh which is executed remotely), I think the immediate need for automatic retry on failing to connect over SSH isn't large.

Thanks, I understand your perspective much better.

Regarding the last point, I've always just re-run my last command. If the commands are broken up into separate independent scripts (e.g. install ElasticSearch, install Kibana, etc.), then I might only re-run the script (-c) that was failing to ensure that works, before re-running the whole spec (-z).

> Because its architecture doesn't fit the job. The same situation is with Ansible or Salt being used for maintaining configuration of servers.

> I don't say shell scripts are worse. I say the "push" architecture is worse for configuring machines than with an agent in "pull" mode. You could easily write an agent that works on a cron-like schedule, downloads commands to be run, (verify their origin with a digital signature,) runs them, and reports the results back, all in shell.

I'm still not completely understanding, so let me see if I can rephrase your objection and please correct me if I'm getting it wrong:

A "pull" architecture for configuration management is better than a "push" architecture because updating configuration shouldn't require synchronous human interaction.

If so, then one example of a counterargument is needing to reload/restart a service after updating its configuration file. What if there's an error reloading/restarting the service? Does the agent then email the administrator? And what if the emails fail?

Again, thanks for the objections, they're making me think hard about the design choices.

> Lucky you if your SSH client never failed to connect. How do you think it reports a network failure?

Currently posixcube handles this by checking the return code and reporting the error.

> You see, there are tools that allow you to easily tell between network failure and remote command failure. RPC protocols are an example.

ssh does have some basic differentiation: "ssh exits with the exit status of the remote command or with 255 if an error occurred." Although, of course, 255 might be returned by a remote command (not sure how likely that is), so it's still ambiguous. I guess you're saying that if it's an SSH error, then that's pretty safe to retry and it would be a nice feature, and I agree so I've opened this issue: https://github.com/myplaceonline/posixcube/issues/6

> Now let's get practical: find me such a library.

I guess it would just be a matter of, first, checking for 255, and next, checking the output of the command to see if it's an SSH error for sure?

> We were talking about configuring servers. You basically tell me that I would have to pay for "agentlessness" with tracking state of each and every machine I would possibly want to reconfigure, and remembering to check if the machine was booted to re-issue a reconfiguration command.

So you bring up a good point. On DigitalOcean, for example, their Fedora images use extlinux for booting and I can easily update the Linux command line (e.g. crashkernel), reboot, and it picks it up. Their Ubuntu distribution uses grub, and update-grub works fine, but for some reason, it requires a hard shutdown instead of a reboot to pick up the change. The way I handle this now is here: https://github.com/myplaceonline/myplaceonline_posixcubes/bl...

We echo this instruction to the user. This prints a red-text error message and stops execution. This seems even better than an agent-based execution because with an agent-based architecture, the agent is now gone (hard powered off); whereas, this style of orchestration in the front-end can use a script on top of posixcube.sh that waits a minute, remotely calls DO to start the droplet, and then continues execution (using the same script since it's essentially idempotent). The only other option would be to have a centralized server that orchestrates all of this, which posixcube avoids.

> Sorry, I'm lost here: what re-running? The command didn't get the chance to run in the first place. I don't know where did you get the idea that I propose running the command again if it got through to the remote server and failed there.

See above, I'll need to try it, but I think the ssh error code for "command didn't get through" along with some simple heuristics on the command output /might/ work; I'll investigate in issue #6.

First, I just want to say that I appreciate this conversation and debate. It's making me think hard about certain concepts, so thank you.

> If you ask this question, it means you've probably never configured SSH in bigger environments, because you don't imagine how easy is to accidentally cut off your access.

Many moons ago, I was mucking with sshd configuration, and locked myself out. It was the first and only time I ever had to truly hack - my own system. The system had a Wordpress installation which, at least back then, allowed you to modify the PHP code through the web console. I knew I had gcc on the box, so I found a NULL-dereference root escalation exploit for my kernel version, jammed a bunch of C code into a php system call, and step by step, escalated, and then overwrote my ssh config (all through escaped C code in PHP!). It was fun. So I totally hear ya, updating sshd configuration can be disastrous, and it's certainly required in certain situations. Posixcube is certainly not ready for massive enterprise usage, but I don't see why it couldn't be. For this particular point, it seems that no matter what technology you're using, SSH can always get mucked up, and I see no inherent reason why shell scripts are worse.

> You actually have provided me another important point: how to tell that the command reached its destination server, but failed there? This is vastly different case than when a command didn't get through to the server, so it can be safely re-sent. (And I was talking about the latter case.)

The way this is handled now is that all cube_* APIs check all return codes and if non-zero, they print the full stack and `exit $code` which is passed back through SSH into the client and reported and the script stops. In the latter case of safely re-sending a command that didn't even get to the server, I don't think that's much of an issue here, because posixcube doesn't execute one command at a time (since the overhead of re-establishing the SSH connection would be massive), but actually generates a shell script on the fly (cube_exec.sh) which has all the commands you want to execute, uploads that as well as all of the cubes, and then executes cube_exec.sh on the other side.

> But now you have a queue which needs to be managed and that also can break. This is not an improvement. Setup with a dedicated agent that works in "pull" mode (instead of one that was not designed for this task, but just happened to be already installed) doesn't have the problem that needs solving with queue in the first place.

Are you talking about a use case where an administrator is not monitoring a deployment (i.e. fire and forget)?

Please share if you can, I'd like to take a look.

You're welcome! Hope you might find this useful, and glad to hear feedback.

I agree shell scripting is a very limited language, especially when trying to keep them POSIX compliant.

Perl would play nicely with posixcube. Just do cube_package install perl, and then do with it as you wish. Posixcube is more about handling the things outside of Perl, Python, Go, etc. like distribution, installation, etc.

I like a lot of things about this including, first of all, the website describing it, and the screencast, and easy-to-follow instructions. It doesn't meet my needs including handling of encrypted variable substitution in file templates, but thanks for sharing.

Thanks, great idea. There's already optional Bash programmable tab auto-completion integration which auto-completes hosts from ~/.ssh/known_hosts (the auto-completion is installed using the `-b` option). I've created GitHub issues for integrating DO, Google Cloud, and AWS [1, 2, 3]. I'll start work with DO first since that's what I'm using now.

[1] https://github.com/myplaceonline/posixcube/issues/1 [2] https://github.com/myplaceonline/posixcube/issues/2 [3] https://github.com/myplaceonline/posixcube/issues/3

> I'm not sure what you mean, can you please elaborate on a use case of a task queue?

Answering my own question by re-reading your comment: I'm not familiar with systems automatically re-trying commands. What if there is some fatal error on the server? It seems like this is something that requires a human, not a task queue. Relatedly, if the system has the `caller` built-in, when there's an error, a stack trace is dumped, including the line number of the failing cube for someone to quickly jump to the failing line in the script.

With that said, a retry mechanism could be built-in if it's a big use case.

Yeah, I'm sure Ansible is good and I'm sure I would've liked it as much, or more, than Chef.

> but you have an agent anyway

By agent, I meant something more narrow: a prerequisite that must be installed on a server before it works. In that sense, the agent here is sshd, but that can be presumed to always exist.

> so once you start to need to configure it, you can break all the things at the same time.

Do you mean if the script updates sshd configuration?

> Oh, and you just dropped proper configuration management. Now you need to add a task queue (another thing that can break and overflow and what not) to re-send configuration commands when some server is down

I'm not sure what you mean, can you please elaborate on a use case of a task queue?

Thanks for the comments!

The impetus for this project was that I wanted to move away from Chef because it was costing me an extra administrative server per month. I wanted to try something new, so I didn't bother looking into chef-solo. I wanted something agent-less, and it seemed like Ansible was the latest alternative, but when I looked at its YAML, I felt fatigue at learning yet another domain specific language. I thought, "Hey, why not shell scripts?" (insert jwz RegEx joke here). I checked search engines and Stack Overflow, but I couldn't find much except for FSS which didn't meet my requirements, and I had always wanted to learn more about shell scripting, so I created my own framework called posixcube.sh (open source, MIT license): https://github.com/myplaceonline/posixcube. There were a few requirements that I had from my Chef cookbooks:

  * Idempotent file templating with variable substitution
  * Plain-text and encrypted variable files
  * Logical packaging of recipes
  * Roles and a way to check for roles in scripts
  * Repeatable sets of actions for certain types of servers

So beyond the basics in posixcube.sh of uploading itself to the remote host and providing a consistent API, the above requirements were solved with: cube_set_file_contents, gpg for encryption, "cubes" for logical packaging (although loose scripts and straight commands are also supported), -r ROLE, and cubespecs.ini, respectively.

After having migrated an entire Ruby on Rails architecture (https://github.com/myplaceonline/myplaceonline_posixcubes) to posixcube, I thought it went very well, although I certainly see some of the downsides of shell scripting such as checking for error codes in a pipeline, strange HEREDOC interpolation, etc. I also see some of the benefits of complete idempotency, but I didn't find the additional procedural checks to emulate idempotency (outside of file templates) that difficult. For the simplest cube example from the previous link which configures an NFS server, see https://github.com/myplaceonline/myplaceonline_posixcubes/bl... and for a more complex nginx+Rails cube, see https://github.com/myplaceonline/myplaceonline_posixcubes/bl...

You can test it as simply as:

  git clone https://github.com/myplaceonline/posixcube
  cd posixcube
  ./posixcube.sh -h $SERVER "cube_echo Hello World"

For more complex examples, see the usage. I'm open to any feedback and any philosophical debate on this approach versus Ansible, Docker, etc.

I guess this is true of almost all the big companies; nevertheless, it's interesting to see it written explicitly:

"tie all of our products together, so we further lock customers into our ecosystem"

I created an app for this: https://myplaceonline.com/

This is one of my favorite comments ever. It captures so much. I often find myself in an infinite loop of "maybe X, but on the other hand Y" and periodically I just need to `kill -9`... until the next time curiosity forks.

A good friend died a few years ago wingsuit BASE jumping in the alps. It hit me hard. He was incredibly smart. He was one of the top guys in the sport. He was careful and deeply researched and understood the physics of wingsuits. Thus why I think the following quote from the article is dangerous because it might give the impression to certain people that they will outsmart the complexities:

"The simple truth is that wingsuit BASE jumpers don’t know what they are getting into, don’t know how to practice the sport safely, and don’t even know enough to know how little they know."

I really appreciate (and am very fascinated by) your extreme honesty in this thread.

I've read Camus twice. The first time I read The Stranger, I took away meaninglessness, in the sense that existentialism seemed true.

The second time, I nearly threw up (no, I wasn't reading Nausea :-D), but in the sense that I don't think I've ever felt anything was so absolutely evil and wrong, and existentialism false.

I can't explain this in a philosophical way, and it may be a delusional and biased opinion given my life experiences between the two readings, but I think observing intuitions does hint at universal meaning. Intuitions might be intellectual observations of objective truths no different, in principle, from observations of atoms, or reason itself.

It would be fascinating to work towards reducing this problem. Any thoughts on how to get into this field or how software could be applied?

I think it'd be interesting to have an option for either type. I find a lot of commentary is suppressed because people don't want other people to see their thoughts, whereas with email, they can just reply directly with thoughts. However, I think you're right, that sometimes it's useful to have a group discussion. I'm thinking that would be a link at the bottom of the email that goes to a webapp that allows a more FB-like experience that people can opt-into.