As in, I like to get my experiment setup (usually distributed and many different components interacting with each other) to a point where one command resets all components, starts them in screen processes on all of the machines with the appropriate timing and setup commands, runs the experiment(s), moves the results between machines, generates intermediate results and exports publication ready plots to the right folder.
Upside: once it's ready, iterating on the research part of the experiment is great. No need to focus on anything else any more, just the actual research problem, not a single unnecessary click to start something (even 2 clicks become irritating when you do them hundreds of times). Need another ablation study/explore another parameter/idea? Just change a flag/line/function, kick off once, and have the plots the next day. No fiddling around.
Downside: full orchestration takes very long initially, but a bit into my research career I now have tons of utilities for all of this. It also has made me much better at command line and general setup nonsense.
My components arent strictly microservices (a mixture of open source components and handwritten tools) and they interact in all sorts o fprotocols with each other (importing jsons, csv, GRPC, HTTP), but I essentially treat the configuration flags as their API, so there are no implicit configurations that I could forget about. The rest is just naming things well, e.g. descriptive names for experiment result files etc.
Initially I thought everyone is doing that, but from talking to PhDs in other domains I noticed that there is a strong bias towards people working in any kind of complex distributed setting having these pipelines.
My friends who devise ML models and just test them on datasets on their laptop never had a real need to get a pipeline in place because they never felt the pain points of setting up large distributed experiments.
Do you or the parent or others take steps to abstract away the distributed computing steps so that others may run the pipelines in their distributed computing environments? More specifically, I use Condor (http://research.cs.wisc.edu/htcondor/) but other batch systems like PBS are also popular. Ideally my pipeline would support both systems (and many others).
However, I also have in mind a pipeline of scripts where one script may be a prerequisite to the other. Condor has some nice abstractions for this by organizing the scripts/jobs as a directed acyclic graph according to their dependencies. I was thinking other batch systems might support this as well. But some of my challenge comes in learning how each batch system would run these DAGs. Each one will have some commands to launch jobs, to wait for a job to finish before running some other job, to check if jobs failed, to rerun failed jobs in the case of hardware failure, etc.
It seems like the DAG representation would contain enough detail for any batch system but there may be some nuances. For example, I tend to think of these jobs as a command to run, the arguments to give that command, and somewhere to put stdout and stderr. But Condor also will report information about the job execution in some other log files. Cases like this illustrate where my DAG representation (or at least the data tracked in nodes) might break down, but I haven't used these other systems like PBS enough to know for sure.
Ultimately though, I rely on some common environment (such as a particular binary existing in PATH) that lives outside the build system and would need to be recreated by whoever who would like to reproduce the calculations. I never looked into abstracting that away with something like Docker.
(I plan to document the build system once it's at least roughly stable and I finish my Phd.)
Classic HN followup (at least I hope): What's currently getting in your way or annoying? What problem would you currently like to just disappear?
When talking about reproducibiluty in science, it's usually about the availability of original data to verify that the original conclusions were correct, but one level higher there's also computational reproducibility with its own challenges (freezing the original environment of the experiment).
So-called orphan repositories (i.e. content doesn't fit into any other bucket) like Zenodo welcome your articles, datasets and code.
I also like to say that the final analysis was automated. It was done entirely in Mathemtica notebooks that talk to a data-processing API, and can be re-ran whenever. The notebooks are getting released along with the journal article for the sake of transparency and reprodibility.
(Also, I automated my SSL cert renewal ;))
I was at university, and Farmville was all the rage on Facebook. My girlfriend wanted me to play because it'd mean she'd be able to trade stuff with me or something (I forget why exactly), and I eventually caved in.
After ten minutes of playing it, I was bored. I couldn't really judge people that would click plants hundreds of times, several times a day, though, because I played World of Warcraft. It was just a more interesting type of grinding...
I figured out that in order to grind through the game most efficiently, I'd need to plant Tomatoes every two hours, so I wrote a bot that would:
1. Spin up a VM.
2. Open the browser to Farmville.
3. Open up an automated clicking application I had written that worked on Flash.
4. Find the outermost vegetable patch.
5. Click in a 20x20 grid (or however big the whole area was).
6. Replant, and close.
I didn't tell my girlfriend about the bot, and I'd turn it off when I went to visit her, so she was shocked when she went on my farm to see that I was a higher level than her. I'd jokingly feign ignorance, saying that I was just playing it like her, until one day when I had left the script running and she saw my farm picking itself while I was studying.
I've got a script that automatically rips, converts and stitches together audiobooks from the library so that I can play them on my phone. It just beeps periodically to tell me to put the next CD in.
I also had a batch job that downloaded Doonesbury cartoons (including some delay logic so I wasn't hammering the server) and built a linked series of html pages by year and month. I've ported it to a couple of other webcomics so that I can binge read.
I also write a lot of LaTeX macros, doing things like automatically import and format code from a github gist into lecture notes (something like \includegist{C,<path/to/gist>), or autogenerate pretty PDF'd marks summaries for students from my home-rolled marks. database.
Another thing I like is building little toys to demonstrate things for students, like a Mathematica page that calculated the convergence rate and error for the trapezoidal rule (numerical integration) with some pretty diagrams.
I once wrote a bunch of lisp code to help with crypto puzzles (the ones that use a substitution code, and you try to figure out the original text). The code did things like identifying letter, digraph and trigraph frequencies, allowed you to test substitutions, etc.
As developers, we tend to focus on these big integrated projects. But one of the biggest advantages that people who can code have is the ability to quickly get a general purpose computer to assist with individual tasks. I write an awful lot of code that only gets run a handful of times, yet some of those projects were the most pleasure I've ever had writing code.
Maybe this can save you some trouble:
I spend many months doing repetitive tasks. And than I realize I should automate them, and proceed to spend hours coming up with scripts/tools to automate them.
Happens way too often...
Once you reach that point, it then becomes a matter of trading-off how much time/money/effort it will take to automate the task against what benefit you get in return.
https://en.wikipedia.org/wiki/Ski_rental_problem
[...] The ski rental problem is the name given to a class of problems in which there is a choice between continuing to pay a repeating cost or paying a one-time cost which eliminates or reduces the repeating cost. [...]
You can use this calculator to estimate how much time is worth investing in automating a particular task =)
So far this has not resulted in us buying a house and the hours that went into the project would have probably long paid for a good real estate agent :)
(1) Funda.nl is a real estate website that more or less has a monopoly in The Netherlands.
- Adding tasks to my todolist client from every app I use(including my bookmarking service when I bookmark with specific tags)
- Changing terminal colours based on time of the day(lower brightness in the evenings and hence dark colours, too much sunlight in the mornings and hence solarized themes)
- Automatically message people who message me based on priority(parents immediately/girlfriend a longer buffer).
- Filters on said messages incase a few require my intervention
- Phone alerts on specific emails
- Waiting for a server which you were working with to recover from a 503(happens often in dev environments) and you are tired of checking every 5 seconds: Ping scripts which message my phone while I go play in the rec area.
- Disable my phone charging when it nears 95% (I'm an android dev and hate that my phone is always charging)
- Scraping websites for specific information and making my laptop ping when the scenario succeeds(I dont like continuously refreshing a page)
I dont think several of these count as automation as opposed to just some script work. But I prefer reducing keystrokes as much as possible for things which are fixed.
Relevant to this discussion:Excerpt from the github page
>OK, so, our build engineer has left for another company. The dude was literally living inside the terminal. You know, that type of a guy who loves Vim, creates diagrams in Dot and writes wiki-posts in Markdown... If something - anything - requires more than 90 seconds of his time, he writes a script to automate that.
I'm curious what these automatic messages say. Are you talking about something like an answering machine message? "I'm at home but my phone hasn't moved in 20 minutes so I'm probably in the shower"?
Charging to 80% and keeping it there by just letting the charger supply the operating current would probably be best if phones supported it.
Any hints appreciated! I'm also experimenting with Telegram, which I think will also solve my issue.
I don't want to be that guy, but a music collection made of probably repeatedly lossy encoded Youtube rips sounds as if it doesn't deserve the name. Not to mention the whole piracy angle - I'm not trying to claim moral authority, but music is DRM-free and reasonably priced these days. There are hardly any arguments for piracy left.
Definitely sounds like you want to be that guy.
It gets a little repetitive and boring at times but I'm able to save so much time and energy this way to focus on what's important to me.
For work clothes, I have 5 decent combos which I repeat M-F. Doesn't bother me to repeat. Helps that I'm a dev so I'm always Feynman-ing myself away from unnecessary meetings and face time with business side clients.
For commute routes, I watched my phone's gps for a few weeks, tried different routes, seeded new ideas for routes and detours by brute forcing options into the backend and now I can consistently get home within 50 mins of leaving work which includes driving down a few floors of parking at 10-15 mph. I used to take anywhere between 45-90 minutes before. Plus my mpg is usually at a respectable 26-28 mpg for a midsize sedan.
Hope this helps.
Here's a <a href="http://i.imgur.com/VNJ08Ra.png">screenshot</a> of the thing I made in Flask to keep track. Orange needs to be washed next time I wear it, red needs to be washed now.
Having five sets of the same clothes certainly makes life easier. And when I finally do change my attire the office gets a laugh out of the fact I actually went into a clothes shop.
One caveat, though - if you do this, commit to it. Don't do what I did and keep one or two pairs of non-conforming socks just because I liked them. Because now they just recirculate through my sock drawer, gumming up my otherwise perfect system. :P
When you don't have to spend mental energy on it, it's automated. I think that's the real benefit of Steve Jobs's standard outfit: don't have to choose, don't have to shop, just wear clothes and get on with the interesting stuff.
When you're young, you have all these things to worry about - should you go there, what about your mother. And you worry, and try to decide, but then something else comes up. It's much easier to just plain decide. Never mind - nothing is going to change your mind. I did that once when I was a student at MIT. I got sick and tired of having to decide what kind of dessert I was going to have at the restaurant, so I decided it would always be chocolate ice cream, and never worried about it again - I had the solution to that problem.
Whenever I'm at a restaurant, or ordering food for whatever reason, I will check the menu for an item that prominently contains mushrooms. If such an item exists (and it doesn't sound awful or ridiculously expensive), then that's what I'm getting. Decision made.
I started doing this mainly because I like mushrooms, enough that it felt like a really easy way to hand off decision making. It's also good in that, unlike the above, I don't get just the exact same thing every time (although mushroom burgers are starting to seem pretty same-y).
Also reduces the odds that you'll be looking at someone else's plate thinking "I wish I'd have ordered that instead" :)
I self publish graphic novels. I have a script that runs on a directory full of page files and outputs a CSV in the format InDesign expects. I wrote it after manually editing a CSV and leaving a page out, and not noticing that until I had an advance copy in my hands and 400 more waiting to be shipped from the printer. That was an expensive learning experience.
I like to rotate my monitor portrait mode sometimes, but hate trying to rotate the Wacom tablet's settings as well. So I have a script that does this all in one go. It used to try to keep track of separate desktop backgrounds for landscape and portrait mode, but this stopped working right, so I took that part out.
I have a bunch of LIFX bulbs in my apartment. The one near the foyer changes color based on the rain forecast and the current temperature, to give me an idea of how to dress when going out, thanks to a little Python script I keep running on my computer. Someday I'll move it to the Raspberry Pi sitting in a drawer.
I recently built a Twitter bot that tweets a random card from the Tarot deck I drew. I've been trying to extend it to talk to Mastodon as well but have been getting "request too large" errors from the API when trying to send the images. Someday I'll spin up a private Mastodon instance and figure out what's going on. Maybe. Until then it sits on a free Heroku account, tweeting a card and an image of its text about once a day.
And does building a custom Wordpress theme that lets me post individual pages of my comics, and show them a whole chapter at a time, count as "automation"? It sure has saved me a lot of hassle.
I must say your coding achievements are impressive (great utility). Any other skills you haven't mentioned yet? :)
There's other art stuff lurking around my site, too: http://egypt.urnash.com
You are a programmer, and a woodworker.
In my head, "carpenter" just has a different connotation. But it's not very clear-cut. I used to DJ a lot at parties during my university years, was I "a DJ"? I prepare dinner for 8-10 guests regularly, am I "a cook"?
It kind of depends, I guess. I've drawn a large number of cartoons in my life, but (with few exceptions) never sold them for money. I wouldn't call myself a cartoonist (mainly because I don't do it as much recently), but I would definitely not call myself "not a cartoonist" :) That would be selling myself short :)
I'm not a programmer professionally either. But then, I do have a CS degree, and I still write code almost daily. I'm a programmer.
I also generally don't feel that the fact if you do something professionally (for money) should really count as "what you do" (identity, in a sense), as this comic illustrates: http://i.imgur.com/MNJzpqL.jpg (I didn't draw this, btw)
A programmer is someone who can program a computer, and chooses to call themselves a programmer. If absolutely nobody is willing to pay them to do so then maybe they're lying when they call themselves one.
If you ask me what I do, I'm not gonna say "I program computers", I'm gonna say "I draw comics and stuff". Because that's what I spend my work life doing. And occasionally I need to do some tedious task, and find a way to automate it, because I spent enough time in my youth fooling around with programming that I'm not afraid to get my hands dirty now and then.
Like, you can be a swimmer, or you can be a professional swimmer. You have to be a painter a long time before you sell your stuff.
Money is as much a corrupting influence as it is a validator of talent and a correlate to time spent.
- Anything related to infra (I do Azure, so I write Azure templates to deploy everything, even PaaS/FaaS stuff)
- Linux provisioning (cloud-init, Ansible, and a Makefile to tailor/deploy my dotfiles on new systems)
- Mail filing (I have the usual sets of rules, plus a few extra to bundle together related e-mails on a topic and re-file as needed)
- Posting links to my blog (with screenshots) using Workflow on iOS
- Sending SMS from my Watch to the local public transport info number to get up-to-the minute bus schedules for some pre-defined locations (also using Workflow)
- Deploying my apps on Linux (I wrote a mini Heroku-like PaaS for that - https://github.com/rcarmo/piku)
- Searching for papers/PDFs on specific topics (built a Python wrapper for arxiv/Google/others that goes and fetches the top 5 matches across them and files them on Dropbox)
- Converting conference videos to podcasts (typically youtube-dl and a Python loop with ffmpeg, plus a private RSS feed for Overcast)
Every day/week I add something new.
(edit: line breaks)
Any chance you would post your code? This looks interesting, especially if it integrates sci-hub/libgen
I for one would be interested in reading more about how you set this up!
Except they don't take care to send one email per person.
I wrote a bot to reserve hotel rooms a year in advance for a national park in the US.
It was so difficult to book. After couple days of failed attempts to reserve my desired dates, and after staying up late into the night one day, I went ahead and wrote a bot to automate the task of checking for availability and then completing the checkout process once available.
And... it worked.
Not trying to be a smart ass, just interested :)
press 1 to book for this year
press 2 to book for next year.
Ha. For a top-ten most visited park in the world? Walk-up permits are your friend. No experience at Yosemite but I got some backcountry permits at Grand Canyon, it helps if it's not peak season, I think people line up at 4am for those. Worst case you are posting up in a random field or Walmart parking lot for the night.
I also don't have the foresight to book that advance. I've settled with staying offsite and just driving. Not the same experience but good enough compromise I guess.
Cad -> robot code compiler is built on top of pythonocc
Around 3 years ago, we started to get a lot of customers for our VoIP tunneling solution, mostly from UAE. Most of these were unfriendly customers abusing our support, so I started to implement a CRM to track "support points". I spend a half year to develop this solution (with lots of other functionality such as service monitoring) and when I finished, there was no any demand for the VoIP tunneling solution anymore :)
This is how I wasted half year instead to focus to solutions relevant for our business.
Thanks good, we started to have new customers again since last year and actually my CRM/support point tracking software is very useful now, but I still don't think that it worth’s 6 months’ time investment.
Conclusion: focus on your main business and don’t spend too much time with automation and other helper software (or hire somebody to do it if your business is big enough)
So I created scrappers for 3 websites + 1 facebook group. It simply looks for apartments with my specifications and notify me when a new one comes up.
I can say, I successfully found an apartment. The whole process usually takes at least 3 months, I did it in 1.
For other bills, I got all but one to put me on "budget billing" (same amount each month, so Fidelity just sends them a check for that amount without seeing the bill). For Windstream, which varies by a dollar or two each month, I just send them an amount on the upper end and then let a credit accrue. Both of these require an update maybe once a year or so.
Windstream is a bit funny - I don't know why they can't pick a number and stick to it. Also, they apparently raised my "guaranteed price for life" a couple of times and didn't notify me until ~8 months later when they were threatening to disconnect my service for being more than a month behind. (They had turned off paper billing on my account but didn't actually enable e-billing - service still worked so I didn't even think about it. We eventually got it straightened out, but Windstream is ... special.)
Beyond that, I made a bot that automatically withdrew Elance earnings to my bank account (that got me banned for a week or so when I posted it to their forum).
I made another bot that bought and sold bitcoins and litecoins and such. It was moderately profitable until my exchange (criptsy) got hacked and lost all of my float (worth ~$60 USD at the time.)
I connected an Arduino IR blaster to my TV to make it automatically turn on my sound bar (the TV would turn it off, but not on?!) - http://www.nfriedly.com/techblog/2015/01/samsung-tv-turn-on-...
Oh, and of course, code tests and deployment. Nearly every git commit I make gets a ton of tests, and for most projects, each tag gets an automated deployment to to npm or bluemix or wherever.
What kind of bills? Like grocery bills? or recurring bills?
Aren't you using direct debits/standing orders/credit cards for those?
My wife handles our groceries and such. We actually have two separate accounts: "mine" gets the paycheck, pays the bills, and transfers money to "hers" twice a month.
We also have a third "giving" account that gets a bit of money set aside to give to or buy things for other people. This can be anything from a car repair to just buying someone lunch.
Direct debit is fine if you trust the other side not to screw up and charge too much, leak your credentials, etc, but I've seen exactly that happen several times. You're basically giving them to withdraw as much as they want from your bank account.
The DD is then cancelled and the company can argue with me, sure, but I get my money back.
I've had it happen to me once and a family member once where we each had to close a bank account to make someone stop taking out money. Both were a while ago, so it may have gotten better in recent years, but it burned my trust in the system.
Woah - just for your information, there is no credential sharing required here.
But, I was mainly referring to the account and routing number. Anyone with those two numbers and access to the ACH system can drain your account. I know one person who wrote a check to a bill collector, then the bill collector entered their numbers into the their ACH system and started taking regular withdrawals from the account without the owners permission. The owner had to close the account to make it stop.
That one was like 15 years ago - I think the laws around debt collections are better today, but the ACH system is still fundamentally the same system.
Over here nearly everything recurring is done with direct debit, in which you authorize corporations to directly and automatically withdraw funds from your bank account, and it's a required feature if you want to connect to the European payment system. (don't worry, you can charge it back in case of errors or malice)
The corporations over there must be more trustworthy than the ones we have over here. I'd never agree to such a thing.
If its >57 days but <13 months you can call the bank and they do it for you.
In Dutch we have a verb for it: storneren. I think it's best translated as 'to write back'. But I don't know a direct translation.
The three shittiest parts of my job every week are:
- Approving timesheets
- Entering in my timesheets
- Entering in my expense reports
I've written a script that goes in using a phantom.js script, and automates the submission of my timesheet on Friday afternoon at 3:00 +/- 15minutes. It now takes into account travel time, Holidays, and approving time if I have time approvals due.
Same holds true for submitting expense reports in Oracle. I upload the receipt to Expensify, and as long as it's tagged properly in Expensify, it'll automatically generate the correct expense report in Oracle for the proper project based on the receipts in Expensify. This saves me, on average, about 6 hours a month.
Luckily I do not pass by those stadiums on my way to work.
A couple successful scraper projects: 1. I automated buying an iPhone 6 when it came out at the time with a scraper + Twilio for SMS notifications. 2. I setup a scraper to alert me when certain blue-ray titles were available to reserve at the library. 3. I found a site that posted full NBA replays the day after a game. I set up a scraper to on their RSS feed for a couple teams and uploaded the videos to S3.
I've also automated notifications for when my sump pump gets flipped off. My sump pump is connected to a ground fault outlet which can rarely get flipped. I plugged in a spare raspberry pi to the same outlet and have my synology ping the IP periodically. If the synology can't find the pi it sends me a sms via twilio.
I wrote a simple lua script for freeswitch that dials the line, follows the prompts, and plays the person's greeting to the PBX. Of course, one day, the damn PBX will be replaced by freeswitch.
I have written a Python script that builds a HTML out of all chapters of a given fan fiction and then calls Calibre to convert it to MOBI for my Kindle.
Unfortunately, my life doesn't have too many automatable aspects... (I am a math researcher.)
Even if the task happens infrequently and the script takes longer than the task, automating it is worth the investment: - It prevents having to remember or re-discover how to handle the task next time. - It ensures the task is handled consistently. - It prevents potential manual errors.
For example, on the financial side, my company runs bank accounts in five countries, each with different GST/VAT taxes. Over time, I've developed scripts that grab the mid-month exchange rates that our Internal Revenue service requires to be used; crunches downloaded bank transaction data into categories (including tax inclusion or not); and exports it all into a huge Google spreadsheet. This provides global and country balance sheets and profit and loss, and when tax reporting time comes for each country, a tab on the spreadsheet provides all the figures so filling returns is a five minute process. Occasionally the scripts will flag an unrecognised transaction, and rather than manually correcting this in the spreadsheet, I'll add a rule to the script so it is recognised next time.
Cumulatively this probably took several tens of hours to code, but it means we don't need to employ an accounts clerk. It takes about fifteen minutes a month to download the bank data (manually - oh how I wish banks had APIs) and run the scripts. Our accountant loves this - the spreadsheet is shared with him, he can check our formulae or add other metrics, and he prepares our annual report an order of magnitude faster than any of his other clients.
At my first job, part of my work (next to junior dev) was to deploy EARs on Websphere. I automated it so that people just had to drop it on a shared folder and I'd just take a look if it failed to install automatically.
I wrote a command-line tool to search and download subtitles https://github.com/patrickdessalle/periscope
I made a browser plugin to compare the price of the European Amazon and a few other websites (it grew to more countries and websites) http://www.shoptimate.com
And now I'm working on a tool that regularly checks if some of my content is getting adblocked because it's something I periodically do by hand http://www.blockedby.com
In the end, automating things can take more time than actually doing it. But if it's used by others and saves them time as well, it's gratifying.
1. Need action right now. I answer them and archive.
2. Need action later. I open a TODO and archive. If I need them later, I can search for them.
3. Do not need action. These are already archived. I can search through them, if I need anything.
Another thing is newsletters. Every two to four weeks I take the time reading them. Invoices are put away (amazon, ebay, electrictity, etc.).
Well I think the biggest win for me is, that mails in my inbox are typically your category 1 and I can react right away. I do not have to spare another thought on category 2 or 3. I can check my mails twice a day and it's enough. Really helps focusing on my other work.
Back when I used GMail I had some of those rules as filters, but the big advantage of imapfilter on the Pi is: I can add rules pretty easy with a git commit and add new Inboxes if needed.
EDIT: As I mentioned my email filter runs only between 8 and 8, which is really nice, because I do not see any new mails in the evening. This really frees my mind. If the house is burning down, don't write me a mail and give me a call.
Sooo slooowly that the homepage still has stock cats&dogs images. The most upsetting thing is that I've got more than one person telling me "I like the homepage". My mental reaction was "wtf!?". </rant>
Anyway, I still don't tweet much, but I'm getting there.
For example, I had this idea for a travel startup for a very, very long time and I decided to build it on Wordpress. The monetization model was selling some E-Commerce items, so I naturally tried out some of the plugins and was shocked at how long it took for me to get a simple task done. I had such a terrible experience that I'd never recommend it to anyone. Wordpress by itself is fine, but when you try to extend it, you face so many hiccups.
That's when I realized there's no use blaming the tool. It's because of the differences in philosophies between me and the core Wordpress team. So, I naturally spent another 4 months writing a Rails app for this travel startup and still wasn't satisfied with my time to market. Clearly, there had to be a better, faster way?
In essence, I realized every online startup requires these components:
1. Authentication / Authorization
2. CMS - To manage content on the site, including home page, landing pages, blog, etc.
3. Analytics - To help track pageviews, campaigns, etc
4. CRM - To manage a sales pipeline and sell to customers. Also to know very well who your customers really are.
So, I went ahead and wrote this mammoth of an application in phoenix (using DDD's architectural patterns), that has all the modules above. Now, everytime I have an idea, I just login into my interface, setup the content and the theme/design and launch a campaign...bam! My idea is now live and I can test it out there on the market.
You can think of it like a complete combination of all the startups out there:
1. Mailchimp - I can send unlimited emails, track opens, analyse them. Handled by my marketing module. I can customize the emails too, of course.
2. Unbounce - I can design my own landing pages. Handled by my CMS.
3. Buffer - I can schedule shares from within my interface based on best times by engagement. Handled by my marketing module.
4. Hubspot - My system has a full, hubspot/zoho clone of CRM.
Here are some of the key highlights:
1. All my data is collected on BigQuery and I own it instead of sending to third parties.
2. There is no forced limitation on my marketing - For example, if you used mailchimp, you know you're limited to just 2000 recepients. If anything more, it quickly gets expensive. But my system is my own, no limitations whatsoever.
3. I can spend less time developing my idea and more time executing it.
4. I have my own custom business dashboard for each of my idea, that tells me how good/bad it's performing, so that I can turn it off when needed.
Probably not the kind of automation you were expecting, but yeah.
EDIT: Added more details.
> I can send unlimited emails, track opens, analyse them. Handled by my marketing module. I can customize the emails too, of course.
I'm curious - what do you use for email sending? How do you handle bounces and ensure deliverability?
For tracking emails, I use Mailgun. For deliverability, I use some standard practices to ensure reasonable deliverability - dedicated IP, sending in batches and analyze, etc.
I don't need complex systems to ensure my campaigns are spammy (guys like MailChimp need it because there are multiple senders) because the sender of these campaigns is just me.
If you ever write a blog post on the process / checklist of "automating your startup", do share it here.
PS: anyway, sorry for digressing away from the topic of the HN thread lol
I was writing that script for Amibroker platform (https://www.amibroker.com) but I believe the market leader is probably something like MetaTrader (https://www.metatrader4.com). Also, for developers or software engineers familiar with Python, Quantopian might be something you'll enjoy using (https://www.quantopian.com).
Often I'll spend as much time writing an automated solution as it would take to do the task manually, even if I'm only going to run the automated solution once. The work is way more fulfilling, and I can fix mistakes easier, and can learn and develop new techniques.
[1] https://opensource.com/article/17/4/automate-podcast-publish...
I wrote about it here: https://red.to/blog/2016/9/15/automatically-controlling-a-de...
and OS'd the Rails app: https://github.com/reddavis/Nest-Dehumidifier
Here's how far I've come: https://github.com/GustavHenning/usefulBash
Chef, puppet or Ansible are also good options I heard.
Receiving and sending documents to proofreading
I described them in details here: https://www.netguru.co/blog/automating-myself-out-of-the-job...
Also, not sure if that counts, but I have monit+scripts monitoring backups timestamps and DB replication
Interfaces with themoviedb.org for plot summary, cast and crew info and such. Interfaces with Google Calendar for writing entries for each movie I'm tracking.
https://github.com/evmcl/movieschedule
Let me know (or put in a pull request) if any of the instructions in the readme file are unclear.
Hope you find it useful.
Naturally I called it Pocket Lint.
mkgithub ~/dev/new-project
fgit pull -- ~/*/.git/.. ~/dev/*/.git/..
~/dev/tilde/.screenlayout/right-tack.sh
mount-image ./*.iso
vcard ~/contacts/*.vcf
~/dev/vcard/sort-lines.sh ~/dev/vcard/sorts/Gmail.re ~/contacts/*.vcf
img2scad < example.png > example.scad
indentect < "$(which indentect)"
qr2scad < ~/dev/qr2scad/tests/example.png > example.scad
schemaspy2svg ~/db
iCloud documents edited on iOS -> versioning and shoving in a private github repo -- https://paultopia.github.io/posts-output/backup-to-git/
CV updates via template to HTML, latex, and docx
I just ... don't get it. I fear I may be a bear of small brain.
As someone else mentioned, there are other valuable considerations the chart ignores, such as likelihood and impact of mistakes.
- How likely is it that you will make a mistake when performing the task manually, and how easy is it to fix the various mistakes that can be made?
- How likely is it that you'll overlook the task when the time comes? (e.g. SSL cert expiration)
- What are the costs of mistakes or failure to complete the task?
- What skills can you learn while developing the automated solution?
- How can techniques developed for the automated solution be reused for other tasks?
Of course it's fair to ask questions on the other side of the equation, too, like: if the automated solution goes bad, or is run in the wrong environment, what are the risks?
- gofmt for Go, Google Java Format for Java
2. Code Style Enforcement
- golint, govet for Go, CheckStyle with Google Style for Java
The more typical workflow is that artists export .JPGs or .PNGs manually from Photoshop and somewhere else save their .PSD files. Similarly with 3SDMax or Maya they'd manually export using some plugin. That seems wasteful to me and error prone. Source files get lost. Artists have to maintain multiple versions and do the export manually which seems like a huge waste of time. So, I automate it.
Also very usefull as web developer to test some javascript on a website.
It is free for 50 monitors.
I managed invitations as a CSV (who had been invited, who responded yes and no, addresses and dietary requirements).
I designed the placecards and seating plan as SVG in inkscape with special text I used as {templating parameters}.
I could then produce all my place cards and seating plan from a simple simple script. This was handy when guests changed their RSVP a week out from the wedding when I had little free time and I could make a change instantly. (Although admittedly I spent more time getting the layout right for the seating chart than if I had done it by hand).
rm -rf ./QuantumPilot* rm -rf ./QuantumPilot* electron-packager ~/ele/electron-quick-start/ QuantumPilot --platform=all --icon=/Users/quantum/Desktop/QuantumPilot.icns open .
for some reason, OSX has trouble deleting the Linux folder the first time. I've heard Itch.io has a CLI for this but I haven't tried it yet. https://github.com/itchio/butler
Code reviews. Using something like CodeClimate to automatically check code quality before anyone actually reads the code.
It lacks (or lacked a couple of months ago when I last tried it) functionality that I deem indispensable, like putting and importing all the movies from the same directory (it wants - or wanted - every movie to be in its own directory).
Also, it uses a lot more memory. Which is probably fine for most people, but since my little server only has 1.5 GB RAM, it really makes a difference for me.
I would add that it's a good idea to automate backups using more than one method as well. For example, use a popular software package but also copy important stuff to an extra drive. A cheap setup with a Raspberry Pi + USB drive plugged in to your router will do just fine.
Now automating Radom file restores and checksumming is a great idea...
I find that converting a lot of unimportant emails into RSS feed items has been a huge win for me.
Automating that is fiddly and tedious, but it's worth it because I can just click a button and get a menu of clients. I choose one, and in about 10 seconds my machine is ready to go on their work.
PS. Also when Internet is back, it alerts again so I can resume online Work if I have to.
So I recently wrote a CLI in Node that takes a URL and a CSS-style query selector (ie, '.album-title'), then scrapes the page, searches for each found instance and adds them all to a spotify playlist.
github.com/markreid/scrapify
Feel free to fork or borrow or contribute!
Then when I need to report all stats of multiple product, there is another automated script for me to aggregate them.
Saved me hours of context switching and copy and pasting.
It takes some getting used to but I feel it helps avoid forearm soreness.
I have automated whenever there are significant events happen in our app, I will get notified. Its simple to implement. Configure the webhook.
Also, I did things like getting notified whenever there is a commit, pull request or push in your source control.
Aircon via temp sensors and node-samsung-airconditioner
still working on Owntracks/mqtt for useful automations on arrival home
lights plus motion sensor, lihht color by time of day (red at late night to save vision)
Simple web interface where I have a list of packages I've ordered with the last status update from post service web tracking for.
I'm glad my national post office provides that data at all via their website.
In the case of f2f (face to face) I just let my phone run me like a peripheral.
I tend to propagate errors and let supervisor handle failing apps. But it's the way it is now.
--
This post has been automatically generated and may not reflect the opinion of the poster.
Use Python/Shell for tasks that are not well suited for a configuration management server. Usually, this is when procedural code makes more sense than the declarative style of Puppet manifests. Interactive "wizards" (i.e. add domain users accounts to a samba server, and create home directories for them) and database/file backups are my usual uses for these types of scripts.
Fabric is a useful tool to use with python. It allows you to send SSH commands that you put into functions to groups of servers in bulk.
I also use python for troubleshooting network issues. It has libraries to interact with all manner of network services/protocols, as well as crafting packets and creating raw sockets.
Look into PowerShell if you work in a Windows environment. Everything from Microsoft is hooked into PowerShell in their newer versions.
Ansible is basically a glorified shell script executed semi-interactively through SSH. It doesn't exhibit a new paradigm in server management by any stretch.
CFEngine, on the other hand, shows a whole another way to approach managing systems, and thus it's quite hard to learn (akin to learning Prolog or Coq after writing in Java). Puppet stems from that, but then slaps on top of that many weird and possibly unnecessary concepts, without much of an apparent plan or strategy, so it ended up a mess. Chef is a Ruby framework and some tooling around it, so no wonder it's more difficult to deploy than shell scripts through SSH.
It is. Is there anything wrong with that if the most important feature to you based on your requirements is that it be the quickest to learn?
I needed something that was very quick to learn and I had most of my setup automated in Ansible before I had even finished trying to learn the other options. Ansible won hands down on the "easy to learn" factor. So far it meets all my needs, and thanks to Ansible Galaxy, I've had to do very little config myself. 95% of the time someone has already written what I need.
> thus it's quite hard to learn
Then it's definitely the wrong product based on my requirements at the time.
> It is. Is there anything wrong with that if the most important feature to you based on your requirements is that it be the quickest to learn?
On its own it's not a wrong thing, but as I said, you didn't learn anything substantially different, so you're stuck with the approach that doesn't scale. You only added shell script SSH executor to your toolbelt, while you still haven't seen a proper configuration management system.
> you didn't learn anything substantially different
So what? Learning something substantially different wasn't one of my requirements. My goal was to automate a few simple deployments, including my personal laptop setup, which previously took at least a day. Ansible got the job done and I haven't looked back.
> You only added shell script SSH executor to your toolbelt
You've said this enough that I'll push back. That's definitely wrong. I can write shell scripts. I would have never attempted this in a shell script. Ansible was far easier than writing a shell script, and it does a lot more than just executing scripts via SSH. For one, it checks the current state of the system it is configuring to determine whether or not software needs to be installed.
> you're stuck with the approach that doesn't scale.
That wasn't one of my requirements either. Although I'm not even sure your claim is true considering that there are some pretty big companies using Ansible, and it was recommended to me by someone who is one of the better programmers I have met in my long career.
The whole point of my original post was that I actually tried products instead of reading about pros and cons. Based on my needs, Ansible was far superior to the other products I tried. Because my requirements aren't your requirements. I'm not so sure why it seems that it's so important to convince me and / or others that Ansible isn't a real tool. People should spend some time trying a few of the most popular tools and decide for themselves which tool best meets their needs. Which may include constraints like how much time they can budget towards learning a tool. "Proper" tool or not.
> You've said this enough that I'll push back. That's definitely wrong. [...] For one, [Ansible] checks the current state of the system it is configuring to determine whether or not software needs to be installed.
So you claim that Ansible learns the state of the system through some magic that is not possible to be called from shell? Because I could do pretty much the same in a shell script just fine.
I've seen a different paradigm for managing servers, I thought about it for a long time figuring out the differences from a shell script, and this is not where they are.
> I'm not even sure your claim [the approach doesn't scale] is true considering that there are some pretty big companies using Ansible
There are many big companies that throw money at dumb processes only to make them running, so this argument is pretty weak.
> [Ansible] was recommended to me by someone who is one of the better programmers I have met in my long career.
You see, precisely this is the issue here. It was recommended to you by a programmer. You should have asked a sysadmin, because this is where these tools come from and whose tasks they do. For some strange reason, programmers tend to avoid tools available to sysadmins, which is a quite big blind spot (even more so, since it's usually unrealized blind spot).
I never claimed that.
> Because I could do pretty much the same in a shell script just fine.
So could I. But I didn't have to. Because Ansible does it for me. So right there, just by comparing my declarative configuration with the current system state, it's clearly adding more value than "shell script executer".
> There are many big companies that throw money at dumb processes only to make them running
You said it can't scale. You're now moving the goalposts to "well, it can scale, but it takes more money".
> You should have asked a sysadmin
Fortunately I didn't ask this sysadmin because you keep confusing your own requirements with mine. I was never looking for a product that is best of class but takes weeks or months to learn. I've already made it dead clear several times that my most important requirement was "easy and quick to learn". Ansible is the best product of those I tried when that's the primary requirement.
> You see, precisely this is the issue here.
The only issue here is that you can't accept that Ansible was the best fit for my requirements.
> For some strange reason, programmers tend to avoid tools available to sysadmins, which is a quite big blind spot
Some programmers have enough common sense to not spend months learning a tool to automate a few hours of work that happens a few times a year. I asked my friend which would be the quickest to learn, he said Ansible, I wasn't hot on it because Chef and Puppet were more popular, so I tried learning Chef and Puppet and when I realized how long that was going to take I tried Ansible. Within a few hours I had everything automated with Ansible. It was hands-down the best solution based on my requirements. So I do agree there is quite a big blind spot here. But I wouldn't say it's on my side, or the friend who's advice turned out to be spot on. ;-)
Very little value in this regard. As I said, I could just as easily write declarative configuration for machine deployment in a shell script.
>> There are many big companies that throw money at dumb processes only to make them running
> You said it can't scale. You're now moving the goalposts to "well, it can scale, but it takes more money".
I'm not moving any goalposts. If a process takes unproportionately more money or effort for more input, it doesn't scale. Heck, in this case it doesn't scale even if it takes the amount of resources proportional to the input!
> [...] you keep confusing your own requirements with mine.
I don't. I'm saying that you didn't learn anything significant, and as such, you would be just as good if you put some thought into how to do exactly the same with the tools you had (shell scripts). But now you're probably worse off, because you most probably got misled by Ansible's marketing into believing it is also a configuration management solution.
While you were over there thinking about it, I had already completed the job. Which was my #1 requirement. It doesn't seem like you'll ever understand that different requirements often result in different tool choices, but that's ok.
> But now you're probably worse off
Probably not. I now have several things automated that I once had to do manually. I may not be the smartest person in tech, but I'm smart enough that if I need a better solution than Ansible at some point in my career then at that point I'll simply learn the better solution. Different requirements = different tool choices.
No true Scotsman...
For instance, I once took a serious look at Angular and then decided it's too difficult. I'm sure there is time and place for Enterprise JavaScript, but none of the things I'm doing right now would benefit from it (and frankly, neither do most of the smaller sites I've seen using it).
https://learnxinyminutes.com/docs/shutit/
use it all the time, eg:
https://zwischenzugs.wordpress.com/2017/03/04/a-complete-ope...
I ended up with Ansible because it seems to require nothing on the far side but python2 and ssh. Everything else needed a specific version of a specific client software. CFEngine was particularly awful in a diverse setup, fighting SSL to make puppet work was never fun. I haven't found anything Ansible won't talk to, or can't be made to talk to eventually. I used Puppet for a bit less than a decade before switching to Ansible maybe two years ago, and I don't miss fighting its SSL at all. I miss when the puppetmaster would randomly crash and I'd find out when a deployment failed or worse failed partially, or when the puppetmaster process would take 5 minutes to start because, well, I donno why, but it sure was annoying.
One interesting anti-pattern I saw over the years was people using puppet (or ansible or otherwise) as the worlds weirdest version control system. You can git pull a release branch, or you can write 10K custom lines (no kidding) of automation software to replace one line of 'git pull'. Depends if you're paid by hour/LoC or not. The borderline between "this is managed with ansible/puppet/wtf" and "this is managed by living in a git repo" can be annoyingly fuzzy and going to either extreme doesn't turn out well...
advobot.co/web
We have 15 main categories, each with their own Twitter, Medium, WP, Blogger, etc. Here's an example of our science Twitter account: http://twitter.com/@Snapzu_Science
Problem is that it isn't ready to for the public. I'll do a show HN next week, but by GOD it is a brilliant piece of automation and scaling :P
Soon (this is more for me than anyone else, i'm literally bursting with pride right now)