Look for the duct tape
rachelbythebay.com
rachelbythebay.com
Looking for a SaaS idea? Ask random people in a target industry what kinds of things they do with Excel.
In a similar vein, there are plenty of small software teams that use nothing more than a whiteboard for project management. Yet there are teams tooled up with Jira/Trello/Slack with much less productivity.
If two business processes interact through a spreadsheet, or more than two people edit, you're going to immediately run up against the rough edges. Multiple people editing the same sheet inevitably ends in error. It can take a really gross amount of time to update the data when you're pulling/pushing from two sources. It may seem small on the surface, but even 2 hours a week adds up to nearly 3 weeks of work over a year. Can you replace this spreadsheet in 3 weeks?
I've seen this play out many times at large and small scale. The first example that comes to mind is a 10-hour day trip my wife took to go out of town and manually update a whiteboard because somebody messed up the shared spreadsheet. She was very unhappy, and this wasted a precious day during a _very_ busy period. Initially, the whiteboard and spreadsheet worked great, so they didn't "fix" it. It didn't scale to the busy times and really screwed them over.
At a much smaller scale, I wrote a quick SQL query to get some business metrics for the CEO. He asked if I could run it for him every week. Since I didn't have a good place to put this query, I just ran it manually and emailed it to him at first. Then he wanted to share it with someone on the opposite coast outside our network, so he asked if I could start putting it in a shared spreadsheet. My initial thought was to bang out a page they could pull up instead, but I figured it wasn't worth it. Then he wanted another metric. And another that required a bit of logic that couldn't be done in SQL. Pretty soon I was spending an hour a week manually running SQL and copying the results in to a spreadsheet. (We're a Google Drive shop. No way to connect Sheets to DB2.) That's simply not worth my time. I should've listened to my first instinct.
The initial investment in a spreadsheet isn't large, but the ongoing maintenance can really rack up hours. I think it's very often worth investigating any spreadsheet that users have been maintaining/recreating for more than a quarter.
FWIW, Google has an API that can be used to get data into and out of things like Sheets. The documentation for it is abysmal and there are only a few libraries out there, most of which aren't complete. But, I've done it, and have some rough (but easy-to-understand) code that can "import" Sheets into MySQL. Going the other way should be a matter of tweaking the Google account permissions and calling a couple of different functions.
It's in PHP, but it's not gross PHP, so you can probably translate it into whatever you're familiar with.
Yes, the App Script API is kind of limited, but also extremely powerful for simple services or to glue services together.
You can for example set up a cron job to post some json to that endpoint and have the js validate it and update a sheet accordingly (or even create a new one).
There's a reason Excel is so powerful, but it can't magically solve for people lacking a process and most software solutions solve it by using complex permissions, versioning and collaboration features. You can just easily lock down the shared spreadsheet to readonly and have 1 or 2 people who are designated editors and get the same effect, or just have the changes done on a new tab and compare that way.
Even the most amazing software can and will get screwed up by users who don't have a process.
If there's a better way then suggest it, or at the very least document the time and effort spent on these reports. If they are that critical (which they usually are for decision making) then it shouldn't be an issue when planning future work.
I’m not sure how “global level” organisations are, but mostly you just can’t convince a typical layman CEO that something looks easy, is really not easy.
Then when you serve the “small request”, next he’s in doubt why adding a small feature could increase the complexity of your work by a few order of magnitude.
The problem isnt Excel or any useful tool, its the misapplication of the tool and seeing people driving that same road already extremely well traveled.
I can't count the many times a business process is done (poorly) by inputing lots of data in an excel file, running a huge (poorly written) macro, and spending the file via email to the next person and so on...
This means that if your computing problem doesn't need to scale, a spreadsheet may just be the perfect tool.
However, I agree with you. The next Facebook might be of whomever builds a spreadsheet that scales.
I tried smartsheets but it’s more calc centric. Asana is great if you are task centric.
You've never seen panic like a multi million dollar business discovering that excel doesn't allow more than some fixed number of rows (1m?).
I think something lost in this conversation is how spreadsheets are largely a part of general primary school curriculum. Generally people have a feeling of confidence around them that is lost with other tools. Particularly "easy database" applications that attempt to replace spreadsheets.
The opposite is also true: You can make a decent living creating, customizing or even just maintaining domain-specific tools, no matter if the customer's business really needs them or not.
It's very easy to criticize something by saying you don't need it. There are very few things in life that you absolutely need.
Yet there are things that add convenience, greater efficiency, etc, and people will pay for them.
A spreadsheet works until it doesn't.
Maybe you load in past, say, 150mb of data.
Perhaps the last person to touch it linked it to a data or content source they only have access to.
Perhaps the person editing it has it checked out from a Sharepoint and forgot to check it back in.
Perhaps the person editing it left it open on their computer and forgot about it being hosted on a shared drive.
Perhaps the business critical sheet is hosted on someone's desktop.
Fundamentally, spreadsheets are cocktail napkins. It's fine to sketch out a basic idea. It's a terribly risky approach to run a business from it.
And I've found out that having excel export of essentially any data in your line of bussines application is indispensable unless you expose some kind of excel-like querying and data manipulation mechanism to users as part of your application.
I work for a small company, we generally use Jira (which we all hate, some with more of a passion than others... ok mainly me).
More recently we sort of accidentally started using google sheets for managing issues on some long running projects: it's easier to get an overview, i can add and remove unnecessary attributes of issues and filter them appropriately and dynamically without having to define a whole new workflow and start over. It takes basically the smallest amount of time possible to log an issue... I can organise them however I want. Issue management is overrated, issues are simple, and having something as flexible as a spreadsheet is a surprisingly good fit for anything not customer facing.
Oh also, I don't need to _learn_ how to do all those things... it's a spreadsheet so it's obvious, but Jira - it's so hard to do anything unless you live by it, I don't want to live by an issue manager I want it to get the fuck out of the way so i can do work.
I one worked at a place where a key part of the finance system ran on a spreadsheet - turns out this SS had mess up or finances and was one of the reasons the company restructured.
If what you are doing is applying math formulas over small or medium sizes of data, then excel is OK.
Otherwise it will suck and will be a waste of man-hours at the Office.
I can't believe I'm about to sink so low, but... criteria!
Criterions are what one adds to soup ;)
You're thinking of crouta.
Besides, I wanted to make a somewhat self-deprecating comment, rather than - as you have here - regurgitate some half-arsed passive aggressive form.
Strongly disagree with this based on personal experience in "traditional" spaces that rely heavily on spreadsheets. There's a reason most of the financial industry uses Excel spreadsheets and it isn't because they can't afford to pay for specialized software. Excel spreadsheets are incredibly versatile. Your data format can change a little bit over time and you'll still be fine. With a custom software solution, you're kind of stuck within the confines of what the developers/product managers gleaned from your working process.
Do the users have massive, error-prone and/or time-consuming workflows built around those spreadsheets? Is it very hard to bring new staff up to speed, or to keep existing staff compliant with process, or to meet day-to-day business goals with those spreadsheets? Is there no solution between "painful spreadsheet" and "$X,000,000 contract w/ bigcorp" to solve those problems? If 'yes' is cropping up in answer to those questions, then lifting the logic from ad-hoc process around spreadsheets into software may be a good idea and/or a good business.
The main downside was that the functions and transformations weren't sharable across documents, and often there ended up being several mildly tweaked versions of the same functions across a bunch of documents. But I think I learned something about human tool use: when motivated, people will find a way to use the general purpose tools they know to improve individual efficiency.
GP is talking about people outside Finance who use Excel as a database. Though he's being a bit facetious, he's actually not far off. For the store data/retrieve data/share data workflows that most of these folks have, there actually is a good SaaS business in putting Excel on a server, wrapping it with a database, putting a nice UI on it and adding a few industry specific features.
One also has to consider the cost of engaging the IT department or others who can do and/or manage development within the organization. That's NOT cheap. Many businesses can barely manage to handle their own shit, let alone successfully do even a small unplanned project to port an excel app running in somebody's cube to something more robust.
I understand where you're coming from, but this is just not good advice. Somebody (I can't remember if it was pg, Joel Spolsky, or someone else) once wrote an entire essay basically saying "DON'T make a startup that's just pivot tables. I have seen so many SaaS startups fail because they didn't realize they were just selling pivot tables at a much higher price than Microsoft wants for Excel."
Also take a look at this HN thread: https://news.ycombinator.com/item?id=12450802 where a whole bunch of people express bafflement of and loathing for people who unnecessarily replace Excel with half-baked software. It seems like a good idea; it's usually not.
Not for a startup, no, but in my experience it works well as a lifestyle business. Companies pay well for this kind of work and it seems the number of spreadsheets they want to have replaced provide enough work for the next few decades to come. Better the replacement isn't half-baked though...
True, but you also have to make sure that you can improve on the spreadsheet way enough to be worth potential customers' switching cost and money.
I actually wrote a post about exactly this only a couple of weeks back: "Your biggest competitor is a spreadsheet" - https://medium.com/@hjalli/your-biggest-competitor-is-a-spre...
- one spreadsheet is shared among more than about 5 people (or they each have their own version of it... shudder)
- the sheet has more than about 10,000 rows
That's the point where Excel truly starts to be not the right answer.
We have rather low requirements, which is why I'm resisting existing shop systems. They all seem like overkill to me.
But from time to time, someone wants a some statistics, or needs to send emails to "everyone who bought in the last four weeks". These are trivial for me to compile in SQL, but a spreadsheet would allow others to quickly work with the data.
Unfortunately, the Sheets API at around one second per request was too slow for my tastes when I last checked it out.
I agree this is a good place to look. I'm not sure a new tool will be an improvement as it will always have limitations and a learning curve Excel doesn't have.
Which is not to say that whatever replaces it is guaranteed to be better, but it will at least have a good start by using the right tools for the job.
What happens when this person leaves? What if business starts booming and you need to scale up? Are you really better off sharing a spreadsheet between a 100 or more people?
Just gonna focus on this part here. The last offices I worked in (and the last places I even dealt with tech stuff beyond teaching it to primary-schoolers), "Track Changes" was simply a godsend. One person "owned" the spreadsheet and could allow use or not; and we had it hosted on OneDrive - of course we all had the OneDrive client on our work PCs and it could tie into Excel natively, so we were all using the same document.
If the entirety of a large org needs the same spreadsheet, you're probably looking at a scenario where they really need multiple sheets, but haven't figured out how to do references to other files and sheets yet. But using Track Changes and having one person designated as Owner so that that one individual makes a decision to approve/disapprove all potential changes makes it much easier to let multiple people work together.
> What happens when this person leaves?
That's a management / succession-planning issue. Ad-hoc solutions shouldn't be kept secret, and allowing employees to just go off and do as they please in that regard is a major issue. If all work-related documents are required to be stored in the company cloud, and you storing a new doc alerts your manager, etc., you'll be able to deal with this much easier. One place I worked for a few weeks on a project did a reset on employee's terminals every Friday at around 7PM, and anyone who claimed they lost data or progress from that on Monday was in trouble, because they didn't save company-related data to the document & data server (which obviously did not get reset) - meaning they'd done something against company policy that they'd been told about the day they were hired.
I’m allergic to the over-engineered solutions so prevalent.
We sometimes even propose to build a native Excel based VBA app. The users can always use something like Dropbox sync or box sync on Google Drive sync to share their work with others.
Think of all the would-be SaaS businesses that have been disrupted by Airtable.
I've once worked with a trader whose magic sauce was encoded a spreadsheet he had been developing and carrying around for 10 years. It had tens of thousands of formulas and references cross ~50 sheets.
He suspected he had introduced an error somewhere, but gave up after a couple of months of trying to, you know, verify the whole thing.
When you have error-prone processes that you do hundreds of times a day based on those spreadsheets, that a bigger application can automate.
When ten of those spreadsheets could really be replaced with one.
I think if you were a product designer you would do well to have users send you their customization files to extract things that were common to all of them to put into the product.
And of course Windows was pretty famous (infamous?) for taking what ever shareware thing people were giving out to make Windows work better and then incorporating it into the base OS. I read a review of Windows '95 that called it "Windows 3.1 with Norton Desktop pre-installed."
The old desire path that is based on user experience not design [1][2]. Never fight the desire path, it is a losing battle.
I've made them look stupid by building something pretty good where they have been failing for 10 years with an entire team. Then I went and wrote another application in a strange new framework that had been invented in the past 15 years, and this startled and frightened them.
Don't be like me, kids. Play the political game. If you don't want your managers being told to dismantle the things you build, simply because you didn't kiss enough butt, learn how to raise your "EQ" and ensure the success of what you build. Otherwise, like me, you will have just written a bunch of stuff users loved, but management hated, and forced them to stop using.
It's too late for me; save yourselves.
I took a new job about a year ago, one that found me in a group of people who were struggling to do "simple" things, but doing them the "hard way". I came into this job, and because the core infrastructure is Windows Server, I learned PowerShell in about 6 months and started "solving" their problems by automating this and that. Things got easier, but the existing people thought me strange that I would resort to the "mere command line" for a solution rather than go about things using a GUI. My rule of thumb has always been "if I have to do it more than twice, script it."
These people have poo-pooed my every effort along the way and the boss has even called me out on it, despite me showing him that we are making strides where the team once did not. Sorry, I'm a command line guy at heart, and this will never change. I'm not a method man by any stretch. PowerShell is the orthodox way of handling Windows Server issues and that these people I work with don't want to embrace it is not going to slow me down.
We engage in useless meetings to have meetings. We talk about senseless things to jabber. I want to write scripts to automate certain tasks, but no. "What happens if you get hit by a bus, bro? No one knows how to code in PowerShell." My response to them was "I came here not knowing PowerShell, but I took the time to learn it and I'm productive in areas where you could be, too. I'll be happy to show you everything I've learned and coach you when you have time." They look at me like the proverbial deer in the headlights. I'm toying with leaving this year, as I want to work around like-minded, always-curious people, not people stuck in yesteryear.
Also if you are that much more effective you could start your own business, hire people and train them, then have 10x bigger margins then your old employer.
I have no idea what to do, what to offer, etc. It's funny, because when I'm at work, I don't really have to think too hard about solving a task. I'm generally the "smooth over" guy who fixes the things other people don't really see as issues, but having been in IT for so long, I can see the problems in the distance for a given job, as I've encountered them before. Most people just fly by the seat of their pants, but I love doing things once and well.
That, and I don't have the temperament to deal with the politics of the business side. I tried it, decades ago, and found it tiresome.
[I work in an educational organisation with around 1000 employees and around ten thousand students. There are something like 1 500 client PCs and the usual collection of servers. It is a microsoft shop. Powershell much in evidence.]
It's about building relationship and trust, and I feel it is less draconian as "kiss the ring".
I recently started as lead dev at a small company experiencing some real growing pains. The CEO wisely had me spend a couple days my first week going around talking to team leads. It was very productive.
I started with almost the exact same question Rachel suggests here. My version:
> What's currently frustrating you?
I deliberately left it open-ended and found many of the non-technical answers worth passing along to management. "Let the fools have their tartar sauce!":
https://www.youtube.com/watch?v=X25rP4rTw3A
The other question I came up after my first couple meetings. I called it the Magic Lamp question:
> If, as the magic genie of this company I could grant you three wishes, what would they be?
By day two, these were the only questions I needed to ask. Following up with the CEO, we came away with a very clear vision of what our development priorities needed to be. Most of it aligned with what I already sensed coming in, but there were still some useful surprises for us both.
(And, yes, found a few clever spreadsheets functioning as de-facto business applications.)
The boss of the team i eventually joined said something like "i'd like to be able to make all of our performance-critical programs fast enough without spending weeks tuning each one".
Having worked there for a year, i can tell you that this is nowhere close to being a significant problem for the team.
So yeah, ask, but don't take the answers at face value. People don't know what the real problems are.
Career Cold Start.
It's a dense passage to be sure.
(probably from another article than the one linked above)
> First thing, do not stick your neck out.
This advise always frustrates me. I know it's true, but feel it's really, really hard to implement.
a) What do you do if you don't get any direct command from your manager? If you then choose to do nothing you get the negative feedback that you didn't do enough. If you go left, you get the feedback that you should go right. If you go right, you get the feedback that you should go left. If you go straight, you get the feedback that you are too direct. With everything you do, there's automatically, intrinsically something you don't do, and that will be used as negative feedback.
b) I don't know about other people, but becoming more successful is really a necessity for me. If you always get negative feedback, how could you progress? So I start hinting at the left and when the negative feedback to go right is incoming I show that I actually moved right. You can believe me, I speak from expeirence, that's when the real shit starts to hit you. Suddenly you are not just a stupid employee, you are also disloyal and an intriguing schemer. You are dangerous and need to get rid off. So apparently that's sticking the neck out.
But then, what else to do? If (a) and (b) both is not acceptable, then what is (c)?
PS: Also this article. Feels so much like my current situation: http://rachelbythebay.com/w/2012/01/09/rigor/
If your manager continues to be ambiguous and you haven't managed to leave, then I would try to go orthogonal. Achieve something unrelated. It is hard to blame someone who succeeds even if that success is unrelated to the main quest.
So no, it's not hard to blame someone for working on the wrong things when they should have been working on something else.
Back then your options were basically Ettus transceivers for >$1K or the little RTL dongles for $20 that didn't have sufficent bandwidth to do much good. Since then a whole crop of SDRs have hit the market in the <$500 range that could do this kind of thing.
I will admit that I actually listened to the original system a lot with headphones, so I had every reason to fine-tune it until it sounded as awesome as possible. If you go back far enough into 2011, you can find a bunch of calls with terrible audio before I figured out the finer points of squelch, tuning errors, and 48 vs. 44.1 kHz :-)
I haven't decided whether it's worth building another system now that I have time to actually listen again. Hmm...
(I love all your posts!)
For a little side project she did a fantastic job (very good write-ups of the trials and tribulations in her blog as well).
BTW, isn't posting captures of radio transmission online illegeal, because it qualiifes as "rebroadcasting" under the FCC regulations?
(a) The term rebroadcast means reception by radio of the programs or other transmissions of a broadcast or any other type of radio station, and the simultaneous or subsequent retransmission of such programs or transmissions by a broadcast station.
Internet websites are not broadcast stations.
Anything in ISM band or anything said on unencypted 'walkie talkie' bands e.g. Family Radio Service?
>"I've been having some wonderful conversations in the real world since resigning and jumping off the merry-go-round."
What merry-go-around or where did she resign from?
>"... and how to succeed in the "2018 version" of the culture which now exists"
Sometimes you can be better off doing things the simple aka duct tape way, when it just works.
Anyone who's not read tacobell programming should take the 2 minutes to do so now because it underlines my point very good.
http://widgetsandshit.com/teddziuba/2010/10/taco-bell-progra...
Incase my point is not clear: you have to differentiate, the concept of work with the concept of problem solving.
You start with a simple script to do a simple job. It works - well. People want more features. You keep going until nobody can understand what you've created.
Then you leave the company.
Any software that works will be extended given enough time. To adapt the Peter Principle, every piece of software rises to its own level of incompetence.
Which, from a simple review of the contemporary software landscape, is obviously false.
maybe you can paint a more detailed picture?
Almost the entire Unix software ecosystem is simple, robust, and composable. I don't know where "golden age" comes from, though the list of existing software that has not risen to their own levels of incompetence stretches back decades.
I think more senior engineers fall into the trap of seeing the maintenance burdens ahead and not wanting to, or not being able to, make the call to use the duct tape.
I have videos, images of our children in various places (my phone, one drive, dropbox, maybe usb sticks) that I want to go into the target directory (probably one drive). the tool would go through all the other directories, checksum them against all the other directories, and then present them as "these are the files that should be merge into your target directory".
Can regular diffing tools do something like that in an easy manner?
I get that for music and video this is actually quite challenging but images - - I would've thought quite easy? I've tried a few tools which claim to do this over the years but they all fail rather miserably / require a paid upgrade to do it better (although what is demonstrated in the free version is never impressive enough to consider paying).
Might be a good project for myself in my spare time, but wondered if anyone knew of anything obvious I was missing? FSLint and stuff is great for general filesystem de-duping, this is a slightly more complex case.
2: https://docs.kde.org/trunk5/en/extragear-graphics/digikam/us...
Also, the -P option (at least on the version I have installed) is for enabling both --partial and --progress. --partial keeps partially transferred files in the event something goes wrong, and --progress shows you the progress of the transfer.
You will probably want to use -r for recursing into your directory structure.
Unintuitive defaults, annoying long command-line options, needing to take many separate steps to complete one task are all problems that can be trivially and permanently solved with shell macros and wrapper scripts.
Performance problems, using the wrong data structures, random crashes, on the other hand cannot be solved this way.
Another group of friends started Personably from exactly the same strategy - we had a script to create dozens of calendar invites for new joiners with all their onboarding details, so things didn't get missed.
Dependabot has ~100 paying customers, and makes just over $2,000 a month. It's still quite a lot of work adding new features, but if I wasn't extending it's functionality it would be passive income.
Personably have raised money and landed some big paying customers - I think there's scope for them to be solving a relatively large problem, having started from a duct-tape fix.
The only proviso I'd add to the article is that sometimes the problems you find like this are small. Dependabot will never be a full-blown startup - it's scope is too small. That's not necessarily a bad thing, though - I think the problem has still been well worth solving thoroughly.
Curious to hear about people’s duct tape fixes for team/company culture issues.
Also better to try and get more nebulous work that can carry a higher estimate/ buffer in the first place. And never refactor in the same ticket. Create a new one. But that won't get done so just let it go.
What's key is not the hammer crack (bad enough?) or the duct tape repair (good enough?).
Instead it's the generality (portable, abstract concepts behind it like 'gluing' 'joining') and availability (common, cheap, simple) of the duct tape that enables the (w)hack.
People don't buy duct tape to fix cracked hammers.
- Afterthought : A Stack Exchange user answers: "why use duct tape when you could just use super glue" (or rubber bands, zip ties...).
This is definitely my duct tape. I am neither a developer nor a sysadmin, just a dumb end user. When something cannot be written using only shell builtins, I use a relatively small set of external programs in my dumb scripts, utilities that would be found on most UNIX install media.
If I had the skills I would write many of these in a compiled language, i.e. C or in assembler. As a non-programmer, I have found fasm to be the most useful e.g. its ability to easily call C libs, but I also keep coming back to inline asm as I like the extra bit of control over registers that if offers. I use a small, old assembly level debugger that I prefer over gdb.
History suggests more than a few well-regarded projects originally began life/were prototyped as shell scripts. (I would bet most readers are using at least one today.)
Even though I lack the knowledge and experience of a career programmer, I do not feel ashamed for using Almquists shell to test ideas. Though I certainly understand the use of the word "dumb". However the truth, no matter how "dumb", is that shell builtins are faster than most of the other scripting languages I have tried, and not just in theory.
For some of these dumb little shell scripts, I have managed to replace them with compiled programs (often with the help of flex/yacc), but most are beyond my C/asm "implementation skills".
The point is that if a C programmer came to me, an end user, asking for ideas of what "new" or "missing" utilities they could write (not sure if thats what the author is describing), I could supply a lengthy list. Double digits, maybe even triple.
As I understand it, programmers want people to use their software. Given that I use many of these dumb little shell scripts every day, I could guarantee at least one loyal end user.
Eg: "tac", reverse lines in input, sample input: "first\nsecond\n", output: "second\nfirst\n".
There's already the "standard" gnu coreutils, as well as suckless tools, and 9base from plan9. And projects like https://github.com/uutils/coreutils
So there's plenty of places to draw inspiration from.