978 karma · joined November 11, 2018
People led by a tool and not the other way around - and you blame the tool?
I hear what you are saying, and I've seen the very symptoms you're describing - I've just stopped chalking it down to the tools.
It's a symptom of something entirely different and much more challenging to deal with than a change in tooling.
It wasn't atlassian that "came along" - someone penned down an agreement and, from what it sounds, there was a lack of clarity both in regards to current and future principles of work and collaboration.
What we can do, as devs, to protect ourselves from the madness you describe is to be explicit about our work processes and their purpose. Nothing much, but just a set of agreed upon principles and ways of working which should have nothing to do with tools.
What you end up with, by being explicit, is for having "something" to be supported by one or several tools.
Makes it easier to evaluate, select and change tooling that is fit for purpose.
However, the final part has nothing to do with jira IMO. You would have the same behavior in these organizations regardless of tool.
The dysfunction you describe goes way beyond a single or set of tools.
At some project size, measured either by software complexity/interoperability or user base, you will need a tool to manage issues and tasks.
What you're talking about is an organization where developers are not empowered - but even empowered developers need an issue tracker or a board of some description.
A "management by jira" culture will not be remediated by tooling.
First batch of users used a windows client on their pc. When we after some time commited to the solution we built a really tight linux os image with only the citrix client that we pxe booted the PC’s with.
This allowed us a fallback in case of emegergency by simply rebooting a user.
This was a fun project in and of itself!
As we grew confident together with the users we started to phase in thin client instead of buying new PC’s - all in all we ended up with a really cost efficent and high performing setup.
We started to move the central infra from “hyperconverged” to commodity 2u servers with local solid state disks which really made it easy and cheap for us to scale.
It also made the whole service more reliable compared the integrated blade and storage solutions we were ushered in to at first.
These integrations was built at HQ, ”works on my laptop style”, but at a branch office 1000km away… no fun at all!
An unholy mess that would take many years and a couple of $100.000.000 to untangle.
I had weekly “continual improvement meetings” with the power/super users, one or two from each branch office. The whole team spent a lot of time at the local branch offices dog fooding the service and talking to the users.
I used the service as my only workplace for three years in a development & ops role.
The success factor here was that we built a solution for a specific set of users that we continuously improved. I’m talking “make something slightly better today than it was yesterday - every day” type of mindset.
We started out with a PoC by brokering sessions to physical machines located in the datacenters. First users was the assigned "super-users". They were assigned due to being the most demanding and loud user base.
After a few week they _demanded_ that every user should get the same setup - we interpreted this as a successful PoC and went on to explore options.
At this time neither Citrix or vmware had a concept called VDI and we started out using pre-release code from vmware.
First prod version was using vmware but we only gained performance, which ofc was a win, but we still needed to manage all the vm's through standard automations (os patches, asset and inventory, application repackaging and roll-outs)
When Citrix bought ardence to get PvS it was a no-brainer - ICA protocol, PvS and app-v streaming was a great combo to rapidly deliver a high performing workplace. BUT - you need automations and reliable network links, both of which we could provide.
We had dedicated hypervisor hosts with the fastest local storage available. You should NEVER user SAN/NAS or other centralized storage systems for these workloads IMO.
Also, monitor the performance closely! Caching all application to host local SSD really took care of a lot of the potential perf issues we saw early on.
Depending on user role they could get dedicated hardware as well, but this was only for a select few users.
It was set up as a specific service delivery and my team was dedicated to improving and supporting the users.
One of the most rewarding projects I've been involved in, mainly due to the really close collaboration with end-users.
I remembered almost correctly then - something about it having unix roots. :)
I helped manage many solaris servers running citrix metaframe at one point, many years ago.
One successful example is a support call-center with 2000 users. The specific role used about 30 apps and they were all packaged and maintained by my team. Using a ci/cd fashion to build os image and apps we could roll out updates to base image and sandboxed applications completely seamless to these users.
(Side note: We built an automation that synced the two latest versions of all applications to a locally attached SSD drive on all hosts - app startup was near instant for all users which was fantastic at this point in time)
Running the clients inside the datacenter brought the latency to a minimum which greatly improved the performance of specific business critical apps.
Managing all parts pertaining to logon properly had the users ready to take calls in 10-15 seconds - a great improvement compared to the previous distributed client/server setup.
- Reducing latency between application and backend was a major win. Greatly appreciated by users as we saw massive reduction in latency for specific operations (crappy apps).
- Being able to integrate with the ip-telephony call-center software required a windows workstation environment hence the choice of “VDI”
- No noticeable os and app updates EVER was greatly appreciated by the users. Even when moving from XP to 7 at one point.
- Our ability to roll out massive OS and app updates at will to pilot groups who can roll back with a logoff/login was greatly appreciated by the team
Granted, this was 10 years ago and my opinion was always that these business applications should be rewritten - BUT I have yet to see a better managed and performing windows client & app delivery.
Citix provisioning services (PVS) coupled with app-v/app streaming was an incredibly progressive way of delivering windows workspaces.
During my time at citrix I often raised the fact that PVS for linux coupled with docker was an amazing setup - never got any traction on this though… everyone was way too focused on windows delivery.
I seem to remember the company was called a more unixy “Citus” early on.
Spent a few years working there with customers using it on solaris and eventually linux as well.
In densely populated areas we’re seeing severe effects on our health thanks to all the other crap being emitted by burning fossil fuels.
I believe in a 100 years people will be like: “…and they burned this crap in the middle of where they lived?! Crazy stuff”.
But yeah, otherwise I agree - it’s a damned if you do, damned if you don’t type situation.
It’s like shaping a system of people. I think of it as 50% of the development endeavor at larger shops.
Retros are used to identify ”technical organization debt” starting to build up, in a way.
Wich makes it all a glorious Catch 22.
These companies, that rely completely on software, are unable to make the required leadership changes. It’s cultural.
Many pieces to this puzzle - generational living, skin tone and diet for example.
I claim using pretty much any tool other than jenkins is easier to maintain over time (as illustrated in this very article).
Regarding your batch scheduling solution - take hashicorp nomad for a spin: a single go binary and your scheduled job can be declared in 10 lines of yaml. Won’t miss a beat.
Drone has this awesome feature where you can have it hook out and receive a pipeline on the fly. We now generate our pipelines in an api and this way we can write the logic in ”something other than groovy” - typescript in our case.
No pipelines required in repo and everything as code without ugly hacks.
Never looked back even once.
Such is the cloud software... :) Cloud, besides APIs, i.e managing hardware at scale was not really what they did.
They did roll 1000s of vms per week through it in ci/cd flows though.
As such it did what it was supposed to do - docker/containers was not a thing at that point in time, and I remember thinking it was pretty awesome.
To many nifty engineers, with long fingers, for their own good though. You need to be strict with automations if you want to keep something like that running reliably over time.
Manhandled in-house though...
It’s easy to chalk a burnout down to emotions and stress, and these might very well be the root cause. For me the above, however positive in nature, gave me physiological “blockers”, in a sense.
Had my shoulder not started to act up (bicep it turned out after a couple of months excruciating shoulder pain - a whole story on its own, fixed by sticking a needle straight in to the muscle to force relaxation) I would not have ended up at the naprapath.
A kind of burn-out - but rest and mindfulness on it’s own would not have been enough to get me back.
I had, a couple of years previously, changed job and had the opportunity to do a complete retake on a traditional business from a tech perspective.
Incredibly motivated and excited - spent my days talking to teams and management and nights coding and building things.
This is not in the US, and I had no pressure at all from management mind you - I just really enjoyed it and we made amazing progress.
Couldn’t wait til the family was asleep so I could start hacking away, often to 3 or 4 AM.
This went on for about a year, and I thought I had the best job, team and most fun ever in my career.
One day going home from work I could feel a spring in my brain let go, much as described in this tweet - the whole world stated to spin and I found myself leaning against a car in the lot not to collapse.
Weird... wth... ok, let’s see if it was just an issue or if we have a problem.
A couple of days later I’m in a conference room to do a demo. Someone throws me the keyboard and as I start to write everything seems to slow down and my fingers are moving in slow motion. I try many times to type my password - impossible. I try to speak - just slurring.
From this point on whatever is going on start to manifest itself on a regular basis. I get a “zing” in my brain, like physical, and loses train of though, words and balance.
Going to work one day my body just stopped, turned around and began walking home. On it’s own! Surreal. I was just taken for the ride back to the bus and home to bed.
At this point I’m starting to feel like “I’m not there”. Like my mind is in the back of my brain and not in front, at my eyes, if you get what I mean. My mind had kind of an out of balance, far back, tunnel vision of everything. This became the new normal. Really disturbing.
Obviously I started to investigate this. Went through the process with different doctors. Lengthy process. At some point I get this unbelievable pain in my shoulder, probably because I had started to work out again quite hard. Way inflamed somewhere.
I go to a naprapath to try to sort it out and he investigates my neck - “woha you seem to have massive inflammation in your upper neck”. He does the naprapath business for 30 min. Boom! I’m back! I’m “in my eyes” again, almost two years later. No more tunnel vision! W. T. H.
Couple of week later I’m scheduled for a neurologist who examines me. He specifically asks if I’m ok in the neck. I describe the above and he explains that issues in the upper neck can cause all kinds of weird neurological issues.
The MR came back clean as the doctor had figured.
Getting back completely have taken it’s time. A good couple of years, and I’m in general more weary and try to listen for signs.
These days - no coding after hours, exercise regularly and in general a better understanding of my body and mind.
Listen to your body, brain included! Stop clenching that jaw.
It’s not going to be a technological shift - it’s a different approach to software and forming teams around it.
I’m writing this and it kind of echoes Brooks words in “the mythical man month” - it was written in the 70s.
It looks like a success until you ask operations and people maintaining the deliverable.
This is usually not factored in and ends up at at different cost center.
You almost never get the full picture of continuity when dealing with a project organization hence it’s really, really hard to judge.
No tricks really, just common sense, right? :)
This wouldn’t as much rub me the wrong way had this been an effort of actual employees and staff at some govt agency where this were a longer term (indefinite?) initiative. At least then it turns in to:
- mostly a zero sum game with my tax money
- opens up for other, future, opportunities and lessons learned
Hiring services from consultancies and third parties that after project end will “hand-off” to maintenance and operations to... another third party. I mean. Come on!
This is a “project as delivery & governance model” issue.
We can blame them for using such a model, yes, but the situation is not much better at most non-tech companies of moderate size.
I stated this to highlight the difference in goals of a democratic government and a private company - seeing as pensions got mentioned.
It is a ”governance” org type problem, sure, but this is prevalent in any big company.
Still sucks though.