procarch2019

In my…not too old, but not young years I am finding that I will quite often forget my phone at home. I find this to be of almost no consequence. I still carry a “wallet”, which is really just a stack of few cards. I live in a city, so if I really needed some info or had to make an emergency call I would just ask a neighbor (most people in my neighborhood are super nice). Most importantly, when I stop for a drink at a bar or I am at the playground with the kids I am forced, and this is a good thing, to not be checking social media/news aggregators/email.

I don’t think I could do without it during business hours because of the nature of my work, but it’s certainly fine for me to go without off hours.

Side note, I recently had a bunch of cash on hand for some reason. It is super convenient to grab a beer and just throw down a bill instead of waiting for a check. Digital pay services are great when splitting a check and are nsuper helpful when you are out of cash, but remember that cash, almost always*, works.

Understood, but the point I was making is that I would not associate the word hose with vacuum unless someone else makes the connection for me.

I guess the broader point I’m making is the same as others have said. It’s a poor measure of creativity as context is important. A word on its own doesn’t necessarily mean the same thing to you as it does to me.

Yes, agreed. I used the word hose, thinking garden hose, and vacuum, thinking of absence of matter. That specific pairing came to 47. I now see why based on the explanation of the test.

However, even if I had meant vacuum like the cleaning tool, I would not associate it with hose naturally because I use a stick vacuum.

I have to admit, as someone who rarely uses the Start button (everything from keyboard) I’ve noticed little difference between 11 and 10. I’ve only left the defaults because nothing has bothered me because I just don’t use it.

Thankfully they didn’t change the Win+x commands.

My biggest gripe with Win 11 is the stability. Reminds me of the early days of Win 10. Periodic blue screens on sleep and sometimes on boot.

Maybe it’s my hardware, but I doubt it. I was just given this laptop by IT and it didn’t come with Win 10 restore media, so this is what I have now.

I’ll preface this with “I think I’m good people.”

Context switching is hard especially when understaffed and most certainly when doing different types of work. I’m still “executing” engineering projects, but I’m also performing inside sales functions, helping advance and maintain our dev systems, and now manage 2 functional groups. Unfortunately stuff does fall by the way side all the time.

Prioritizing things is key, but at the end of the day there is a capacity to what one can do. It takes good management skills to realize the capacity of the employees and when things need to change. Luckily for me I’m working with groups in all three of the areas I was working in to offload those extra responsibilities to other people so I can focus on what I’m supposed to be doing (managing).

It would be interesting to see how they operate.

Yes, considering how fast they move I’m wondering if it’s closer to self regulated/peer quality control. Some QA/QC can make work come to a screeching halt in biopharma space (makes sense when a “batch” in the cell and gene therapy space is a single person and their wellbeing).

It’s not like regulation and quality have not fallen by the wayside or been too relaxed for other aerospace companies, especially in recent history. Quality should be meaningful and not just a rubber stamp.

It’s a fine line

I’m not sure that I took this away from the article. Looked like they just didn’t go into detail. They mentioned storing data for keeping track of equipment health. To me that sounds like a super broad stroke that would be equivalent to historized time series data as well as contextualized asset data.

I’m not familiar with the standards you called out (not an aerospace engineer), but I would imagine it’s similar to what we do in the controls engineering and automation space.

I’m actually really curious about what software/methods they use to contextualize assets and what those data points are. How do They digitize the equipment data that comes from visual inspections/manual testing.

I wish it was a little more detailed in general.

Yes, it makes perfect sense now, but after looking at the different radius's I will not run to the window when I hear blasts.

Internal documentation (2) …good luck finding it or finding something that is relevant still.

Documentation for customers (9) is usually clean, thorough and well maintained. Granted most of our customers are pharma, so heavily regulated.

It bothers me to no end that we can’t do for ourselves what we do for customers. It’s also short sited as many can waste significant hours piecing software, hardware and networking together because there’s nothing for them to start from. Someone tackling a problem or putting together a demo may go through the same frustration as someone did just a year ago.

Honestly, that’s dirt cheap for a PLC. I’d say that you should have a look on Craigslist or eBay, but you probably won’t find anything much cheaper. Also, a lot of PLCs are “cheaper” because they have relatively expensive software packages required for programming them.

You’d probably do much better with a RPi and a RTD module (1-wire, i2c, etc). You wouldn’t get the ladder logic, but you could use node-red. Heck, there might even be a ladder logic based program for the RPi. I wouldn’t be surprised.

We were looking at using something like this for a hacked together solution. The customer had field equipment that could not be replaced. It communicated back to a gateway via some protocol via 900MHz radio (from what I recall not one of the common ones). The customer found a new gateway that could aggregate the signals, but it only spit out some open TCP data stream. Our equipment is traditionally industrial, so it only supports industrial protocols (modbus, modbus tcp, EthernetIP, etc). It’s also not programmable at a low level (all drivers developed by the application developer).

We had used node-red in non-GMP environments before, but this was very much a GMP environment. What we figured out was that it would be impossible to qualify and validate the code base by any means that would satisfy QA.

So…great for labs and pre-manufacturing (development) where you have a bunch of skids or equipment that talk different protocols, but a difficult to use in any validated environment.

Now, being open source, it would be great if someone took the base application and built the other tools around it that would make it great for manufacturing where there are more strict regulations. Maybe Siemens is headed that way since they are rolling it.

Yup, sorry. Dug into it a bit more and saw this. Thanks for the direct link though

Love the follow up comments about other file systems. I can’t believe autocorrect happened so many times without notice.

One of the replies suggested it was done on purpose as to not get flagged. Is there validity there? I honestly am asking and don’t know why that would be needed.

Surprisingly enough you see this in other security driven, slow moving (static) industries.

Just look at industrial controls. At the base you can say “air gap the system and it’s pretty secure.” Then you add on reliability, maintainability, optimization, etc. The people responsible for maintaining the control system are not IT, so they deploy a concept OT (operations technology). Now OT and IT have to play nice to deliver business solutions that make everyone feel warm and fuzzy about stability and uptime, but in reality the overhead and complication actually diminish returns usually.

The clash between the two often results in an endless cycle of trying to achieve security and efficiency, but since the two groups are beholden to different incentives they are regularly not aligned.

This often leads to reimplementing solutions regularly in an endless churn. I’ve worked a couple times on projects to replace solutions that have not been fully developed.

TBF, hubs in general seem to fail regardless of brand. See my other comment about Dell hubs vs docking stations. A model of ‘hubs are cheap because they fail frequently’ is a poor one. Hubs should be cheap because they can’t deliver power and device connections or they have limited port types.

USB-C hubs that I’ve used have been garbage. They’ve all crapped out after less than a year or so of use. The Dell USB-C docking stations I’ve had have been flawless. Not sure what the diff is as my Dell puck USB-C hub sucked (went through 2 of them in a year), but the docking stations are solid.

The one difference I see is that the docking stations are meant to be independently powered and I think it has its own fan. I’m sure the internals are different too, but I don’t know how. Curious if anyone knows more about the difference between the hardware/firmware on the hubs vs the stations.

Yes. I agree with this whole heartedly. It’s usually an issue with not understanding the other side’s perspective. Experienced/good engineers who know the space they are operating in can get a better sense of what the customer wants even when the customer may fail to adequately express their needs. It always helps when there’s people on both sides who have clear visions and a means to communicate those ideas with their counterparts.

> A correct solution, even when not clean, isn't going to earn money

Was that meant to be ‘is’?

I’m not sure I follow how you came to that conclusion. I think their point is, paraphrased, spaghetti code that meets the requirements, schedule and budget is better than pretty code that functions, but does not meet one or more of the aforementioned. That’s not to say that only spaghetti code can make a good project, but to illustrate the point.

Sure, sometimes everything works out where you can refine something to the point where it’s a golden (or near perfect) solution, but any solution that meets business requirement x is better than one that does not at all.

I’m not sure if it really comes down to this, but my standards for internal tools are much higher than things developed on the customer facing side. I think I mostly attribute that to the fact that we know how to successfully develop our user/business requirements versus our customers who do not. Hell, I have customers who have asked us to write their requirements. That’s a huge red flag for an unsuccessful solution.

My first boss once asked me in a performance review if I was a perfectionist. I was and told him so. He then explained something that left an impression on me: the value of shipping something that works even if it’s not elegant. I think there’s a happy medium between getting stuff done and getting stuff done ‘the right way.’

A lot of young engineers seem to miss the forest for the trees and this can be seen early in a project’s timeframe. Sometimes customers don’t help either because they lack the vision to know what they are really asking for as well. I know there’s a problem when people start suggesting solutions, elegant or otherwise, before the user requirements are fully fleshed out.

I work in an industry where CI/CD isn’t really a thing due to qualification/validation, so getting things perfect for re-using or servicing codebase is really inconsequential, but you’ll see people fall into the trap of ‘this has to be genericized 100% so we can use it everywhere.’ In reality nothing really gets re-used in such a way that it’s effective in reducing effort. I’m not saying it’s that way for everyone, but in my company’s pseudo programming space this is true.

Interesting, I'll have to take a look at yubikey. I just installed a Tofino firewall with the Modbus Enforcer LSM between one of our DCS and accompanying SIS systems. We have never had a system communicate process data directly up the networks except through OPC (mostly DA, which is even more problematic for firewalls). Luckily OPC UA is now natively supported on our application, so things are starting to move in that direction.

Luckily a lot of our customers use PI, so we install the PI OPC interface on the application layer and only PI ports need to be opened to the next level.

Even more so the vendor we work with, Emerson, even has IPD firewalls to go between the DCS computers (engineering, historians, operator stations) and the I/O (what we refer to as level 2). The price tag can really jump when you implement all these security features, but an argument can easily be made that it's worth it when you consider some of our customers run batches that can be worth $500K or more per batch.

Yea, that is correct. I typically put together the solutions for new systems, including security. I give the sales team part numbers and hours for security software and related hardware. They then add that as an option to quotes. No principal automation engineer wants to take that on and no IT want to be involved. Also, when money is tight that’s an easy target for them to pass on.

Luckily I’ve pushed enough over the years that we at least include A/V software as mandatory.

I’ve been able to carve out a nice space within my company bridging the IT/OT divide. It’s been particularly good recently since the bigger companies are dictating good cyber practices, but rely on integrators and vendors to implement.

I don’t think there will ever be an out of the box solution unless a system stands on its own, which is becoming increasingly harder with modernization and reliability efforts. Add on top of that privileged access, remote monitoring and support, automated (kind of) patching, etc. you have to interface with the IT side a bit.

Some time series data interfaces only work with tcp comms, which means you can’t always rely on unidirectional networks. I agree you should use them where possible though.

I replied to a comment on a dupe post regarding PAT, in which analysis is done on process data and fed back into the process to increase efficiency or yield. Obviously there are varying levels of criticality where the risk vs the business reward might not be worth it though.

That's massively inconvenient, although I'm sure necessary in some cases. Some businesses actually perform analysis in 'real time' so they can adjust the process accordingly, witch requires that data be accessible. This may actually be such a case as I'm sure they have to interface with customers (tank farms) to react to supply/demand on the branches. For all I know Colonial does have a private network for that purpose though. Usually PAT is really for chemical processes where you are looking for a particular yield and those analytical services are located closer to the process (in terms of networks).

There are devices called data diodes that provide unidirectional network topology, but not all time series data interfaces can work with them.

All in all, I agree that total air gap is obviously the best way to mitigate network attack vectors, but sometimes not practical. No controlling device should be at level 3 or 4 though (business or enterprise level).

I think the issue there is data, even on critical infra. Modernization, reliability and the such require data analysis. There are definitely ‘strong’ ways of protecting the assets and mitigating attack vectors, but almost no way to eliminate them entirely. For example, event if you isolate the process computers you’ll typically have an interface node that presents the data up a level (hopefully to a DMZ). Obviously you can be compromised if that interface node is.

Some critical infra is air gapped though. Other systems implement SIS systems in parallel with general process systems to mitigate catastrophic failure further.

It is mind boggling the lack of basic security principles some people have. I won't just put that on the plants and their IT/OT, or lack thereof. I've seen plenty of vendors and integrators do some cringe worthy stuff too.

I get what you're saying and that could very well be the case, but I think the 'pipeline' as a whole requires a lot of handshaking between the different stations. They would not be able to do this without their supervisory control later (or at least it would be particularly difficult). That alone could have caused them to shut it down.

Additionally, if there was a whiff of malicious software or unintended access I would imagine they would want to make sure it didn't get into other systems. That would involve isolating and possibly shutting down machines and equipment.

I guess we'll see when they release more information. I would imagine that we'll get more details since this is critical infrastructure.

Totally agree, see it all the time. I even know of a few NT systems floating around out there. At least most companies are getting their IT involved to mitigate (usually they work with the vendor because they know nothing about control systems). They usually provide funding to the automation groups. People are starting to take it seriously.

I work in control systems OT space. A lot of distributed control systems and scada systems interface with the business layer in some fashion to provide access to time series and event data and to allow for alerts via email/mobile. Some people do this properly with good network segmentation, firewalls, A/V and patching, etc (there are several standards that dictate best practice). That said, even when doing it properly you're introducing attack vectors. I don't think it would be a firmware vulnerability, but instead something malicious affecting the computers they use to control the process.