Comment by tptacek | Hacker News Reader

Comment by tptacek | Hacker News Reader

Parent Full thread

I've never spoken to anybody in management at NCC about why they fired you, but your coworkers have told stories about it, and I'm not sure you want the conversation you're asking to have. It is not the same story you tell.

I don't know why you're shocked at the number of times I "showed up" at the NCC Chicago office. I'll say it again: you and I never worked together. NCC hired you several months after I left. I know this because I still have the email thread of you asking me about the interview process. How often do you show up at ex-employer offices?

Wally was, at the time, your line manager at NCC. You get what a line manager is, right? Nobody was seriously asking Wally for the rights to anything, but I have no trouble believing you have trouble interpreting a joke.

You just claimed, a comment earlier, that NCC engagements were "a week, max". I stand by my previous comment. I have better information, and you have weird information. If your personal experience of NCC was a back-to-back sequence of 2-day projects, you were put in some strange back-bench situation.

I left Matasano and almost immediately started another consultancy, and I'll said it again: unless things have drastically changed since 2020, when I left for Fly.io, the modal pentest engagement is 3 person-weeks, not 2 days. People do not in fact spend 50% of their time writing reports.

We can compare notes if you like, but I don't think it's going to make you any happier to do so.

A moment later

I'm just sitting here thinking about this and your claim about documentation is even more risible than I had realized. By the time you were working at NCC, documentation was almost fully automated; we had the whole team working with Litany of Failure, our docgen system. Litany became such a big deal after I left that a year ago, at someone's going-away party, Erin and I got pins, which they'd made for everyone who worked on it. You were sure as shit using it in 2014.

By the time you were working at NCC, the experience of documenting a pentest was filing bugs in a bug tracker that autogenerated descriptions of things like XSS, and then pushing a button to have a PDF pop up.

50% of your time. Give me a break.

I spoke with someone who worked contemporaneously with you, and who thinks you might just be confused --- your "projects last 2 days and typically consist of 50% documentation" claim would be consistent with you working SARs rather than pentests. What you're describing does sound a lot more like arch review than pentesting. Maybe that's the miscommunication here?

Edit

Somebody else pointed out that if you were mostly working on retests --- that's the project where you take someone else's report and look to see if they fixed all the findings --- you'd also have had a 1-3 day documentation-intensive workload.

Another person pointed out that netpen projects fit this bill too, but of course, you weren't doing back-to-back runs of network penetration tests out of that office.

All I care about is the claim upthread that people spend 50% of their time on pentests documenting things. If you engage a firm to do a software assessment and they spend 50% of their time in doc, fire them.

As a fellow YC founder, just wanted to pipe in to say this whole thread really isn’t a good look.

Whether or not the employee was right or wrong, it’s probably best to just leave and accept the conversation where it is.

Edit: changed “your employee” to “the employee”

This person was not my employee. Even if I had been at the firm at the same time, he still wouldn't have been my employee.

It bothers me that they've continued to claim over the years that they were "fired for narcolepsy". The NCC I'm familiar with paid multiple people for years who weren't able to deliver work because of health issues, and did so without blinking. There's not a lot that I especially like about NCC management, but their handling of health and disability issues would have been at the bottom of my list of complaints.

Edited my comment to say “the employee” instead of “your employee”.

Either way, this thread really isn’t a very good look for you or the person you’re responding to.

Sometimes not engaging is better than engaging. (And here is when I myself will stop engaging…)

> It bothers me that they've continued to claim over the years that they were "fired for narcolepsy". The NCC I'm familiar with paid multiple people for years who weren't able to deliver work because of health issues, and did so without blinking.

Maybe they've got special policies on health. I am not impressed with NCC Group's firing policies. I was fired because, as far as I can see, I was recruited to the bug bounty team with an offered perk (work from anywhere), I said I wanted the perk, they said no problem, my boss was promoted, and the new boss didn't want to allow the perk. He made repeated comments to the effect that he wasn't comfortable having me on his team after I'd made a request that he wasn't willing to grant immediately. He also told me that, if the worst came to the worst, I would be transferred back to active consultant status rather than being fired, which didn't happen.

I am certainly not sticking up for NCC's policies about firing, or even Matasano's.

Your ass is showing bro.

It's fine to challenge other users in a thoughtful way where you think they're wrong. "Your ass is showing bro" is not that.

We ban accounts that degenerate to that level, so please don't do it again.

https://news.ycombinator.com/newsguidelines.html

Acknowledged.

> your coworkers have told stories about it, and I'm not sure you want the conversation you're asking to have. It is not the same story you tell.

I hereby give you permission to lay out the full story, full-stop. Don't hold back. I know it's usually not a classy move, but you're hereby absolved of that.

> I don't know why you're shocked at the number of times I "showed up" at the NCC Chicago office.

I wasn't shocked. It was to point out that you weren't in the field anymore. We were.

> Wally was, at the time, your line manager at NCC. You get what a line manager is, right? Nobody was seriously asking Wally for the rights to anything, but I have no trouble believing you have trouble interpreting a joke.

No doubt. But that raises the question of why you met with Wally privately. That's a bit at odds with your immediate previous claim that there's no reason to show up to your ex-employer's office. But this is just a distraction.

> You just claimed, a comment earlier, that NCC engagements were "a week, max". I stand by my previous comment. I have better information, and you have weird information. If your personal experience of NCC was a back-to-back sequence of 2-day projects, you were put in some strange back-bench situation.

Pop quiz: If I worked there for a full year -- 52 weeks -- then how did I complete over 40 engagements?

> I'm just sitting here thinking about this and your claim about documentation is even more risible than I had realized. By the time you were working at NCC, documentation was almost fully automated; we had the whole team working with Litany of Failure, our docgen system. Litany became such a big deal after I left that a year ago, at someone's going-away party, Erin and I got pins, which they'd made for everyone who worked on it. You were sure as shit using it in 2014.

Yes, we used Litany exclusively for our docgen. That's the templating system that creates the PDFs. It's quite a nice system; thanks for making it.

It doesn't change the fact that you have to actually fill it out with words.

> 50% of your time. Give me a break.

We did.

So, let's hear those stories. I imagine it'll go something like this: "That Shawn was such a slacker that we couldn't figure out what to do with him. He spent most of his time fooling around on the computer. At one point he went to the bathroom for a full half hour."

My work spoke for itself. Pentesting was a necessary part of the job, but the product is the report. The report is what the client sees, and (in some cases) what they pay several hundred thousand dollars for. Is it any surprise that a huge amount of time is spent preparing this extremely-valuable product for consumption? This isn't even a particularly strange claim; Damien was the one who pointed out to me that the PDF is what clients are paying for.

I'd be interested to compare notes. What did you have in mind?

See upthread.

I think these claims are pretty much risible. The report you're talking about is autogenerated from a bug tracker. The client sees your bugs. If you spend 50% of your time writing them, you're cheating the client.

What I really think happens is that you misinterpret things people tell you and blow them into weird directions. Somebody told you that "the PDF is what clients are paying for". Well, no shit. The PDF is the only deliverable from the project. It's where the bugs are written down.

I wasn't there for whatever you got told, but it sounds to me like the subtext of it was probably "so it doesn't matter much what you do on a project as long as the client ends up with a report that they can use to claim they've had an assessment done". That's a cynical thing to say, but definitely a thing that gets said. It's also, strictly speaking, true.

What I hear you saying is, "the PDF is the only thing that matters, so we should spend most of our time making the best possible PDF". That's not only silly, but also actively difficult to do in a practice where the reports are autogenerated.

The actual figure of merit from a software pentest is the list of bugs, full stop. Yes, the list is delivered in PDF. Don't let that confuse you.

I don't think you've worked in this field seriously enough or long enough to use the word "we" the way you are.

Can confirm a 2 day engagement is unusual, and 50% of time writing the report is possible but very much an outlier for standard pen tests. Some interesting exceptions include:

* Some regions have a much shorter average engagement time. North America is usually pretty generous, where markets in other countries will only bear half or a third of the time.

* If you are a junior or less skilled you are perhaps more likely to get the small jobs while you are learning.

* External inf can be short on testing time and long in reporting if you find lots of issues, but automation helps the reporting in that regard.

* Some pentests are very documentation intense for specific reasons, such as M&A due diligence, or clients who want threat models and design reviews incuded. Still isn't 50% though.

And others. But in general what Thomas describes has been my experience over the years.

Disclaimer: I work for NCC, but nothing related to former Matasano and I don't know Thomas. Opinions are my own.

Thank you. (And thanks for being dispassionate; it's a nice change.)

It sounds like the most likely explanation is that Matasano was an outlier. My career was cut short before I had a window into the rest of the pentesting world, but it's good to hear that places exist that aren't so obsessive about the actual writing process. I also happened to experience most of your list, so it sounds like it was an exceptional situation in general, so it's best not to draw sweeping conclusions from it.

Cheers!

It's interesting to read about other philosophies for engagements. In the places I've worked it would be rare to send a junior engineer on a short engagement. The reason being that short engagements are usually 1 engineer, maybe 2. There are always tools and tests that take time and it's better to have 1 engineer for 2 days than 2 engineers for 1 day. We'd send our junior engineers on the multiweek engagements so they'd learn more. They'd get a chance to encounter all types of systems and networks, and would be able to see how the senior engineers approach problems. We could even leave them to figure out complex topics on their own in some cases (and often they'd teach us new things in the process!).

But as I said in another comment, depending on what people consider to include as "report writing" I can definitely see some engagements needing 50% time there. So maybe this person did just get unlucky.

Sub-week software pentest engagements at established firms are pretty rare. There's a logistical reason for that: engagements are overwhelmingly measured in person/weeks, and if you book out a consultant for two days, you fuck the schedule for the rest of that person's week. It's the same reason (or one of them) that you shouldn't bill hourly if you do your own consulting work: if a client books you for a couple hours in a day, they've fucked the rest of the day for you.

A 1 person-week engagement is pretty short. On a 1 p/w engagement, you'll have scoped back drastically what you can test; maybe one functional area of a smallish web app, or, every once in awhile, you'll get a big client that has the budget flexibility to do things like book "one week of just looking for SQLI and nothing else across all our internal web apps".

The typical CRUD app for a small tech company would tend to come in between 3-4 person weeks. Sometimes, those engagements would have their last 2 days explicitly reserved for doc in the SOW. I felt like (still feel like) that's rustproofing; clients are paying for testing, not writing. Usually there's a couple days of "discovery" at the beginning. The rest of it is just testing.

The typical order of a project with a public report (those are pretty infrequent) is that the public report is done after the the original test is accepted. That's in part because clients want to triage and remediate findings before they release a public report; you sort of can't drop a public report and the internal report at the same time. So public report writing shouldn't have much of an impact on the project delivery schedule, because it's not done at the same time.

For sure, a short engagement of 1-2 days would be rare. We'd occasionally do them to get a foot in the door or as a follow-up for a regular customer. We'd still not want a junior engineer on them. You want to make as good of an impression as you can so you get the bigger contracts and you don't want someone there who isn't experienced communicating with clients.

Yeah, that's probably true.

But, per the thread, there are some special-case projects that are short and do take junior delivery staff; SARs, which are "pentest the documentation" projects meant to derive a thread model and inform a later, real, test (I don't like SARs and think they're a kind of consulting rustproofing, but people smarter than me disagree strongly with this), and, of course, retests.

It's an interesting tactic to set up a strawman and then beat it up. Where am I to start when the reply will mostly be "That's not true" or "I didn't say that"? This is devolving into being boring for the audience, but you're (for some reason) attacking my reputation. You're also backpedaling; I thought you were going to tell stories? I'd like to hear them. Or did you check with someone, and they said "Um, actually, Shawn wasn't that bad"?

> I think these claims are pretty much risible. The report you're talking about is autogenerated from a bug tracker. The client sees your bugs. If you spend 50% of your time writing them, you're cheating the client.

You keep saying the report is autogenerated because Litany existed. This is a bit like claiming that scientific papers are autogenerated because LaTeX exists. Yes, it does a lot of heavy lifting. No, it doesn't change the fact that you start with a template and then rework it into the proper form. As any grad student will tell you, that work takes a lot of time. My experience at Matasano was similar. I bet Drew, Andy, Damien, Dmitri, and a few other folks would at least say that reporting occupied a significant chunk of time.

From where I'm sitting, the claim seems unremarkable. Look at how long this thread's security assessment is. Most of the words are boilerplate, but you can't simply ship boilerplate. And yeah, the reports went through multiple rounds of back-and-forth before they got shipped, during which every detail was combed over.

> What I really think happens is that you misinterpret things people tell you and blow them into weird directions. Somebody told you that "the PDF is what clients are paying for". Well, no shit. The PDF is the only deliverable from the project. It's where the bugs are written down.

It wasn't "someone." Damien was one of the most experienced pentesters at Matasano. He led several redteam projects, and taught me a lot of interesting tricks for sneaking your way into a network.

> I wasn't there for whatever you got told, but it sounds to me like the subtext of it was probably "so it doesn't matter much what you do on a project as long as the client ends up with a report that they can use to claim they've had an assessment done". That's a cynical thing to say, but definitely a thing that gets said. It's also, strictly speaking, true.

This is a strawman. What he was saying was that we need to do a good job on the report, in addition to the hacking. I don't know why you'd spin it into some cynical thing, but at least you're consistent.

> The actual figure of merit from a software pentest is the list of bugs, full stop. Yes, the list is delivered in PDF. Don't let that confuse you.

We can debate who's the one confused, but at this point it's pretty clear that our experiences were dramatically different. What possible benefit would it be to me to sit here and lie about it? Not only would you (and everyone else) call me out, but I'd have to keep making up more and more elaborate falsehoods.

Sounds exhausting. I'm just reporting what I saw, and what I did.

I don't see a productive way to continue this. Your position is that "nobody spends 50% of their time on reports." I'll concede that maybe it's closer to 40%. But it's certainly not 10%, or whatever small amount you're hinting at. And as you pointed out, my last month was filled with 100% documentation-writing.

Let's agree to disagree and move on.

I'm comfortable with what the thread says about how we came at this disagreement.

As you've acknowledged upthread: your claim that documentation is 50% of the time on a pentest doesn't hold up. I believe it took 50% of your time, because you say it did, but like you said upthread: it's was an exceptional case. Maybe:

1. You spent more time on doc than was actually required

2. You worked a bunch of SARs, which are short doc-focused projects (and not pentests)

3. You were given a bunch of retests, which are short doc-focused projects; this almost fits, since it's some of the first work that's given to new team members after they're done shadowing on projects, except that it would be weird for there to be so much retest work that you could do them back-to-back for a long period of time (most clients don't purchase retests)

4. You worked ENPTs (external netpens), which are short projects and have a huge doc component of "fitting tool results into a report". But (a) your office didn't do a lot of those (Matasano in general didn't do a lot of netpen work) and (b) it wasn't low-seniority work; as I recall, the people who did netpens specialized in it.

It could be some combination of all these factors.

Meanwhile: Litany is nothing at all like LaTeX (though after I left LaTeX staged a coup and replaced the commercial PDF library it had been using). Litany is a bug tracker. You enter a finding title, a URL/location, and a description --- which Litany autogenerates for common bug classes --- and when you're done with the project you push a button and get a complete PDF. This would have been the primary way all reporting was done during your tenure.

Interestingly, one of the few major disputes between Matasano and iSec Partners (the big two constituent firms in NCC US, iSec bigger by a factor of almost 2) was report generation. iSec's actually had a LaTeX template consultants would use to generate reports; they wrote them by hand. "I refuse to use Litany" (and lose control over my report formatting) was supposedly such a big thing that they had to rename Litany when they re-introduced it across the firm; it's now called Limb. Which is a tragedy, because Litany of Failure is one of the all-time great product names (credit, I believe, to Craig Brozefsky).

Some of the people you've mentioned in this thread have reached out to me. I stand by literally everything I've said. It's OK to be wrong, and you are.

The rest of it:

d0136ada7858ad143675c2c3302db6343869190878191c8fc6a1547f0f23530e

Just in case this ever comes up again, I'll at least commit to not having changed my story. :)

Long story short: don't pay for software pentests that spend 50% of their time in doc. You're paying for the testing, not the report, even if the report is all you ultimately care about.

Reply on news.ycombinator.com