When interviewing senior people, I like to get them going on war stories. "Can you talk about a project you experienced that went very badly?" is a good one. It's not necessarily that they went badly, but we've all been on death marches before (and I absolutely would not hire an architect who's never done a death march!).
The beauty of this is you can find out how they react to impossible demands and pressures. What they're proud of accomplishing in dire circumstances. Possibly what they're ashamed or embarrassed about, or mistakes/lessons learned (if they're that bold). You can learn a ton about how they talk about their colleagues! Are they full of praise, or contempt? Do they find someone to blame, or talk about how someone else saved their asses? You can also learn a lot about their actual approach to technical decisions - their taste, for lack of a better word.
You can ask probing questions here, but the important part is to get them speaking in an unguarded way. Nobody makes it to the architect level in this field without at least kind of loving the job. Find out what they love. Find out what they hate. Find out what they regret. Find out any strong technical biases they have, and why. But most of all, find out how they play with others. Because the architect's job isn't just to design, but to communicate. You don't just want a technical expert. You want someone that can value and respect the rest of the team, or they're just going to cause problems.
Likely outcome to meet the criteria is burn-out / resignation / death of personnel, especially if repeated.
In the original context, it is of course: https://en.wikipedia.org/wiki/Bataan_Death_March
https://en.m.wikipedia.org/wiki/Death_march_(project_managem...
The source:
https://www.amazon.com/Death-March-2nd-Edward-Yourdon/dp/013...
I agree that death marches are counterproductive. However, if you've been around long enough you've run across situations where they are unfortunately necessary. Another example: production goes down Friday at 5pm. Do you say "we'll pick this up Monday morning"? No... it's all hands on deck to get production back up working as long as it takes, assuming your company isn't large enough to have its own dedicated 24/7 incident response team. Hopefully there's a lot of process and automation which helps to minimize the effort, and if not that's a great takeaway to improving the process down the road.
So then the question is actually asking, "have you ever worked at a company that was two weeks away from complete failure and bankruptcy?"
> Another example: production goes down Friday at 5pm.
While highly stressful, this is not what a death march is.
> A lot of hard deadlines are made up.
Yes, but not all of them are. A lot of my clients face legislative deadlines that don't take developer comfort or work-life balance into account. We need to help them ship X by January Y. It's rare that overtime happens but it has happened and there's little you can do to get around it when a bill is signed into law mandating you do something in 30 days or the state becomes a litigation target.
The worst death march I was ever on was against a legislative deadline. Oh god, may I never do that again.
This also happened to be during a hiring freeze, so no new contractors, no temp employees, nothing. Current team of half a dozen or so had to drop everything they were doing, switch over to this project and work ~12 hour days six days a week just to deliver a mostly-broken implementation right at the deadline and spend another week and a half at the same schedule fixing bugs and other issues.
It would have been much worse if the timelines were extended, if it was 6 months worth of work to be done in 90 days I think the team probably would have quit. They were mostly contractors so paid hourly, so they got paid for all the work, but that's still a lot of hours for that many weeks in a row.
I have been very lucky not to come across anything like that yet.
There are many failure modes other than "death march". I'd be shocked if an architect doesn't have at least one good answer to "talk about a project that went badly". Or, failing that, "almost went badly".
> I absolutely would not hire an architect who's never done a death march
> "Can you talk about a project you experienced that went very badly?"
And that question, as phrased, seems quite reasonable to ask any prospective architect. The answer tells you quite a bit about how they handled themselves, how they worked with their team, how they talk about their project and team, and how they diagnosed and addressed the situation.
An architect role isn't just a technical role. It's a leadership role. An architect who cracks under pressure easily is a liability for the entire team. If someone has never had to work when they know they're going to lose, they have no experience with it, and don't know what to do when it happens.
[EDIT] remove word that didn't need to be there.
I'd like to offer a spin on this.
In my experience, asking a question that demands one to recall specific details about a painful event in an interview, which is typically a stressful situation, may produce poor results.
What do you think about giving the candidate an opportunity to get advance notice of that you'll be asking that question, in an interview, and ask them to prepare three to five bullet points to discuss? Whether the candidate sends the bullet points back to you before the interview, or just brings them to the interview, is beyond the scope of what I'm proposing.
Let's say your trying to assess a 3 things:
* Has Architect been through the meat grinder at all? Bullet points with little depth log a warning, as there has been plenty of time to prepare a thoughtful answer.
* Does Architect talk about failure in a calm and composed manner? Bullet points give ability to ability to recall challenges before stress of interview sets in.
Does Architect trash talk former employer or coworkers? The stress of interview can erode one's asshole filter, and cause them to be a bit of the jerk they've worked really hard to not be (speaking a bit for myself here).
To wrap this up, I like interview questions that are functional with minimal side effects, meaning the person is given every opportunity to provide the answer you are looking without external pressure or distraction. If you want to see how a person handles stress, give them a situation that is stressful in a way it'd be on the job, perhaps leading with "apologies in advance for the following situation, which is intentionally stressful..."
Depends on the level you want to interview for. For a higher-level position, I don't think it's unreasonable to rely on someone's communication and presentation skills to adapt on the fly. Explicitly tell them that they can take a minute to think about it, rather than having to have an instant response.
Now, I do my level best to make interviews unstressful. I don't think a stressed candidate is going to do their best, or be their best. They're going to be second-guessing their answers.
That's the point of war stories. Every experienced candidate has them. We've all done memorable things in our career, both good and bad. If it sticks with you, the details aren't what matter.
edit: This goes both ways. I expect the war stories to be an exchange, a patter. They should be learning about me and my co-interviewers as well. And, as someone who gets interviewed, if I feel the interview is completely one-sided and not revealing anything about potential colleagues and the working environment, I'm going to hesitate.
Me, I'm always good for that kind of conversation. It's not going to stress me out if they're asking me things like this.
Stressful and painful are not the same. Every engineer is going to go through stressful situations. Good ones will learn from them: to minimize the risk of them happening, and to better cope when they do.
> If you want to see how a person handles stress, give them a situation that is stressful in a way it'd be on the job
How is this any better?
> "apologies in advance for the following situation, which is intentionally stressful..."
Stressful situations don't send calendar invites.
I'm not arguing one should be an asshole during the interview. I've been interviewed by clearly hostile engineers. I didn't like it, and it surely impacted my performance. But real life is like that. Being able to react in a polite and composed manner to an unexpected situation, and recover from it, is a superb data point in an interview.
Mainly because you're paying someone to do a job, not to interview. I've interviewed a lot of people, and sometimes interviewing just sucks for reasons that have nothing to do with the interview.
For example, say you're homeless, and you get your first interview in quite a while. Landing the job means housing for you. The pressure to perform is enormous, and it's not something you can prepare for. I was homeless for a bit, and in that situation.
I don't expect to get a job for which I'm not qualified, but it'd be nice if an interview process allows me to show I can meet the requirements of the position, regardless of the circumstances of my life situation outside of work.
I like screening people in for the right reasons, and not screen them out for reasons that aren't really all that important. And when you have a hard time getting any qualified applicants, can you really afford to false negatives?
I agree, having someone be clearly hostile to you during an interview (which is often the first impression of what the culture at a particular company is like) is a great data point for the decision on whether to continue with the process of interviewing there.
People tend to slide into a comfort zone and open up a bit about their successes. They then -tend- to stay open and offer up useful insights into their past behaviors under pressure. I've found that it helps to ask the "success" question first, then ask the "failure" question second, rather than directly asking about someone's failures.
As an aside, the technique outlined above is something taken from "Influence: The Psychology of Persuasion," [1] and is a useful read to recognize situations where someone might be using certain techniques to influence the outcome of a situation. :-) Also - note that the general prompt above expects someone to answer in the "STAR" technique. [2]
[1] https://www.amazon.com/gp/product/B002BD2UUC/
[2] https://careerservices.wayne.edu/behavioralinterviewinfo.pdf
I've been doing this and advocating for the practice for years. What is with the "gotcha" interview?
To some extent, yes. But the flip side is that for candidates that have had negative experiences in poorly managed environments (i.e. negative experience with past employers) this allows them to vent this information in a somewhat more clinical (neutral) way.
How do you align this with the "don't talk shit about previous employers" rule?
One of my most difficult experiences was the very last day on a job, when an "Everything is going to crash in four hours if you don't fix this and you were the last person to touch the code" bug appeared (we would have run out of disk space). It turned out the bug wasn't mine - it had been latent in the code for years, caused by the difference between NULL and 0 in a C pointer. The changes I had made just made it visible. Terrifying, stressful, and I'm kind of proud I was able to resolve it under such stress.
See? Terrible situation, not a single bad thing to say about my employer, my co-workers, or my own work. And none of those were actually the root cause of the problem.
What? In C pointers, `NULL` is equivalent to `(void *) 0`, is it not?
Were you compiling in a debug mode where your compiler reinterprets NULL as a bad pointer value other than literal 0?
Technical detail in this case... we were creating rows with a BLOB column in Sybase. If you passed in NULL, Sybase did not allocate space for the blob. But if you passed in 0, it allocated 8k.
We had been storing data in the blobs, in the dozens-to-hundreds of k size range. Sometimes we didn't store data in a row, but mostly we did. The data got moved to the filesystem, resulting in a huge performance boost. Because we pulled the blobs out of Sybase, we massively downsized the databases, to about 10% of their original size - which was still a few times larger than what was needed for an entire production season (about three months of data).
Now, this is where I badmouth co-workers... I had argued strenuously for removing that BLOB column completely, as it smelled wrong to me to leave it. But the tech lead, a rather nervous man who was always afraid of things breaking unexpectedly, insisted we leave it there, because removing it might break something else (I wouldn't be introduced to the Lava Flow Antipattern until years later...) (also, welcome to the world of coding in C without testing frameworks).
The code worked fine in testing. Beautifully. But testing couldn't even fill 1% of a database. The wasted space from the BLOB allocation was invisible... until production started, and rows were 100 times larger than they should have been on our biggest table. Oops.
Thankfully, Sybase has excellent documentation. The gleaming row of carefully indexed, spiral-bound manuals helped me find detailed example code quicky, which got me to the latent bug. We'd never noticed the 8k being allocated for empty blobs before because they were swamped by the size of the actual blobs.
Hmmm, no, sorry. `NULL` is #defined as `((void )0)`. There is no 99% or other probability involved.
Sorry to be so pedantic as it's kind of an aside to the discussion at hand. I thought perhaps you knew something about NULL in C that I never knew and was eager to be enlightened.
I think you're conflating C pointer `NULL` with the database concept of `NULL` which is not necessarily represented the same way when interfacing with database client APIs.
There's probably a detail you glossed over whereby you're passing 0 vs NULL via separate method calls to the database client API. The distinction cannot be due to literal `NULL` vs. literal `0` purely in C programming terms.
The limitations of C require that database-NULL be represented differently from integral values in terms of data structures. Even if you do pass `NULL` and `0` in the exact same parameter location to the exact same method name, you'd have to ignore a warning about the implicit type conversion from `void ` to `int` and still the semantics do not change and you have no runtime bug as you described.
The middle of the interview should be reserved for a bit of uncomfortable talk where you talk to him about something that went bad. Thrash some of his ideas, question a few approaches and let him react to it. I also try to make it point to make him work on a situation where he doesn't have control over a few variables(like the stack, or the timelines, or the people or the requirements).
I keep the last few minutes for agreement, the conversation should leave a happy memory after all.
For each theme on the list; make 3~5 variants of things which show the insight(s) related to the theme at varying levels of quality/discernment/depth/etc. Assign an appropriate numerical weight to that tier of answer. Your list is now two-dimensional; in teaching circles it would be called "a rubric."
Grade interviews against the rubric. On security, this candidate said X, Y, and Z, which are elements which most closely match a Good answer; 7 of 10 possible points awarded.
Even better, grade written notes taken during interviews against the rubric.
Now, and this is the brutally organizationally difficult part, stop overriding the rubric when it spits out point values which imply next steps regarding a particular candidate that are clearly not what one wanted to do. Instead, fix the scenario and/or rubric.
My least favorite interview ever, was essentially "Implement this generic problem that our team had to solve, but you don't have the context of our infrastructure, and I'll criticize you every time you deviate from the actual solution we came up with because your solution wouldn't work in our context". It really felt like a "read my mind" type of problem.
No further point. Just glad to see you point out the relatively obvious point that you need to evaluate a question for someone who doesn't have your current context.
Definitely rough on candidates, but it does give a reasonable window into their thought processes and who the interact with other people.
However, it also feels quite artificial and stressful, which is a bad combo for a lot of people. Unless the position actually requires rapid fire system design and architecture, I'd be cautious using it as a primary filter. For most positions, designing a system architecture includes include some amount of research, reflection, iteration, etc.
In other words, you're evaluating something the job doesn't actually require (typically), and you're potentially missing out on people who might be brilliant architects given a little time to absorb the problem and think through options.
1) Explain to me a system you built/architected. How it worked, what it did, etc. I'm looking for a deep understanding around the requirements of the business, and how that played into technical choices.
2) Talk me through some bugs you encountered or mistakes you made, and how you course corrected. I try to make a candidate comfortable prior to asking this and usually tell a quick and horrifying war story from my own career. Here I want to understand how a candidate processes information and adapts the plan accordingly. Like mike tyson said, everyone's got a plan until they get hit.
3) If you had to build the system again today, what would you do differently? I'm looking for an awareness of how technology has evolved since they designed the system in question, as well as what didn't work well over time from an operational perspective (e.g. we didn't burn off time series data over time so table index rebuilds started to take forever, etc). I want to make sure the candidate supported the system they designed over time, and felt the pain/benefit of their decisions. I usually specifically ask the candidate at the beginning to pick a system they built and supported for an extended period of time.
I also ask about areas of technical expertise, and areas where the candidate feels less strong technically. For the former, I make sure they have the kind of expertise they claim, and for the latter, I want to see how they communicate with team members that are expert in those areas (and how they gain familiarity with new technologies).
To your list I'd also add some human factors. Some behavioral interview questions about dealing with difficult people, entrenched opinions, people under stress.
You've also got questions such as "what is your favorite/last book on this domain and what was your take away". Or more generally what you like to read and how you stay in the loop.
I try to look forward and not dwell on the past. Or maybe that's how I rationalize bad memory. Either way, when asked about details of something I worked on 8 years ago, I don't have much to say.
I still would struggle if you asked too many more details beyond that, though. I had to dig deep into my brain in order to write those paragraphs in the first place.
I do the same with technical questions. Anything I get asked (that I remember afterwards) goes into a document (with paragraph answers) that I review before future interviews. I also try to anticipate some other questions and put them in there as well.
Because the scope of Computer Science is so big I still get questions I didn't prepare for asked of me pretty much every interview, though, and I struggle through those questions.
Despite this, I do have a good memory for work and can keep entire systems in my brain. But in doing so this other crap gets shoved deeper and deeper and are more difficult to retrieve.
If you don't have any projects or work to talk about, the interview is basically over as I don't see much value in talking about cookie cutter problems.
A lot of "architects" will only repeat generalities and have nothing left to go deeper. I also don't like too many buzzwords like "integration patterns" or "serverless architecture".
Happens all the time, a person wants to buy a waterproof computer, but really what they want is to see the results of a program while they are in a rainstorm. Or they want a "website for managing inventory" but they really want a way of tracking employee theft. So architects listen to the problem statement(s) and think about what is in play, will ask some clarifying questions and then reframe as the actual problem statement. A senior engineer will ask evaluation level questions (how fast, how much memory, etc) and come up respond with a solution to the stated problem.
A bad software architect will not be able to stop abstracting and will start with a 50,000' view and then float into orbit. A "good" software architect will add in the business or operational constraints to the problem and articulate where the ceiling is and why.
I am glad that you brought this up. My question was more from a perspective of distinguishing senior engineer vs architect interview questions. There is only a limited time in the interview. Shouldn't an architect be also asked the senior engineer questions in addition to the ones discussed here. Especially my our org setup they are also hands-on and individual contributor to the code base. One solution could be to restrict the 1st couple of rounds of an architect interview to senior engineer questions only . Fair ?
Your comment that "in our org ... they are also hands-on" is the an echo of the Joel Spolsky 'architecture astronaut' type. Someone who spends all day architecting and none of their time coding. But here is the thing, at the end of the day its about how much product the organization ships at the end of the day and whether or not that product had does the job it needs to do right? So there is some nuance there to consider.
I certainly believe that anyone who considers themselves to have software architecture skills, should also be able to do anything a senior engineer can do caveat a deep discipline/subject matter engineer[1]. So interviewing for senior engineers will also get candidates who show architecture tendencies, and you will recognize them because of the questions they ask. If you want to provoke those questions then leave some of your coding or puzzle questions a bit ambiguous to see if they let their ideas play out or if they are sticking with the task.
Here is the thing though, you can have lots of really good programmers and all they need is a bit of structure around where they are going so that together they can make something awesome. That needs an architect who is spending all of their time keeping everyone on the same page, and if changes need to be made, moving to the new page. If your programming team is fairly senior and just "gets it" and will work out amongst themselves which parts need to fit with the other parts, then you just need another senior engineer like that. In my experience though group 1 is more common than group 2.
[1] There are irreplaceable senior engineers who have become so deeply expert in a particular technology you don't want them to do anything but own that code.
Most of th programmers are so young in this part of the world that Anyone having more than 9 years of tech would expect an architect position and there are high chances that is not actively programming in his previous company and thus it becomes nesssary to eveluate coding skills
That might be ok if the role is a pure R&D role, where they're "playing around" with the latest technology and ideas. We have some of those positions at my current employer. But, it's not very good for an architect who will be embedded with a team of developers.
After you've had a conversation about a project in the candidate's track record, ask some pointed questions:
-- what would you lose in performance and functionality if you reworked that project to run on cheaper infrastructure? (By that I mean such things as "standard edition" rather than "enterprise edition" of DBMSs, open source vs. paid licenses, etc.) This is a good way to get a handle on the candidate's economic reasoning ability.
-- what was a significant performance bottleneck in that system? How did you address the problem?
-- with the benefit of hindsight, what parts of that system would you say you overdesigned? underdesigned?
-- what was a security vulnerability in the system? How did you plan for security? If you had to do security remediation, please describe it.
X being a currently widely known service (I'm usually going with Spotify). There is no "right answer" to this question but this shows you how the candidate thinks and how well they can protect their opinion if challenged
It's also good if you show the ability to see trade-offs where they exist. E.g. the trade-offs for a relational database vs. something else might be familiarity / time to implement, vs. ease solving some type of situational performance problem.
You could lead with this question and then jump in with hypothetical roadblocks to see how the candidate reacts.
Ex:
- "A network of microservices _would_ be a clean way to implement this. How would you foresee the operational burden of this impacting a small team?" - "Imagine that you build this and it works well, but the product it powers ends up not resonating with customers. The team now wants to pivot to X. What changes would you make to the system to address this new problem space?"
A good answer would account for the fact that our initial approach to big projects tends to differ quite a bit from what the the end result is. Agility of architectural design is generally underrated in my experience.
1. RFID cards that tie into
2. Electronic wallets
3. The busses are not always internet connected
4. You need to be able to refill your card either online, or via kiosk.
5. You also want to be able to refund card balances.
What system can solve this? What drawbacks would the system have? How could the system be scammed?
Bonus questions are: "6. The LINK rail allows you go scan your ORCA card when you get on and get off the train, so you only pay for the number of stops you're on. How would you include that? 7. ORCA cards also allow for free transfers-- so if you take a second ride within 3 hours, it will be free. How would you include that?"
- SOLID principles
- The Expression Problem
- DRY
- Dependency Injection
- Composition vs Inheritance vs Delegation
- Connascence
- Law of Demeter
- Invariance, Contravariance, and Covariance
If someone is claiming to be an architect and they've never done that, that's a red flag for me. Kinda like asking a building architect to talk about framing. You'd be surprised at how many of those guy have never swung a hammer and they "design" stuff that can't be built. Sort of the same with software, IMHO.
At Google, more senior software engineers will have a "system design" type question as part of their interview. There are sample problems for this all over the internet[0]. For example, you could ask them "Design a URL shortening service like bit.ly.". When you ask open ended questions like these, there are so many parts of the system you can dive into that you can really test the depth and breadth of a candidate's knowledge.
What architectural mistakes have you made in your career - what did you learn from them?
What architectural decisions are you proud of - how relevant are those ideas to other systems?
How much code should an architect write?
Have you been responsible for maintaining a complex system that you didn't design?
Number 2, 3 and 5 is asking about past jobs and may be even illegal depending on the applicant. NDAs are everywhere. Rephrase them to be generic.
Number 5 is worthless yes/no type of question as well.
On the subject of writing code - the question is really trying to find where on a spectrum someone is from:
"Code, I don't write code, I draw UML and throw it to our offshore development partners"
through to
"I always keep all the most complex and demanding parts of the development process to myself as everyone else is an idiot and only I can realize the amazing architectural vision I have <maniacal laugh>"
[And yes, the space probably isn't one dimensional and those probably aren't the endpoints] :-)
I've known extremely successful "architects" (in terms of £££$$$) who fit both of those descriptions...
1) Peer presentation - talk about past project
2) System design - design at the high level
3) Debugging - given a failing test fix the code so the test passes.
4) Coding - Straight forward problem to code and create.
For the peer presentation, we are expecting the candidate to be prepared. They have lots of notice about this portion of the interview. We want them to be articulate and clear about the technical problems, project management problems and unexpected issues.
System design - we come up with a sample issue that is easy to articulate quickly. For example, "we are going to design the first version of Wordpress". We then ask questions and extend the problem during the interview.
Debugging - we give them a random open source project and we break one of the unit tests. We then ask the developer to track down and fix the issue.
Coding - smaller scale than the System design question but focused on determining if the candidate can construct a small program end to end. So this is not a portion-of-a-program, but rather construct a quick command line program to do X.
For junior people we drop the peer presentation and add additional coding questions.
--------
I liked this interview the best of all the places I interviewed. There was no "gotcha" and the problems I was asked were relevant to Evernote's business.
Walking out I didn't feel like I did the best I should have, (I have high standards) but I felt the problems were incredibly fair.
Now that I am at Evernote, hands down where interviews go badly are in the debugging section.
The peer presentation shows people who have not thought about the "why" behind their coding / design decisions - and it separates the juniors from the seniors.
Lastly, we expect and can demand a far higher interview performance because we tell candidates how to prepare. If they don't prepare, it shows soooo painfully. For candidates that do prepare we can have a more interesting and engaging interview that allows us to sell Evernote as a company.
This was for an interview for backend engineering at uber, so by no means did I have any particular knowledge or speciality in browser design. Because I knew the product very well from a user's standpoint, and common knowledge about web standards, it was really fun and exciting to figure the architecture out.
Typically the questions revolve more around: "How would you design twitter/pinterest/facebook newsfeed?"
What was the worst requirement you had to meet?
Where would you store 100Mb user data? 10Gb? 1000Gb?
Preferred deploy to production strategy?
Worst process you followed? ..what did you do to make it better?
Thanks for mentioning that one - it means I get to think of a sensible answer rather than convulsing in laughter for a few moments. It did involve physical buttons for "Burger", "Beer" and "Porn".
- Tell me some pros and cons of a library or framework that you have used recently.
- Here's an hypothetical problem (describe the problem). How would you approach solving it, from the point of view of software architecture? Please think aloud and ask questions, the idea here is getting to know how you think, not necessarily arriving to a perfect solution.
My favored approach for it has been to ask people questions that mirror problems I've been forced to solve in the past at the current venue you're hiring for. I think this approach has a number of advantages:
1. You know how YOU solved the problem, so you've got at least some qualifications to discuss it with authority.
2. You can answer any and all questions the prospective architect wants to ask. These questions and their quantity are often even more telling than any hastily described solution ever could. Do they ask about infrastructure? Do they ask about expertise? Do they mention specific tooling you yourself decided on?
3. This gives them a chance to wow you in a controlled environment. If their idea for a solution is better than yours, you know you have a very high value candidate.
4. This also is a supreme test of technical communication skills. They need to rapidly get YOU to tell them things by asking questions, formulate an idea, and then communicate it to you in a way you can understand.
Of course this assumes the interviewer has done some architecture, but I think that's probably required to do this job right.
Personally, I think all this talk of "war stories" and whatnot is not a very good idea. If you're concerned about someone's personality and grace under pressure, you should treat that separately from their technical skills. That way you can weigh those attributes separately and fairly when it comes time to decide if you will hire the candidate.
Most importantly they should light up when they get the question. Like someone just sat a piece of cake in-front of a fat kid.
Practical overviews of design decisions in familiar open source software. I wish there were books like that!
You have written some code you'd like people to be able to reuse. Should you share it as a library or expose it as a service? Talk me through how you'd make this decision for a few cases and what the pros and cons are for each choice.
My ideal response would be one struct with nothing but a suit enum and int value. A deck is just an array of these, as is a hand. But usually people start with a deck class, then try to reuse that for the hand, discard pile, etc. Things get complicated quickly, just like in real world problems, and a good architect can keep that in check while still providing useful abstractions.
Ask a bit of trivia about the stack you're hiring for. They don't need to know the right answer, but their reasoning about it should corroborate with their experience.
My advice is to see how they operate... Give them a problem, even a vague one, but then solve it together, have them use google to seek out ideas. See how the person approaches it, don't expect a bunch of memorized answers off the top of their head.
The big thing is to see what questions they ask. How will you and the person collaborate in a normal working environment.
The personality and adaptability are far more important than memorizing the exact command line or function call.
When tasked to interview I have found that asking a question which you know the candidate will be unable to answer will show you the most-intriguing parts of their character. Something deceptively simple like a puzzle or logic problem but scales up quite quickly.
This is when you can learn things about them that they would not put to paper and thereby determine if they are for you.
Another question is to pick a project they've worked on, and ask them how would they do it if they were given a clean slate.
'What do you expect to be involved in as a software architect and what can you contribute?'
and take it from there. If you are interviewing for that role then surely what they do in the context of your organisation is key and open questions help you understand whether the person is a fit for that.
See this example: https://www.youtube.com/watch?v=cfyWvJdsDRI
I forget the blog, but it was from an unused script off of silicon valley, interviewing for a new CEO of PP.
Let's face it, we would all love the meaty architecture questions, but most interviewers are still stuck on the CS101 algorithms.
Given such a system, how could you process a document to subvert it?
How would you fix the system?
How could you subvert the fix?
...