NumberSix

Regarding privacy, when human beings meet a new person, they do not need tons of data about that person to understand their spoken language. We learn English or any other language once and then rarely need to adapt to new people, unless they have a strong accent or speak a truly different dialect of the language. This demonstrates that the much touted speech recognition technologies from Google, Amazon, and Apple do not match human level performance. Are they really exceeding the marginal performance of the Hidden Markov Model based Dragon Naturally Speaking which took around two months of training to achieve its maximum accuracy ten years ago? Or are they just running similar models with huge numbers of adjustable parameters tuned to each user on the "cloud?"

If Apple invested in genuinely new, creative AI technologies that matched human level speech recognition and other tasks then they could preserve their purported emphasis on privacy. They would not need to collect huge amounts of personal data on most customers, unlike Google or Amazon.

The article makes the following statement:

Whiteboard interviews have a very high false negative rate (rejecting people who are good), but they also have a low false positive rate (hiring people who are not good).

On what basis is this statement made? No study or source appears to be cited. The specific false negative and false positive rates are not given. Also, what exactly is the definition of "good" used?

Bah Humbug!

Many academic fields, notably including astronomy and physics, have been doing heavy duty software development and data analysis for many decades, long before the current "data science" hype.

Chuck Moore developed FORTH in 1968 (yes 1968) for controlling radio telescopes: https://en.wikipedia.org/wiki/Charles_H._Moore

Astronomers developed a widely used interactive scripting language/tool for astronomy data analysis known as IDL in 1977 long before vanilla Python was created, let alone recent add-ons such as NumPy, SciPy, etc. Python is playing catch up.

https://en.wikipedia.org/wiki/IDL_(programming_language)

The Hubble Space Telescope was launched in the 1980s and has been using Python for over 10 years: http://www.fperez.org/py4science/2009_guido_ucb/perry_STScI4...

http://www.stsci.edu/institute/software_hardware/pyraf/stsci...

Perry Greenfield - The Development and Future of Python at STScI https://youtu.be/R_UcjjUC8bE

Only a few of many examples of software development and data analysis in astronomy specifically.

The market leading data analysis tool SAS derived from academic agricultural research in the 1970s. Ronald Fisher developed many of his still relevant techniques for agricultural projects at the Rothamstead Research (https://en.wikipedia.org/wiki/Rothamsted_Research)

The second biggest data analysis tool SPSS is short for Statistics Package for Social Sciences, also came out of academic research ... in social sciences in the 1970s.

Mathworks MATLAB is derived from numerical and statistical software developed for physics research at the Argonne National Laboratory. https://www.mathworks.com/company/newsletters/articles/the-o...

Mathematica has its roots in Stephen Wolfram's work on SMP as a graduate student in high energy physics (particle physics) at Caltech. http://www.stephenwolfram.com/publications/smp-symbolic-mani...

Deep learning/neural networks -- as well as a number of other "machine learning" methods -- is fitting a mathematical model with a huge number of fitted (tunable) parameters and component functions to data. It has been known for a long time -- arguably back to the discovery of the Taylor series expansion in the early days of calculus -- that any function can be approximated arbitrarily well by the composition of an arbitrarily large composition of other functions.

If the task is interpolation between the data points this can be highly accurate. It the task is extrapolation such as prediction or designing a truly new machine or system in an engineering application, the approximation will often fail. The about one percent error rate in predictions of planetary motions from epicycles is one of the earliest cases of this problem.

The simple example is approximating data with a polynomial with an arbitrary number of terms such as in the Taylor series expansion. With enough terms a polynomial model can always approximate any data set arbitrarily well. Polynomial models can interpolate very well unless the data has some generally unusual behavior -- changing unpredictably at successively finer scales for example. However, finite polynomial models almost always extrapolate grossly incorrectly. As you move away from the data set in the space of independent variables such as X, the largest power N in X^N dominates and the polynomial approximation function blows up to either plus or minus infinity which is rarely physical.

What we think of as "understanding" corresponds conceptually in part to the ability to make accurate predictions. "Understanding" or "explanation" corresponds mathematically not to some arbitrary super-complex function with large numbers of arbitrary parameters but rather to a mathematical object such as as system of differential equations (e.g. Maxwell's Equations for Electromagnetism or the General Theory of Relativity) that express interrelationships among the variables and data points.

iPhones are not cheap. Like almost all Apple products, they are expensive compared to comparable competing products. Several studies have shown that iPhones cost 2-3 times the cost of the components from which they are assembled. Apple claims huge profits and over $200 billion stashed overseas to avoid US taxes.

https://www.macrumors.com/2015/09/30/iphone-6s-plus-componen...

Apple reported an astonishing $2,136,273 in revenues per employee in 2015. http://www.businessinsider.com/revenue-per-employee-at-apple...

Apple contracts a lot of work out to H1-Bs through firms like Wipro. This may well result in larger profits for Apple, but it certainly does not translate to cheaper products for Americans or jobs for Americans.

The article does not discuss the impact of the open office movement. I've worked from home, in actual offices, in cubicles both cramped and spacious, and open offices. It can be quite difficult to focus at home due to various distractions, especially if you do not have a separate room or area for work. It is difficult to separate work and non-work with a home office. There are a variety of communications issues with colleagues when working from home. However, programming requires high levels of concentration and focus; I work on complex algorithms and mathematical software which requires extremely high levels of concentration and focus. Open offices are simply too noisy and full of interruptions. Thus, given a choice between an open office and home, home wins.

But in general I would rather work in a quiet office near my colleagues. There is a clear separation between work and non-work. Communication is faster and easier, less prone to misunderstandings. This however means something like an actual office or a quiet spacious cubicle with high sound absorbent walls.

In his book Deep Work, Cal Newport argues in favor of a "hub and spoke" office layout where knowledge workers can work in quiet in offices on the spokes, but meet and collaborate at hubs, such as common areas with food, coffee machines, printers etc. This seems like a much better way to balance the need for deep concentration on the one hand with "collaboration," annoying MBA buzz-word at present.

In general, the way to objectively determine if software works is to give it to those pesky end users, have them use it, and tell you if it works or not, and how well it works. Lacking end users, have testers and QA folks who play the role of end users evaluate it.

In some areas of software development, such as heavy duty algorithmic/mathematical programs such as encryption, video compression, computer graphics, there are pretty rigorous objective ways to measure whether the program works and how well, without human testers or QA people. Usually best to do both even in these cases, just in case there are some subtle issues that the performance metrics don't capture.

On the other hand, predicting the future is noted for being rather hard. Predicting correctly whether a program will need to be changed in two years, in what way, and whether the program can in fact be changed easily all two years in the future is speculation, a matter of opinion, rarely objective at all.

Conspicuous by its absence is:

Does your software work?

These criteria are all rituals and processes, rather than the end result.

A follow up as a point of reference:

The median home price in Plano, TX is $317,800 according to Zillow (March 12, 2017 https://www.zillow.com/plano-tx/home-values/)

One bedroom apartments in Plano rent for $956 a month on average and two bedroom apartment rents average $1293. (according to RentJungle https://www.rentjungle.com/average-rent-in-plano-rent-trends...)

Plano (https://en.wikipedia.org/wiki/Plano%2C_Texas) is a suburb of Dallas-Ft.Worth where many people who work in the Telecom Corridor https://en.wikipedia.org/wiki/Telecom_Corridor live. It is similar to the Silicon Valley and more generally urban areas in California but with much lower home prices and apartment rental rates.

The high housing prices are not a consistent nationwide pattern.

Urban areas in California, where most of the people live, such as the Silicon Valley have extremely high home ownership and apartment rental costs. The article notes this briefly and somewhat obliquely, but it is worth emphasizing.

The median home price in San Jose, CA is about $850,000 at present (March 12, 2017) according to Zillow.

The average rent for a one bedroom apartment in San Jose is about $2473 per month (https://www.rentjungle.com/average-rent-in-san-jose-rent-tre...)

The US Census lists a median home price for the United States as a whole of $221,800 in 2010 (https://www.census.gov/const/uspriceann.pdf)

Median monthly gross residential rent in the United States was $934 in 2014 according to the Census ACS survey. (http://www.deptofnumbers.com/rent/us/)

Other living expenses are often 10-25 percent higher in the major urban areas in California where most people live than other parts of the United States. (personal experience)

It is housing and rental costs that dominate the disparity with much of the country however.

Someone who is genuinely good at writing code should be able to produce code that has no or very few bugs. Similarly, they should be pretty good at inferring what is wrong in the rare case of bugs by simply running the program. Therefore, they would find writing unit tests and similar rituals unnecessary and time consuming.

They would most probably write specialized tests where appropriate to catch subtle bugs that elude even their superior skills. This would be a matter of personal judgment based on the problem at hand.

One issue that the article overlooks is how many jobs the Microsofts, Googles, Apples, Facebooks, etc. outsource to the H1-B outsourcing companies such as WiPro, Cognizant etc. I have met many consultants in the Silicon Valley who work for Apple through a firm such as WiPro. These contractors salaries presumably would not show up in the salaries for H1-B employees working directly for these large, high-profile companies.

This is the opening paragraph of the BLS press release on employment from Feb. 28, 2017. https://www.bls.gov/news.release/srgune.nr0.htm

For release 10:00 a.m. (EST) Tuesday, February 28, 2017 USDL-17-0286

Technical information: (202) 691-6392 * lausinfo@bls.gov * www.bls.gov/lau Media contact: (202) 691-5902 * PressOffice@bls.gov

                REGIONAL AND STATE UNEMPLOYMENT -- 2016 ANNUAL AVERAGES

Annual average unemployment rates decreased in 38 states and the District of Columbia, increased in 9 states, and were unchanged in 3 states in 2016, the U.S. Bureau of Labor Statistics reported today. Employment-population ratios increased in 36 states and the District, decreased in 12 states, and were unchanged in 2 states. The U.S. jobless rate declined by 0.4 percentage point from the prior year to 4.9 percent, and the national employment-population ratio rose by 0.4 point to 59.7 percent.

Note that BLS uses the U-3 "unemployment rate" of 4.9 percent without qualifications or explanation. Yes, if you read through the fine print in BLS documentation (not the press release), you can determine that U-3 omits allegedly "discouraged workers" as in fact does the broader U-6 measure as well.

The mass media is following the government/BLS lead in promoting the U-3 measure as the "jobless rate." Again, there is no qualification in the opening paragraph, no mention of the broader U-6 measure or the general issue of purportedly discouraged workers.

The uncritical, almost universal repetition of the U-3 number raises the question of the independence of the major mass media, the so-called Mainstream Media or MSM, and the government, and vice versa.

I use "massages" to cover two factors. First, the Bureau of Labor Statistics has redefined the way it measures unemployment and employment several times over the decades. In addition, there are a range of seasonal adjustments and other possible variations in the surveys and number crunching that are more difficult to understand and less well documented.

See for example this discussion of discrepancies between the BLS unemployment rate and the Gallup Poll unemployment rate (since 2010):

http://unemploymentdata.com/unemployment/is-the-government-f...

It is difficult to know if in practice the household surveys are done the same way each time and also what seasonal and "one-time" adjustments may be occurring either during the first measurement or subsequent corrections.

As the discrepancies between the BLS and Gallup numbers indicate, there is evidence of shorter term massaging not captured in the major redefinitions, such as excluding "discouraged workers" who want a job but supposedly have not looked for one recently, in the 1994 revisions to the unemployment measures.

The explanation is that the government massages unemployment figures to make the economy look better than it is. The government has been doing this for decades, under Republicans and Democrats, liberals and conservatives. Unemployment is one of the most widely watched and therefor political numbers; anyone who is in power wants it to look as good as possible.

One of the tactics to make the official unemployment rates look better is to classify some unemployed workers as "not looking for work." Voila! They don't show up in the official estimates and it is their fault!

Most economists are funded by the National Science Foundation (NSF) and other powerful institutions that have little interest in questioning the official numbers. Hence lengthy academic discussions to try to "explain" the seemingly odd behavior.

There is a question regarding the definition of "data science jobs." The author explains his methodology in a lengthy report http://r4stats.com/articles/how-to-search-for-data-science-j...

The issue is what is "data science" really? In what respect is it different from traditional statistics and data analysis and not just a new buzzphrase?

Probably many jobs using SAS could be considered "data science" but don't use the specific buzz words and phrases that the author specifies in his methodology to identify "data science" jobs. Thus, the headline that "R Passes SAS" could be inaccurate, except in the sense that R is more popular among statistics and data analysis jobs that use "data science" buzzwords and phrases.

How long have the onsite interviews been? An in-person interview with at least three people and at least three hours total duration is what I call "a serious interview." At least for software engineers, there seems to be a roughly 1 in 2 to 1 in 3 chance of a job offer if you get a "serious interview." In general, shorter in-person interviews -- say one person for 1/2 to 1 hour are not very serious and can have very low success rate, usually at best they lead to a further "serious interview."

If all seven onsite interviews have been "serious interviews" then zero job offers is a bit low, but still could be just bad luck: 1 chance in 2 to the 7 (128).

That said, mid thirties -- thirty-five to be exact -- is the age where employees in the high technology industry start to report increased difficulties finding jobs. Look around -- not a lot of people who look over thirty-five.

Very few companies seem to be looking for 10+ years of experience. Most job postings on LinkedIn and other job boards will ask for 3-5 or 3-7 years of experience. Some for ten years, but not that many. You state you have twelve (12) years of experience.

It is difficult to know the reasons for this pattern in high tech. Probably all of these contribute:

o conscious and sub-conscious age discrimination

o companies are looking for young people who can work long hours, unpaid overtime. Overall stamina does decline with age. It is also harder to get more experienced people to work unpaid overtime as they recognize what is happening -- aren't fooled by unrealistic schedules, manufactured crunches over trade show appearances, and other deceptions and demand to be paid.

o Companies and managers feel threatened by employees who know a lot, preferring young people with little or limited experience.

US born "native" white men with perfect English report same difficulties frequently. It may have little to do with being Indian or having an accent, although it is surely good advice to work on accent and mastery of American English which differs in a number of ways from Indian English. Indian English is derived from British English in the 1940's and 1950's that was taught in missionary schools in India. There are a lot of web sites and blog posts now on the differences between the two dialects.

Also, if you are on a visa like the H1-B, that can be an issue as well. Probably easier to switch jobs with US permanent residency (green card) or US citizenship.

If you are unhappy with your current job and have trouble leaving it, non-work social activities, physical exercise, and sometimes meditation can be helpful. Unfortunately, under stress it is very easy to cut back on social activities and exercise, but actually you should do the opposite -- increase these -- under sustained stress.

Perhaps talk to some older product managers who are not trying to hire you, colleagues rather than employers, about their experiences and what might be an issue in the job hunt.

Good luck!

A few points regarding CS 101/201 data structures and algorithm coding interviews.

The coding interview is a new phenomenon. In the 1990’s and early 00’s, these type of interviews were rare. Typical interviews were more well rounded with questions about interests (what do you want to do), past experience (what have you done), personality and culture (how do you like to work), as well as some technical and coding questions, often relevant to the actual job to be performed. Also some brain teasers like “why are manhole covers round” which Google has since condemned.

As someone who works largely on complex algorithms and mathematically oriented software such as video compression, speech recognition, and gesture recognition, I can say from experience that these tests frequently have nothing to do with what algorithm developers actually do. These are solved problems in CS such as how to sort data, how to implement a hash table, etc. In practice, developers rarely reinvent the wheel but rather use existing implementations in libraries or source code. The coding interview questions tend therefore to discriminate against older, more experienced developers who are many years past CS 101 or never took these classes.

About half of practicing developers are “self-taught” which typically means they come from some other STEM (Science, Technology, Engineering, or Mathematics) field where they learned to develop software by doing, not by taking CS classes. The tests tend to discriminate against them as well.

As for women and underrepresented minorities, which typically seems to mean people with visible African ancestry and Hispanics with visible American Indian ancestry, this is difficult for me to judge. But in general the CS101/201 data structures and algorithm coding interviews are not a test of the skills used in most (not all) professional software development.

What we can say is that these tests are also not blind tests of technical skill. They are analogous to the bad old days when orchestras auditioned prospective musicians by viewing the musician playing and few women ever made it through. The introduction of blind auditions where the candidates performed behind a sheet and with other measures to hide their gender, race, and other presumably unimportant factors — only the actual music was evaluated — seems to have resulted in an increase in women passing the auditions.

The United States has a total resident population (as of January 23, 2017) of 324,420,000 people (about 324 million). Source: https://en.wikipedia.org/wiki/Demographics_of_the_United_Sta...

Thus, $4.1 trillion in consumer debt works out to $12,638 per person.

The Gross Domestic Product (GDP) in the United States was worth 18036.65 billion (about $18 trillion) US dollars in 2015. That is about $55,424 per person.

Note that the "consumer debt" is divided into several categories.

Student loans represent a long term investment and may be reasonably paid off over decades.

Auto loans are typically paid off in 3-5 years.

Ideally, credit card debt should be paid off immediately. It is difficult to evaluate credit card debt because credit cards have increasingly become the substitute for cash in the United States. How much of the debt is extremely short term and essentially represents what used to be cash transactions?

What is "other" consumer debt -- payday loans? IOU's?

The point is that $4.1 trillion in consumer debt in a nation with over 300 million people and a GDP over $18 trillion per year is neither unreasonable nor a crisis, particularly when that debt includes longer term items such as auto loans and student loans.

A number of points in the post/article are questionable.

First, it assumes the developers had substantial control over the schedule for the project ("Giving excessive importance to estimates"). Certainly in my experience this is unusual. More frequently, the schedule is dictated by management, frequently by sales/marketing executives in commercial software development. It is very difficult to push back and a good way to lose your job.

Sales: We have closed this great deal with BigCorp. Can you do X (complicated, challenging software project) by the end of the quarter?

Developers: Err, um, X sounds like a project that will take six months.

Sales: We really need to make our quarterly numbers. Our CEO Bob used to be a developer and he says any competent programmer can do it and we only hire the best. Competent doesn’t cut it here! You are a rockstar ninja, aren’t you? Can you prove you can’t do it by the end of the quarter?

Developers: Well, no. The schedules are driven by some unexpected problem or problems that usually happen. But, well, if nothing unexpected happens, we can do it by the end of the quarter.

Sales: Great! Bob is expecting results by the end of the quarter.

So much for the beautiful, elegant software design methodologies taught in college and university CS programs and peddled by high priced consultants.

Second (“Giving no importance to project knowledge”), high technology employers seem to have extremely high turnover rates of software developers and other employers. Payscale produced a study claiming that the average employee tenure and Amazon and Google is only one year. Many companies seem to target employees with more than seven years of paid work experience — Logan’s Run style — for layoffs and “constructive discharge,” (https://en.wikipedia.org/wiki/Constructive_dismissal) where employees are made uncomfortable and quit “voluntarily.” Undoubtedly, this is costly as the author implies, but it seems to be common practice.

Yes, metrics like “issues closed,” “commits per day,” or “lines of code” don’t work very well. Once employees realize they are being tracked and evaluated on some metric, they have a strong motivation to figure out how to manipulate the metric. Even if the employees don’t try to manipulate the metrics, the metrics all have serious weaknesses and map imperfectly to value added (biz speak).

Third, are code reviews and unit testing proven processes especially for normal non-Microsoft companies? In the early days of Test Driven Development (TDD), Kent Beck and his colleagues made numerous claims about the success of Test Driven Development in the Chrysler Comprehensive Compensation System (C3) payroll project, an attempt to create a unified company wide payroll system for Chrysler. This project in fact had a range of problems and was eventually cancelled by Chrysler in 2000, without replacing the Chrysler payroll systems successfully.

As the problems with C3 have become well documented and well known, TDD enthusiasts have shifted to citing studies at Microsoft and some other gigantic companies that claim practices like TDD and code reviews work well. Are these really true or do these case studies have hidden issues as C3 did?

Further, Microsoft, Google, and other companies that have played a big role in promoting these practices are very unusual companies, phenomenally successful super-unicorns with sales in the range of 40-100 billion (with a B) dollars with near monopoly positions and anomalously high revenues and frequently profits per employee. Microsoft claims to have revenues of $732,224 per employee. Google claims an astonishing $1,154,896 per employee. (http://www.businessinsider.com/top-tech-companies-revenue-pe...) This compares to $100-200,000 per employee for most successful companies.

Fergus Henderson at Google recently published an article on Google’s software engineering practices (https://arxiv.org/abs/1702.01715) with the following statements:

2.11. Frequent rewrites

Most software at Google gets rewritten every few years.

This may seem incredibly costly. Indeed, it does consume a large fraction of Google’s resources.

Note: “incredibly costly”

Companies like Microsoft and Google have enormous resources including monopoly power and can follow practices that are extremely costly and inefficient, which may work for them. Even if these practices are quite harmful, they have the resources to succeed nonetheless — at least for the immediate future, the next five years.

From a business point of view, it may even be in the interests of Microsoft, Google, and other giant near monopolies to promote software development practices that smaller competitors and potential competitors simply can’t afford and that will bankrupt them if adopted.

Both code reviews and unit tests are clearly time consuming up front. Code reviews using tools like Google’s Gerrit or Phabricator (a spin-off from Facebook, another super-unicorn) are committee meetings on every line of code.

Regarding:

Imagine my dismay when I had to collaborate with a colleague on that legacy project and his screen displayed Notepad in its full glory. Using “search” to find methods might have been rad back in the nineties, but these days, refraining from using tools such as modern IDEs, version control and code inspection will set you back tremendously. They are now absolutely required for projects of any size.

Using “search” to find methods was not rad back in the 1990’s. IDE’s and code browsers specifically have been in widespread use since the 1980’s. Turbo Pascal (https://en.wikipedia.org/wiki/Turbo_Pascal) was introduced in 1983 and featured a fully functional IDE, soon to be followed by IDE’s in many other products. Version control dates back at least to SCCS (https://en.wikipedia.org/wiki/Source_Code_Control_System) which was released in 1972. RCS was released in 1981 and version control was common in the 1980s and since.

Code reviews have been around for a long time. However, in the 1990’s and earlier they were restricted to relatively special projects such as the Space Shuttle avionics where very high levels or safety and reliability, far beyond most commercial software, were required. This speaks to the “incredibly costly” quote about Google above.

Without more context, it is difficult to evaluate the use of Notepad. Simple code/text editors like Notepad and vim (formerly vi ) are very fast to start up and can be a better option for some quick projects than starting an IDE.

Some IDE’s are particularly hard to use. Early versions of Apple’s Xcode circa 2010 were particularly difficult to use in practice; it has improved somewhat in the current releases.

People vary significantly. Some developers seem to find stripped down tools like vim or Notepad or Notepad++ (on Windows) a better option than complicated IDE’s. I am more of an emacs or IDE person.

The fact that someone else works differently than you do does not mean they are worse (or better) than you. The fact that something works well for someone else also does not mean it will work well for you — or vice versa.

There are sound reasons for duplicating code, cutting and pasting, rather than creating a function or object called in several location in the code. If the developer anticipates that the code may subsequently diverge, then duplication is often best.

Like grand master chess players, highly experienced developers, especially under tight time constraints (like a chess tournament), code by intuition, not by laboriously reasoning out every step. If it feels like the code is likely to diverge in the future, duplicate. If it does not diverge, no problem, it can be merged back later if needed.

In the bad old days of structured design (1980’s) and object-oriented design (OOD — 1990s), software development projects suffered from Big Design Up Front (BDUF), grandiose attempts to design a perfect software system before writing a line of code. This often resulted in massive cost and schedule overruns and total failures. It often proves better to just throw (“hack”) something together quickly — a prototype, proof of concept, Minimum Viable Product (MVP). Just “get something working.”

Inevitably these prototypes and early stage software projects are going to compare poorly to some theoretical perfectly designed system with 20-20 hindsight. That is what seduced people into BDUF twenty, thirty years ago.

Modern Agile software development methodologies are foolishly trying to have it both ways, have an initial quick iteration BUT that first iteration should be perfectly designed up front — beautiful, elegant, with hundreds of tests, endless committee meetings on coding style and design (code reviews), all sorts of supposed best practices, no code duplication, etc. This is a seductive fantasy doomed to fail in most cases.

Google is highly non-representative of businesses and technology businesses in particular. It has a near monopoly on the search business and has enormous amounts of money from a single source -- advertising.

Google Fiscal Year 2015

Revenues: $74.54 Billion (source: Google) Advertising Revenue: $67.39 Billion (source: https://www.statista.com/statistics/266249/advertising-reven...) Profits: $23.4 Billion

Market Capitalization: $570 Billion

Google Not a Startup/Has Not Been a Startup for Many Years

Founded: September 4, 1998 (19 years ago) IPO: August 19, 2004 (13 years ago)

Number of Employees (2015): 57,100 Revenues per Employee: $1.3 Million Profits per Employee: $409,000

The issue is that Google has so much money and is so successful that it can do all sorts of things that are extremely inefficient, even very harmful and still do fine, unlike smaller companies or startups in particular.

For example the arxiv article states:

2.11. Frequent rewrites

Most software at Google gets rewritten every few years.

This may seem incredibly costly. Indeed, it does consume a large fraction of Google’s resources.

Google has the money to do this. As the article argues, it may work for Google. On the other hand, Google has so much money and such a dominant market position, it probably can keep succeeding even if continual code rewriting is actively harmful to Google.

In orthodox software engineering theory, competent software engineers, let alone the best of the best that Google claims to hire, should write modular highly reusable code that does not need to be rewritten.

Many rewrites and "refactorings" are justified by claiming the "legacy" code is "bad code" that is not reusable and must be written by "real software engineers" to be reusable/maintainable/scalable etc. One rewrite ought then to be enough.

Even highly successful businesses, for example $1 Billion dollars in revenues with $50 million in profits and 5000 employees (revenues per employee of $200,000), have nowhere near these vast resources --- either in total dollars or per employee or product unit shipped. Blindly copying highly expensive software development processes from Google or other super-unicorn companies like Apple or Facebook is likely a prescription for failure.

The finger pointing at the graduate student Heather Ames reminds me of the concept of gas lighting, derived from the 1944 movie Gaslight:

gas·light

ˈɡaslīt/

verb

gerund or present participle: gaslighting manipulate (someone) by psychological means into questioning their own sanity.

https://en.wikipedia.org/wiki/Gaslighting

This is a fairly common bullying tactic where the bully both bullies, usually secretively, and tries to make the victim look like a nut case.

The story, with others pointing the finger at Heather Ames, raises the question whether more people were involved than the single researcher who was caught on video. While self-sabotage was certainly a possibility, sabotage by a colleague or co-worker was clearly a substantial possibility and reportedly proved to be the case.

"Legacy" is a very loaded term -- generally used in a derogatory way.

There is an old engineering truism: "if it ain't broke, don't fix it"

When Lindbergh flew across the Atlantic in the Spirit of St. Louis, the Wright Aviation Company which made the engine for his plane, wanted to do its best to make sure he made it. They carefully built and checked the engine and hand delivered it with a note: "don't monkey with it" (or words to that effect). Lindbergh was a careful guy and he didn't.

Businesses and developers need to distinguish between warranted or needed changes to working systems (software or otherwise) and ego trips.

Real systems are built in limited time, with limited resources, even if they have a multi-billion dollar budget. Invariably there are things someone can make an issue about even in a system with a perfect performance record. Why did you hard code 3.14159267 instead of having a general variable for PI initialized to 3.14159267, and by the way why do you have only 8 decimal places -- wouldn't 16 be better? It is an ego trip and it is the ancient territorial imperative, marking your territory by deprecating the previous person or team -- or a colleague.

Joel Spolsky wrote a famous and widely ignored blog post on this sort of thing:

https://www.joelonsoftware.com/2000/04/06/things-you-should-...

Take a deep breath and ask yourself: is this something we really need to do or am I (are we) showing off/putting someone down/behaving like a spoiled adolescent?

Programming has not changed much since the 1960's and 1970's when Unix and C were developed. Modern computer programming languages simply mix and match different features from the "Cambrian explosion" of experimental and production programming languages in the 1960's. Hence it is not surprising that books on general programming concepts and principles are still relevant years later.

C is still in widespread use in operating system internals, embedded software development (think wearables and Internet of Things), complex algorithms like video compression, and miscellaneous other fields.

Version control dates back at least to SCCS in the 1970's and has been in widespread use at least since RCS in the 1980's. I mention this because I occasionally encounter people who seem to think version control either was invented with Git (not true) or rarely used before Git (not true).

Testing software is not new. Testing fanaticism is new.

Things that have improved in programming are:

Optimizing compilers have improved and all but eliminated low level loop optimizations by hand and similar "micro" optimizations by hand. This has also largely eliminated assembly programming.

Computers are much faster than in the 1960's, not much faster than the late 1990's, which means many things can be done with scripting languages that were once too slow. Compiled languages like C and C++ still dominate at the very high end of performance.

Scripting languages play fast and loose with variable types which makes it easier to write code quickly, but makes it harder to achieve the speed of a compiled executable from a relatively strongly typed language (compared to a scripting language -- how strongly typed is C really with its casts, pointers, and other unsafe but useful features?).

Compiler error and warning messages are much clearer than years ago.

There have been some incremental improvements in programming languages like using the plus sign (+) to concatenate strings that have made many new languages a bit easier to read and write. Compare strcat(blatz, " and now") to blatz + " and now"

None of these, or other improvements in the last fifty years, are revolutionary advances that would obsolete an older book on general programming principles.

Computer programming remains labor intensive and often highly compensated because many of the problems and issues remain largely unchanged since the early days (1960's/1970's).

USC Dornsife picked the winner. Other polls did not.

Trump got 46.1 percent of the popular vote:

http://www.cnn.com/2016/12/21/politics/donald-trump-hillary-...

That is almost the same as the 46.8 percent of the popular vote for Trump that USC Dornsife predicted. Pretty close to spot on.

The other polls, including 538, underestimated the size of Trump's support.

The USC Dornsife/LA Times Daybreak Election Poll

http://www.latimes.com/politics/la-na-pol-usc-latimes-poll-2...

did a significantly better job than Nate Silver and 538 in estimating the level of Donald Trump support.

I worked at Apple from 2014 to 2016 and the iPhone, iPad, etc. teams I was working with were in offices, one or two to an office, up until a big move to an open office in early 2016. They appeared to have been in offices for years.

Apple is sufficiently secretive that indeed some other teams might have been in open offices. Not the ones that I saw or interacted with. I was working on a key part of the iPhone and iPad technology.

Apple has only very recently begun to adopt open offices -- meaning in 2016. At the time the iPhone was created and soared to market dominance Apple workers were in actual offices with locks due in part to Apple's obsession with secrecy ( 1 or 2 to an office ).

At least so far as Apple has adopted the open office craze, innovation and sales have dropped with Tim Cook taking a recent pay cut for poor performance.

Yes, Apple is moving to open offices in the new Cupertino headquarters, the "spaceship," but that is not how they achieved their big success with the iPhone.

The phrase was misleading twenty-five years ago when applied to private sector positions, most of which were software development back then as well. There were few private sector positions analogous to a "tenure track" position even then, possibly a few at corporate research laboratories like HP Labs and Xerox PARC.

In any case, with the closure or heavy cutbacks at corporate research labs, AIP should update its language to reflect current realities.

The phrase "potentially permanent position" is misleading. Is it intentional? I don't know.

The primary audience for a report on outcomes for Ph.D.'s in Physics one year after graduation, getting their Ph.D. degree, is students (and parents of students) evaluating whether to pursue a Ph.D.. Obviously, they should investigate longer term outcomes, but young people often don't, focusing the next step in their life/career.

Potentially permanent position sounds like a general version of "tenure track research job" as another commenter correctly noted below.

Students with little or no work experience often do not have an accurate impression of salaries, working conditions, and other aspects of the work world (or academia), specifically in the private sector computer industry where most Ph.D's in Physics currently end up.

News coverage of computer companies like Google emphasizes many far out research-like projects such as the Google self-driving car, the AlphaGo deep learning project, and so forth. These sounds like academic research, so why wouldn't these companies have comparable positions to tenure track research jobs? Indeed, in very rare cases, they may have such positions.

However, the vast majority of industry jobs are at will full time jobs without a specified end date. Especially in the computer industry, they are quite insecure and often short-lived, very different from what potentially permanent position implies. They are not analogous to tenure track jobs.