596 karma · joined November 29, 2013
The computer industry is notoriously unstable with jobs often quite short term in practice, with layoffs common.
Just to be clear, the referenced AIP report starts with the phrase:
Positions accepted by PhD degree recipients following receipt of their degrees fall into three categories: postdoctoral fellowships, potentially permanent positions and other temporary positions.
The figure on page two of the AIP report specifically lists a "private sector" block under "potentially permanent".
Tabel 1 on page 3 lists 70 percent of potentially permanent positions as "private sector." Again this is very misleading as private sector positions are potentially permanent only is the sense that there is a small chance that you might continue at the same employer until retirement; there is nothing like a guarantee.
A high profile example is Emanuel Derman, author of My Life as a Quant (2004) and later books, who worked at Bell Labs from 1980 to 1985 before moving on to Wall Street. He mentions quite a number of other physicists at Bell Labs at the same time.
Most physicists end up in some sort of software development. The high profile "quant" jobs are actually rather rare and hard to get. The Wall Street firms are typically going after very strong physicists, especially theoretical physicists like Derman.
Nathan Myhrvold of Microsoft and Intellectual Ventures fame (or infamy) has a Ph.D. in theoretical physics from Princeton. Did not go to Wall Street. :-)
The Large Hadron Collider (LHC) produced a huge surplus of experimental particle physics (high energy physics) Ph.D.'s with no jobs in the field. Experimental particle physics involves large amounts of software development for data acquisition, instrument monitoring and control, and data analysis, mostly in C and C++, although there is still some "legacy" FORTRAN software. The heyday of FORTRAN in physics was a long time ago.
Although there have been attempts to use neural networks and other machine learning methods in particle physics, the workhorse of data analysis in the field is Ronald Fisher's maximum likelihood estimation and classification -- primarily estimation of parameters such as the mass and width of the Higgs Boson. The discovery of the Higgs was a maximum likelihood analysis.
Although it is undoubtedly possible to map maximum likelihood onto neural networks, in practice they are different. Neural networks are an attempt to simulate the low level structure of the neurons in the brain and solve problems by brute force fitting of data to models with huge numbers of adjustable parameters. In contrast, maximum likelihood involves attempts to understand the phenomenon under study and model it as a small number of functions corresponding to higher level concepts such as the Higgs Boson. A neural net could exactly fit the Higgs Boson peak yet never produce or confirm a physical model of what causes the peak.
http://science.energy.gov/~/media/hep/hepap/pdf/201403/Rolli...
The DOE HEP Theory budget has supposedly been dropping since 2014 -- actually 2012:
http://www.math.columbia.edu/~woit/wordpress/?p=8998
I suspect the amount of time spent on writing grants and informal fund raising is related to the level of competition, how many theorists are pursuing how little money rather than the absolute amount of money available.
The National Institutes of Health (NIH) has many billions, but the probability of getting a grant is very low due to the huge overproduction of biology Ph.D.'s consequently researchers and would be researchers spend enormous amounts of time writing grant proposals, most of which are turned down.
https://nexus.od.nih.gov/all/2015/06/29/what-are-the-chances...
The success/award rates in the above article (about 18 percent) actually seem high. I have seen numbers as low as 9 percent in some articles.
Predictive power has not always trumped understanding as a goal. The original Copernican theory did not work as well as the Ptolemaic model used by astronomers. The Ptolemaic model as it had evolved in the 16th century was extremely complex with many fit parameters and predicted planetary motions relatively well.
All of the models shared the incorrect assumption of uniform circular motion and epicycles, both the Copernican, Tyco Brahe's hybrid system, and the traditional Ptolemaic model which by virtue of the thousands of years of effort invested in it -- it arguably dates back to ancient Sumeria -- actually predicted the motions of Mars and other planets best.
It was only the addition of elliptical orbits with non-uniform motion by Kepler that resulted in the new model surpassing the Ptolemaic model.
In the case of planetary motions, understanding ultimately proved critical. We can predict planetary motions to about 1 in 100,000 accuracy versus about 1 in 100 with the Ptolemaic model of the 16th century.
Modern physics has very few long term secure jobs to think about actual physics. Tenured faculty spend most of their time writing grant applications and fund raising -- not the same thing.
In particular the "losers" at the 1927 Solvay Congress -- Schrodinger, Einstein, and de Broglie -- seem to be the antithesis of "shut up and calculate" and all contributed very critical elements of quantum theory.
Schrodinger found the Schrodinger equation, the differential equation that describes so much correctly, through his approach. This is significant because the operator methodology of Heisenberg and Gottingen either did not work, despite the claims it is equivalent to the Schrodinger equation, or was essentially impossible to use to make predictive calculations. Schrodinger's publication of the Schrodinger equation provided an easy way to calculate results in QM that proved correct using well-know differential equations/calculus methods.
For whatever reason, perhaps to keep their research program afloat, the "winners" at the Solvay Congress jettisoned the conceptual underpinnings of Schrodinger's approach and married it to the operator formalism leading to quantum field theory with its intractable infinities and other problems that remain to this day.
Quantum mechanics is probably "incomplete" as Einstein argued. Hence attempts to unify general relativity and the current quantum theory are likely to fail, as they appear to, since a revised quantum theory is needed.
If the data -- angular velocity distributions of start etc. -- used to support "dark matter," "dark energy" and other patches to the prevailing theory of the Big Bang and cosmology is in fact evidence that Newtonian gravity does not apply at galactic scales and above, then general relativity is not correct at galactic scales and above. Again this would make unifying the established quantum theory and the established general relativity theory incapable of matching observed reality.
The ubiquitous lack of secure longer term jobs like Einstein's civil service job at the patent office -- he was not a post-doc -- make deeper conceptual analysis of the outstanding problems in physics today difficult, probably impossible.
The primary function of a version control system is to recover earlier versions of the software in rare error situations where it is hard to figure out what has gone wrong. Backup to something that worked and identify the change that caused the problem.
For most software developers, a version control system should appear very simple so they can focus on solving the end user or customer or their own problem.
check-in <file | folder> o if the file is new for the project, the VCS should ask do you want to add this to the project? o check-in <folder> checks everything in the folder (directory) all at once. If the folder is new: do you want to add this folder to the project?
checkout <file | folder | project> [ <version> ] o checkout <file> checks out the most recent version of the file <file>. o checkout <folder> checks out the most recent versions of the files in <folder> and its sub-folders o checkout <project> checks out the most recent version of the entire project. The project can just be the specification of the top level folder! o <version> is an easy, human readable/understandable specification of the earlier version, a file sequence number such as 1.43, a folder or module sequence number, or an overall project sequence number (all generated automatically by the VCS -- at least by default).
All sorts of pain and suffering is avoided by dividing the project into folders for different components and contributors and/or teams. These check-in/check-out their folder independently.
The VCS should be very simple and take care of everything, such as maintaining a sequential overall project number as well as file sequence numbers, automatically behind the scenes.For a single developer or a team in the same office or building or for that matter office park with a fast network, that is all you need. For remote collaboration -- such as China and USA -- then you might need one or two command to push/pull the code to remote repositories; this too should be simple!
KISS (Keep It Simple Stupid)
I have seen remote work go very smoothly and go very badly. It depends a lot on how well specified the remote work is.
The "agile" and "issue tracker" methods currently popular do not teach good organizational and design skills needed to partition projects well for remote work. We've gone from one extreme -- rigid up front requirements and designs, the much maligned "waterfall" -- to the other extreme of disorganized, short-term chaos where the big picture is lost. Micromanagement without the big picture planning that is the proper role of project managers.
The OP is arguing for remote work versus cramped, noisy, interruption filled open offices modeled on garment industry sweat shops. Somehow most companies are doing this as if they got together in a back room somewhere and formed a cartel to impose garment industry working conditions in a bizarre attempt to magically lower costs by pretending engineers are seamstresses from poverty stricken nations making sub-minimum wages doing piece-work.
Are quiet offices either with high-walled quiet cubicles or offices with doors where engineers can concentrate and focus on difficult work but still walk over to someone and quietly discuss a pressing issue better than remote work? Probably so, but these sensible working environments seemed to have been banned by some mysterious executive decree.
o clear written requirements expressing the business and high level technical needs.
o well defined acceptance test procedures to ensure that the delivered software (or hardware or whatever) meets those requirements
o thoughtful partitioning of the work between team members to minimize the needed communication and enable the team members to focus and work on their part without having a committee meeting every few minutes on every minor issue.
None of these are new practices. They, or something similar, is needed to work effectively remotely or for that matter in quiet offices with closed doors.
Most engineers should be performing two operations/two command s the vast majority of the time:
check-in <latest change> check-out <latest version> or <last good version>
If a file is new, check-in should ask you if you want to add it to the project.
That should be it.
The proliferation of tutorials, cheat sheets, books, front end user interfaces like SourceTree for Git is a symptom of a problem.
KISS (Keep It Simple Stupid)
cites Kent Beck's books Extreme Programming Explained (1999) and Test Driven Development: By Example (2002)
Kent Beck is Mr. TDD It is reasonable to question his claims about C3 as many have done.
The IBM article Integrating Software Assurance into the Software Development Life Cycle (SDLC) https://www.researchgate.net/publication/255965523_Integrati...
has the following section:
PROCESS TO SECURE CODE
In the event of a vulnerability finding, the software code may require redesign and implementation. This iterative cycle is costly in time and resources. To truly understand security threats to a system, security must be addressed beginning with the initiation phase of the development process. For an organization this means they must allow the IA controls and requirements to drive design and influence the software requirements. Therefore, any identified security threats found during the requirements and analysis phase will drive design requirements and implementation. Security defects discovered can then be addressed at a component level before implementation. The cost of discovery and mitigation can be absorbed within the review, analysis and quality check performed during the design, and implementation of our SDLC. The resultant product is one with security built in rather than security retrofitted. A study was performed by the IBM System Science Institute in order determine the relative cost in order to fix defects within the SDLC. Figure 2 displays their findings.
followed by a figure 3 (not 2) which is a simple graphic.
There is no identifiable reference to the underlying data which is presumably some sort of study for the US Department of Defense for security software issues presumably for military related software projects. This does not appear to be a general study, nor is it at all clear what was studied or how.
I am not questioning credentials but extrapolation from in Kent Beck's case a payroll system to general software development or in the IBM case from some sort of DoD software security projects to general software development.
DoD and aerospace projects often have unusually high costs for bugs in the production systems. A bug in a flight avionics system can cause a literal crash with loss of millions or even billions of dollars (Space Shuttle for example) and lives.
Of course, this is a fictional example making a lot of assumptions. Your mileage will vary based on lots of factors, including your team’s experience with TDD, the presence or absence of other quality measures (such as code review), etc…
Here are the assumptions made:
Production bugs without TDD: 100
Production bugs with TDD: 40 (60% fewer, middle ground between 40% — 80%)
Average time to fix bug at implementation (TDD) phase: 1 hour (this number is only used to derive the production bug fix cost)
Average time to fix a production bug: ~15 hours
The author does try to justify this by saying:The relative results of this fictional example ring true to me based on my experience with real production projects, both with and without TDD.
BUT...
Where is the specific data and how representative is the author's experience of all or most software development?
Software development is enormously varied. Kent Beck initially hawked TDD based on a project to rewrite Chrysler's payroll system that was cancelled.
https://en.wikipedia.org/wiki/Chrysler_Comprehensive_Compens...
Also see:
https://stackoverflow.com/questions/10149798/how-tdd-is-rela...
The Chrysler C3 project is not the most convincing argument for TDD or other of Beck's ideas. But even assuming the C3 project was a roaring success -- a highly debatable proposition, one should ask how applicable the results are to many other quite different types of software projects.
Similarly, what is the author of this article's actual experience and how applicable is it to other types of software projects?
I don't question that automated tests can be helpful and indeed critical in some projects but I do question rigid orthodoxies like TDD or requiring unit tests for every function or class after implementation (many people promote unit tests everywhere while claiming they are not doing TDD).
Tests take additional time to design and implement and thus cost more. They also have maintenance costs. Architectural and other changes in the software as it is developed can and do result in revising or rewriting the tests as well.
Many successful software projects were designed and implemented in the 1970's, 1980's, 1990's, and even today without TDD and without unit tests -- or code reviews although I am focusing here on TDD and unit tests.
https://blog.gitprime.com/impact-a-better-way-to-measure-cod...
What I found is:
Impact takes the following into account: The amount of code in the change
What percentage of the work is edits to old code
The surface area of the change (think ‘number of edit locations’)
The number of files affected
The severity of changes when old code is modified
How this change compares to others from the project history
No exact or detailed formula for "impact" is given. "takes the following into account" is extremely vague as it could indicate any relationship. Is impact higher with more files changed in the commit? Why would it be better if more files are changed? Is it lower? Why would it be better if fewer files are changed? Is it some totally unobvious non-linear function such as a trigonometric sine?Based on the vague description above, nothing in "impact" is directly related to the actual end user/paying customer experience or a reasonable proxy such as systematic end user testing by a QA team.
This lack of a direct relationship to the desired end result is the same problem that lines of code (loc or LoC) and many other metrics of software engineer output have.
The "impact" metric, whatever it precisely is, looks suspiciously like it would naturally be positively correlated with a large number of commits/high frequency of commits.
Also the plot is labeled with "volume" on the horizontal axis and not the mysterious "impact" metric. The text implies this horizontal axis is the "impact" metric. Why is the horizontal axis not labeled impact?
Even more peculiar, "impact" is claimed to measure cognitive load:
Impact attempts to answer the question: “Roughly how much cognitive load did the engineer carry when implementing these changes?”
A good engineer will attempt to find a low or no cognitive load solution to a problem! In general this will be faster and less error prone and cheaper! Reinventing the wheel has a very high cognitive load.
Search with an exact phrase. Put quotation marks around words "[any word]" to search for an exact phrase in an exact order. Keep in mind that searching with quotes might exclude relevant results. For instance, a search for "Alexander Bell" will miss pages that refer to Alexander G. Bell. Search Tips & Tricks – Inside Search – Google https://www.google.com/intl/en_u/insidesearch/tipstricks/all...
(1) Algorithms don't make decisions. People choose the algorithms and design the algorithms. The algorithms reflect the biases, failings, and agendas, both hidden and overt, of people.
(2) Hillary Clinton was severely damaged by factually accurate or mostly factually accurate articles on conflicts of interest involving the Clinton Foundation, her husband, and herself.
It is not just my personal opinion. Polls, granted polls have proven fallible in this election, showed public awareness of the Clinton Foundation conflicts of interest stories and that it was adversely impacting her campaign.
https://www.washingtonpost.com/news/the-fix/wp/2016/09/21/th...
An ABC/Washington Post poll in September showed that about 6 in 10 (59 percent) thought special favors had been done for Clinton Foundation donors.
This shows public was well aware of the issue.
In the context of an interview, it is highly likely a highly experienced software engineer will miss some of the corner cases. The exception would mostly be someone who has specifically studied and drilled on implementing isNumber(char * string) or specifically works in implementing these kind of functions which is rare. As I said there are libraries with these today.
This is one of the problems with these types of questions, including the CS 101 type questions that often make up purported coding tests during hiring. In most cases, they do not test what practicing software engineers actually do.
The cluster study that fingered Dugas as Patient Zero was used to argue that AIDS was a sexually transmitted disease rather than, for example, the results of cumulative toxic effects of drug abuse among the victims, notably the use of "poppers" by large fraction if not all of the original victims.
https://en.wikipedia.org/wiki/Poppers
The study traced sexual contacts and was interpreted to show a sexually transmitted disease that resulted in death in about 10 to 18 months after transmission.
Dugas was said, not just in Shilts book, to have continued having unprotected sex with other men despite having been warned by the researchers that he was carrying an as yet unidentified super-killer sexually transmitted disease.
How true is this? Who knows, but it was widely repeated in documentaries and other sources well before Shilts book. It seems to have been promoted by the CDC.
In the subsequent three decades, the purported latent period between infection and death of AIDS/HIV has been lengthened to at least ten years. The modern theory of the disease, to the extent that one can pin it down, is grossly inconsistent with the early studies and claims by the CDC.
In the mid 1980's, after HIV was proposed as the cause of AIDS, the government did what one would expect. They infected about two hundred chimpanzees, thought to be man's closest animal relative, with HIV, or more accurately fluids thought to contain HIV. Of these, two have died of symptoms said to be similar to AIDS.
http://discovermagazine.com/2002/may/featchimp
One, known as Jerom, apparently came down with the symptoms spontaneously.
http://www.releasechimps.org/chimpanzees/their-stories/jerom
The second was supposedly infected with Jerom's blood or fluids from him and so is not a spontaneous case.
One out of two-hundred is 0.5 percent, about the incidence of AIDS in the general human population.
Here is another article about the remarkable immunity of chimpanzees to HIV.
Since the 1980's there have been numerous high profile projects that adhered to each successive software development methodology and imploded, often with rancorous finger pointing.
Business people as well as many software developers are attracted to the will-o-the-wisp that there is a one size fits all magic formula for software development.
Software development is extremely varied. Some projects are small. Some are large. Some, like flight avionics or medical radiation control software, require very high levels of quality and reliability. Others do not. "Quality" means different things to different people and in different situations.
Some software is like the bridge that you never want to collapse. Some software is like the plastic picnic cups that you use a few times and expect to break easily. It depends on the situation, the goals, the priorities which vary substantially and depend on human judgment.
Software developers are varied. What is "readable" code for one won't be readable for another. Perceptions of software designs reflect variations in culture, background, training, personality, and so forth.
There is no one size fits all methodology and attempting to impose one is often costly and disastrous.
Developers vary as well. Some temperamentally find something like TDD useful. Others do not.
There is no one software development methodology to rule them all.
https://en.wikipedia.org/wiki/Source_Code_Control_System
Version control is not a new concept and was in widespread use in the 1980's.
:-)
Subversion's History
In early 2000, CollabNet, Inc. (http://www.collab.net) began seeking developers to write a replacement for CVS. CollabNet offered[3] a collaboration software suite called CollabNet Enterprise Edition (CEE), of which one component was version control. Although CEE used CVS as its initial version control system, CVS's limitations were obvious from the beginning, and CollabNet knew it would eventually have to find something better. Unfortunately, CVS had become the de facto standard in the open source world largely because there wasn't anything better, at least not under a free license. So CollabNet determined to write a new version control system from scratch, retaining the basic ideas of CVS, but without the bugs and misfeatures.
Note "retaining the basic ideas of CVS, but without the bugs and misfeatures."
derive
base a concept on a logical extension or modification of (another concept).
Nonsense. Version control software has existed and been in widespread use since at least the 1980's.
https://en.wikipedia.org/wiki/Revision_Control_System
Walter Tichy released RCS in 1982. CVS is derived from RCS, Subversion from CVS.
As a business, if I buy a computer on a company credit card, I know immediately that I owe $1200.00 at 23 percent interest per year. In contrast, if I develop a piece of software and decide to use global variables to speed up development, I may immediately gain a mythical man-month in development time BUT I have no idea how much it may slow me down in the future if at all. Sometimes global variables are the way to go and development will be consistently faster for the entire lifetime of the software compared to using local variables. This tends to happen, not surprisingly, when the data in the global variables needs to be used widely throughout the program.