Prices of Home Depot runs fluctuated massively for certain baskets of goods in 2021, like never before.
1,637 karma · joined November 3, 2012
Personal blog: https://neapowers.com/
Github: https://github.com/mrpowers
From New York, spend most of my time in Colombia & Brazil.
Speak Spanish fluently, learning Portuguese.
Prices of Home Depot runs fluctuated massively for certain baskets of goods in 2021, like never before.
* using "owners equivalent rent" rather than house prices: https://www.bls.gov/cpi/factsheets/owners-equivalent-rent-an...
* factoring in technology as deflationary. yea, your phone is 20% more expensive, but it has twice the computation power, so it's actually a deflationary component
* substituting goods to account for substitution bias (https://www.frbsf.org/economic-research/publications/economi....). Substituting chicken for beef was heavily criticized.
Back in July, Siegel pointed out that the M2 money supply is 30% over pre-pandemic levels and predicted that would translate into 20% cumulative inflation over the next 3 years. Here are my talk notes / video links if you're interested: https://neapowers.com/investing/jeremy-siegel/
In July Siegel said "There is zero need for the Fed to be buying $120 billion in bonds every month given market conditions." I don't know why the Fed is buying mortgage backed securities in the midst of an apparent housing bubble.
Larry Summers was voicing inflation concerns back in March: https://www.youtube.com/watch?v=PBnaahSe7JU&ab_channel=Bloom...
The main macroeconomists I follow (Siegel, Gundlach, Summers) are all puzzled by the Feds actions recently.
The only Typelevel library I've used is frameless, which I blogged about here: https://mungingdata.com/apache-spark/frameless-typed-dataset.... Don't think the lib is really usable. Typelevel stuff is not what I'm referring to when I say better Python. I'll try to make another blog to clarify.
I think Scala is a better language than Python. But for some reason it just seems like Scala is full of mind bending code that I can't understand. Full disclosure: I am way less smart than lots of the Scala ppl I've worked with.
I agree that Scala works well in some skilled hands (see Li's libs), but the people that write the most terrible Scala programs actually know a lot about Scala. Scala has a unique way of encouraging genius programmers to write crazy code. People that know little Scala can just use it as a "better Python" language and actually write amazing Scala code.
I agree that Scala codebases that follow the Principle of Least Power can be amazing: https://www.lihaoyi.com/post/StrategicScalaStylePrincipleofL...
Scala programmers generally frown on programmers that want to use it as a better Python and I think it should be the opposite.
If you follow the design principles outlined in this doc (https://www.lihaoyi.com/post/StrategicScalaStylePrincipleofL...), Scala projects are really refactorable, which is nice.
But I've also worked on lots of large Scala codebases that are completely incomprehensible and painful to change. Scala allows for devs to make beautiful monoliths, but that's not usually what happens.
Here's a blog post I wrote on reading / writing JSON using one of Li's libs: https://mungingdata.com/scala/read-write-json/ There's now a completely sensible option for JSON in Scala.
But there are also 16 other Scala JSON lib options: https://stackoverflow.com/questions/8054018/what-json-librar...
It's so annoying working on a Scala project with dependencies that all use different JSON libs.
Even new, awesome projects like Delta Lake still use Scala dependencies like Scalatest and SBT that should be replaced with Li's libs in my opinion (utest & Mill).
Scala could be an amazing ecosystem if the community unified around Li's libs. It just doesn't seem like that's going to happen. Li wrote the "whats wrong with SBT" article back in 2017 and it seems like everyone is still using SBT.
I wrote an article on the maintenance challenges with Scala projects that trended on HN a while back: https://mungingdata.com/scala/maintenance-nightmare-upgrade/. Li's libs would solve a lot of these issues cause most of them are intentionally dependency free.
It was frustrating for me to see how Li basically singlehandedly fixed the Scala library ecosystem, but how none of the projects I actually used depended on his libs. I ended up leaving Scala after several years even though I like the language when it's used a certain way.
Scala is such a powerful programming language that has so much potential to be mainstream, but it's just so abused by "genius programmers".
> since that number is not based on actual sales of the land in question, it's just hand-waving
Actual land is sold in a lot of markets. That sets a great benchmark for "comparable transaction" types of valuation analyses which tend to be the most reliable.
If one acre of land is sold, there is a great baseline for the whole surrounding area.
I passed all 3 CFA exams and agree it's not an exact science, but as far a valuation analyses go, this one is pretty easy. Valuing something like intellectual property, goodwill impairment analyses, 409a valuations are much harder.
I'd argue that the factors you've listed are relatively easy to factor into an appraisal.
Property taxes, in comparison, are really hard to compute. Each jurisdiction has different rules. There are different valuation methodologies. If you add a 2 car garage, what the useful life. What depreciation schedule should be used. How should inflation be accounted for.
I agree with your comment generally and am not sure what part of the original comment doesn't make sense to you.
EDIT: Reread my original comment and see your point now. I phrased the "easy" part too strongly. I mean "easy" in relative terms... compared to what's being done currently, but agree the wording wasn't great.
Land taxes are great because there is less of a deadweight loss. Governments need money to operate. All else equal, we should get governments the money they need without wasting people's time to comply with complicated tax laws.
Humans also generally agree that progressive taxation is a good idea. Rich people should pay more to fund the government. Rich people own a lot of land, so they'd pay a lot of land tax. Poor people generally don't own any land and wouldn't be subject to this tax. Land taxes align well with what a lot of people think is fair.
Rich people own most of the land. There are no tax loopholes to evade land taxes.
EDIT: some support: https://www.progress.org/articles/the-implementation-of-land...
> virtually all economists agree that land value tax has zero "deadweight loss"–a fancy word for a drag on the economy that makes certain activities no longer profitable
Property taxes tax the land + the building. Building taxes are complicated to assess. They require tax assessments, appraisers, tax code, accountants to compute depreciation schedules, etc. The deadweight loss of property taxes is the economic loss of having resources devoted to filling out property tax filings. Appraisers and accountants would do other productive economic activities if they didn't spend all their time filling out tax forms.
Property taxes also distort incentives. If you improve your property, then you pay more tax. Property taxes can discourage you from using your land to its highest and best use.
Land taxes offer the opposite incentives. They encourage you to make your property better.
Land taxes would arguably help a lot of the housing crises. Landowners have to pay a lot more taxes if they want to convert a lot with a single family home to a 4-plex. A land tax wouldn't impose an additional tax to the landowner for improving the value of their land.
> the big government spending programmes of the post-Covid era present a new opportunity to try and push money into more productive sectors, which could ultimately bring wealth and growth back into alignment
We've seem some huge big government spending programs recently (monetary expansion and fiscal stimulus) and it's pretty clear the money has flowed into speculative asset classes, equities, and real estate. The conclusion is a non sequitir. Government spending programs have arguably caused the real estate bubble, not sure why more of them would have a different impact.
Here are some government programs that would probably help lowering housing prices:
* allow for multifamily housing
* remove the mortgage interest deduction
* implement land taxes instead of property taxes
* have the Fed stop purchasing mortgage backed securities (in the midst of a housing bubble)
* get the government out of the mortgage securitization business
Here's what I've found thus far with Dask:
* way easier to build custom distributed compute systems with Dask than other alternatives. Dask futures & delayed APIs give you access to the "engine" so you can build your own custom race car
* Lots of data scientists are a lot more productive with Dask compared to Java / Scala technologies they're not comfortable with (I have a popular Spark blog / Spark books & love Spark, but some ppl just can't get productive with Spark)
* Dask cluster visualizations are so nice. So easy to understand how clusters are being computed in real time.
* We've been working closely with NVIDIA folks to provide real cutting edge GPU support.
I'm a believer in multi-tech data pipelines in the future. A data engineering team that loves Scala may write some ETL code in Spark and then pass off the baton to a data science team that loves Python and uses Dask to train machine learning models.
Seems like most other players in the data space want to take over an organization's entire data platform. I like how Dask likes to play nicely as part of the overall PyData ecosystem. I've always liked the Unix philosophy of building little tools that can be easily composed to solve a variety of different problems and feel at home in the PyData ecosystem.
Data mining is common in finance. Finance PhDs are constantly identifying new "factors" (e.g. value, momentum, price/book) based on historical data that don't pan out as time goes on. See here for some academic research: https://www.researchaffiliates.com/publications/articles/710...
Robert Shiller has does extensive research on how psychology impacts real estate prices. Monetary policy isn't standard right now. Machine learning isn't always a good tool for making predictions. Look at Shiller's interviews and surveys for predicting home prices. He doesn't use machine learning or advanced analytics.
The libs are implemented with advance Scala programming and expose elegant, Python-like interfaces. Li's ability to make beautiful abstractions is incredible. His book Hands on Scala Programming shows you how to create different applications using his library ecosystem. It's a magnum opus that's worth reading, even if you're not interested in Scala.
Li's Principle of Least Power blog post is another great read. Scala would be a much more popular language if the community just used his libraries and followed his design patterns.
Use the new string dtype that requires way less memory, see this video: https://youtu.be/_zoPmQ6J1aE. object types are really memory hungry and this new type is a game changer.
Use Parquet and leverage column pruning. `usecols` doesn't leverage column pruning. You need to use columnar file formats and specify the `columns` argument with `read_parquet`. You can never truly "skip" a column when using row based file formats like CSV. Spark optimizer does column projections automagically - you need to do them manually with Pandas.
Use predicate pushdown filtering to limit the data that's read into the DataFrame, here's a blog post I wrote on this: https://coiled.io/blog/parquet-column-pruning-predicate-push...
Use a technology like Dask (each partition in a Dask DataFrame is a Pandas DataFrame) that doesn't require everything to be stored in memory and can run computations in a streaming manner.
Paul Graham also talks about the power of different programming languages in Beating the Averages, see The Blub Paradox: http://www.paulgraham.com/avg.html
Powerful / weird languages usually aren't the productivity hack you'd hope. A few team members might be more productive with a weird language, but other folks will have trouble understanding their code, so they'll be less productive.
Weird programming languages also require different types of library support (programmers that are passionate about immutability can't use a JSON library that mutates stuff). So they'll build a separate JSON library, but that fractures communities.
It's great to learn about different languages, but programmers seem susceptible to the allure of extra productivity from weird languages and that's not what I've seen.
Specifically this line "So if you want to invest two years in something that will help you succeed in business, the evidence suggests you'd do better to learn how to hack than get an MBA."
I spent 5 years building a good business school resume and this post encouraged me to try out programming instead. The flexibility of a programming job let me escape the NYC / San Fran scene and start living abroad. Having way more fun living in different countries around the world now.
* Parquet stores schema in the metadata, so schema inference isn't required (schema inference is expensive for big datasets)
* Parquet files are columnar so individual columns can be grabbed for analyses (Spark does this automatically). This is a huge performance improvement.
* Row groups contain min/max info for each column, which allows for predicate pushdown filtering data skipping. Highly recommend playing with PyArrow + Parquet to see the metadata that's available.
* Columnar file formats are easier to compress. Binary files are way smaller than text files like CSV even without compression.
I wrote a blog post that shows how a Parquet query that leverages column pruning and predicate pushdown filtering can be 85x faster than an equivalent CSV query: https://coiled.io/parquet-column-pruning-predicate-pushdown/
CSVs are great for small datasets when human readability is an important feature. They're also great when you need to mutate the file. Parquet files are immutable.
It's easy to convert CSVs => Parquet/Delta/Avro with Pandas/Dask/Spark.
The world is already shifting to different file formats. We just need to show folks how Parquet is easy to use and will greatly increase their analysis speeds & they'll be happy to start using it.
Small nit: the article implies Apache Arrow is a file format. It's a memory format.
Learning about logical fallacies and identifying them in conversations is great. Don't tell the counterparty of their logical fallacies in conversations cause that's off putting. Just note them internally for a more rational inner dialogue.
Learning other languages and cultures is another way to learn about how different societies interact with objective truth. Living other places taught me a lot about how denial works in different places.
Thinking rationally is quite hard and I've learned how to abandon it in a lot of situations in favor of human emotions. How someone feels is more important than how they should feel.
* details are so technical that they're hard to explain without "just look at the code type explanations". I've run into this roadblock trying to explain the Spark Catalyst optimizer.
* Not truly understanding all the details, so you can't explain the tech. Sometimes the documentation writer can't provide a "plain English" overview of the technology.
* the details are the proprietary IP / secret sauce of the company
The inner workings of a proprietary query engine are a good example of all three. It may be great tech, but hard to explain and part of it are the secret sauce that they don't want to expose. Not exposing something doesn't necessarily mean they're hiding something / being dishonest.
I have a popular tech blog (https://mungingdata.com/) and have a ton of goodwill with my users, mainly cause I'm not trying to sell them anything. You can provide users with equally informative content if you can avoid making it overly biased.
Developers are understandably skeptical in the big data space because they're often given biased benchmarks. It's hard to replicate benchmarking results (e.g. person X says that XYZ runtime can run a certain query in 1 second, then they try out the tech and it's way slower).
Don't be surprised if you're building a big data technology and devs don't like your biased benchmarking.
If you give honest results and try to inform users about your project objectively (the pros and cons), then they're more likely to respond well. Your goal should be to give relevant information, even if it's only tangentially related to your product. Your product should sell itself when users are provided facts that aren't tinted by rose colored glasses. Disclaimer: I work as a Tech Evangelist at Coiled.