tkyjonathan

Misadventures in AWS Lambda, connection pooling and killing the database with unclosed transactions.

Isn't this DataOps?

Almost none of the company leaders or even VCs fully understand what AI even is or does. They just like to hear thats its there.

If you don't have some AI in your company, you won't get investors.

"Normalization was built for a world with very different assumptions. In the data centers of the 1980s, storage was at a premium and compute was relatively cheap."

But forget to do normalisation and you will be paying 5 figures a month on your AWS RDS server.

"Storage is cheap as can be, while compute is at a premium."

This person fundamentally does not understand databases. Compute has almost nothing to do with the data layer - or at least, if your DB is maxing on CPU, then something is wrong like a missing index. And for storage, its not like you are just keeping old movies on your old hard disk - you are actively accessing that data.

It would be more correct to say: Disk storage is cheap, but SDRAM cache is x1000 more expensive.

The main issue with databases is IO and the more data you have to read, process and keep in cache, the slower your database becomes. Relational or non-relation still follows these rules of physics.

Location: UK, Horsham Remote: Yes Willing to relocate: No Technologies: Data Engineering, Data Architecture, Data Performance Engineering, Java, Python, SQL. Résumé/CV: https://www.linkedin.com/in/jonathanlevin/ Email: mail@jonathanlevin.co.uk

Location: West Sussex, UK

Remote: Yes

Willing to relocate: No

Technologies: Data Architect, Data Modeling, Data Performance Engineer, SQL, MySQL, Percona, MariaDB, Galera, Postgres, Kafka, ElasticSearch, RDS, Redshift, Infobright, Java, Python, R.

Resume/CV: https://www.linkedin.com/in/jonathanlevin/

Email: mail@jonathanlevin.co.uk

This is obvious to me. Since Hadoop came out, (a lot of) people have been giving up on even forming algorithms and just dumping data into machine learning and hoping for the best. I recall someone high up at Google complaining about it.

We need to get back to forming algorithms as well as concepts and first principles. We cannot and should not expect ML to brute force finding patterns and just sit back and relax.

Here is another prediction for you: we will not solve ray-tracing in games and movie CGI with more hardware. We will need some algorithm that gets us 80-90% of the way there in a smart way.

I said this 3 years ago, the future is having an SQLite DB inside a container for each one of your customers.

/s

If its minimum of 70k, then its probably a good idea, because you would only hire if you absolutely have to.

37signals would be proud.

If you decentivize the most productive people, then you might hurt the rest of the company.

I reduce my RDS bill by 66% by keeping it on EC2 and using Percona Server.

Batching is the multi-threadedness of databases.

Its also important to remember that in databases, you are more often optimising for IO usage than CPU.

That could be a fault of complex physical data models that need simplification.

It is worth mentioning that there are some data-process heavy work that can be done much faster and more efficiently in SQL.

Real life example for a regulatory batch job:

6 tomcat servers + 1 RDS. 30X lines of code + UTs in java. 30+ minutes time.

In SQL No tomcat servers + 1 RDS instance. 1X lines of code(SQL) + UTs (in java). 3+ minutes

Here is a good book on it: https://www.amazon.co.uk/Relational-Database-Programming-Set...

Imagine my shock. Asking for regulation simply to attack the competition or new entrants..

This is very impressive. The average lifespan of a company on the S&P 500 is 15 years.

I'm still on pen and paper..

I feel that philosophy of systems and theory of constraints would be a better philosophy to use with computer systems.

And a lot of those lakes remain untouched for years..

Already started a DataOps team in my company!

Where has this been since 2015?

This is going to be your next cloud agnostic data warehouse.

Not sure which DB you are using, but you can load the csv file into the DB directly on a single thread using something like LOAD DATA INFILE.

If you have some good indexes and do some push-down work (give the database aggregation tasks to do instead of your python code), you should probably be more than fine.

For a 250Gb file.. should be ok.. maybe add some partitioning too.

Sysadmins could code scripts since day 1.

Its not like they ran the backups at midnight manually.

You can persuade them to invest in nuclear micro-reactors and hydro-electric.

I was under the impression that parts of the ozone have been repairing..

> his massive short on the pound

In 2016 or recently? because if its recently then the pound has gone up.

Think about having kids as like starting your own start up which you want to invest in for years to come and will very likely IPO with huge investment one day.

Could you point out the graph that shows that wages haven't improved in the last 40 years?

I would rather focus on reducing absolute poverty than reducing inequality (gini-coefficient).

I am not even sure that developers around me know how to do concurrency, since moving to micro services.