243 karma · joined October 21, 2019
Good parents with good parenting skills raise good kids. Bad parents/single parents - it is much harder to raise kids and give them good education.
if you want reforms - you need to adopt Scandinavian policies in America (high taxes, universal healthcare, universal childare, universal K-12, universal college system) - that way it will be easier to raise well educated kids
It is so much easier to send one/two americans as expats to India/East Europe to manage outsourced staff and the trend will just accelerate.
unless it is live production support, for which I get incident notification from ServiceNow anyways
There should be also load balancing/HA, rate limiting, authn/authz, injection, all sorts of protections against OWASP top10 type attacks.
Apple+Appstore is actually a repeat of MSFT+IE monopoly and should be broken up as such
definitely not passing secrets in ENV (cause any process can access ENV and exfiltrate) or as command line argument (cause they will be logged as all tty commands are)
rule #2: its not only secrets management, the whole stack should go through security hardening and regular security review.
What I found is that if data is high quality - then ELT is totally fine, often times it ends up being just EL without much T. But if the data is crap, and you have a lot of wide columns, then even loading it takes time, before we even get to processing stage. In this scenario ETL works much faster.
you can configure your job to retry unless it returns success - and unless someone nukes your windows machine - it will get executed. if you are afraid that your windows machine will get nuked - then you can use SQL Server Agent on High Availability Cluster - and it will do the job.
if you want your jobs as a code - you can code them in Powershell and store in git/azure devops repo. You can deploy your jobs with the same powershell, or even use a CI/CD pipeline to do that.
1. Does airflow have native (read high speed) connectors to destination databases (oracle, mysql, mssql) ?
2. How the typical ETL in Airflow compares to one in Informatica/SSIS in terms of speed of development, performance (throughput and latency), memory consumptipon? Is it the same speed, or slower due to using Python interpreter?
3. Is it easy or hard to use parallel transformations with processes/threads/async ? For example, ingest data from your source in 20 threads at once, as opposed to serial processing
When your income crosses certain threshold, it makes sense to spend hundreds of thousands on consultants for tax planning, optimizing, etc, usually staffed with former tax auditors and lawyers
in cloud - it all new. people are still figuring out how to deploy their software so that it works both for users and developers. That's why on average onprem is more secure than cloud.
It is a new and ever changing field, there are many cloud vendors and their product line and configurations change all the time - meaning it will take a lot of time until majority of IT specialists become familiar with configuring secure cloud and majority of users of those cloud services will not make security mistakes.
Why is PHP trying to become java so hard, if there is already java and .NET ?
Until some major data drift happens, but you would notoce it anyways
getting a feedback from your early customer will be more than enough to keep you motivated