HNHacker News
TopNewBestAskShowJobs

iblaine

716 karma · joined May 6, 2012

Data Engineer doing Data Engineer things www.iblaine.com
submissionscomments
iblaine··on Show HN: TimetoHerd – Days left until 70% of your country is Covid-19 vaccinated
Maybe replace the number of days with an actual date.
iblaine··on Ask HN: Who is hiring? (March 2021)
One Medical | San Francisco, CA or Austin, TX | Full-time | https://www.onemedical.com/careers/

At One Medical, we are passionate about revolutionizing the healthcare industry by offering a new approach to primary care. We combine people-centered design, technology, and a team of talented healthcare providers to give our members an amazing experience (in-house developed apps, infrastructure focused on healthcare, medical records, and virtual care to name a few).

Security at One Medical: https://www.onemedical.com/careers/security/ - Senior Security Engineer, Logging and Automation - (SF or Austin) - Senior Application Security - (SF or Austin)

Engineering and Corporate Roles: https://www.onemedical.com/careers/all-departments/

Our focus on technology and preventative healthcare has allowed us to help address aspects of the current covid crisis. We've been doing telemedicine for years, setting up outdoor covid stations across the nation, and as of last month we now offer antibody testing, outside of all of the other services we provide.

For a better understanding of One Medical, please check out our Instagram (https://www.instagram.com/onemedical), reach out to our Recruiting team (https://www.linkedin.com/in/sbunker) or me.

iblaine··on 23andMe to merge with VG Acquisition Corp. to become publicly-traded company
SPACs are today’s ICOs. For every good company like 23andMe we should expect dozens of bad ones.
iblaine··on Ask HN: Who is hiring? (February 2021)
One Medical | San Francisco, CA or Austin, TX | Full-time | https://www.onemedical.com/careers/

At One Medical, we are passionate about revolutionizing the healthcare industry by offering a new approach to primary care. We combine people-centered design, technology, and a team of talented healthcare providers to give our members an amazing experience (in-house developed apps, infrastructure focused on healthcare, medical records, and virtual care to name a few).

Security at One Medical: https://www.onemedical.com/careers/security/ - Senior Security Engineer, Logging and Automation - (SF or Austin) - Senior Application Security - (SF or Austin)

Engineering and Corporate Roles: https://www.onemedical.com/careers/all-departments/ - Staff Software Engineer, Charts (SF) - Senior IT Field Technician (Remote) - Senior Product Designer, Design Systems (SF)

Our focus on technology and preventative healthcare has allowed us to help address aspects of the current covid crisis. We've been doing telemedicine for years, setting up outdoor covid stations across the nation, and as of last month we now offer antibody testing, outside of all of the other services we provide.

For a better understanding of One Medical, please check out our Instagram (https://www.instagram.com/onemedical), reach out to our Recruiting team (https://www.linkedin.com/in/sbunker) or me.

iblaine··on Ask HN: Who is hiring? (January 2021)
One Medical | San Francisco, CA or Austin, TX | Full-time | https://www.onemedical.com/careers/

One Medical is the fastest-growing primary care system in the country with over ~100 locations in major cities nationwide such as Boston, Chicago, Los Angeles, New York, SF, DC, Atlanta, and most recently, Austin. Investment into our own technology represents one of our advantages so we expect you to have a significant impact on our members and hopefully our industry.

Security at One Medical: https://www.onemedical.com/careers/security/ - Senior Security Automation and Tooling Engineer - (SF or Austin) - Senior Application Security - (SF or Austin)

Engineering and Corporate Roles: https://www.onemedical.com/careers/all-departments/ - Staff Software Engineer, Charts (SF) - Senior IT Field Technician (Remote) - Senior Product Designer, Design Systems (SF)

Our focus on technology and preventative healthcare has allowed us to help address aspects of the current covid crisis. We've been doing telemedicine for years, setting up outdoor covid stations across the nation, and as of last month we now offer antibody testing, outside of all of the other services we provide.

For a better understanding of One Medical, please check out our Instagram (https://www.instagram.com/onemedical), reach out to our Recruiting team (https://www.linkedin.com/in/sbunker) or me.

iblaine··on Why Jira Sucks
Did someone at Zepel.io create this Why Jira Sucks site?
iblaine··on S3 Strong Consistency
AWS Re:invent talked about improvements to S3 earlier today. You can see that talk here https://virtual.awsevents.com/media/1_usujusng

For any AWS EMR/hadoop users out there, this means the end of emrfs.

For the uninitiated, emrfs recognizes consistency problems but does not fix them. It'll throw an exception if there's a consistency problem, even give you some tools to fix the problem, but the tools may give false positives. The result is you've got to fix some consistency problems yourself, parse items out of the emrfs DynamoDB catalog, match them up to s3, then make adjustments where needed. It's an unnecessary chore.

It surprises me that this issue has not got more attention over the years and thankfully it'll be solved soon.

iblaine··on Ask HN: Who is hiring? (December 2020)
One Medical | San Francisco, CA or Austin, TX | Full-time | https://www.onemedical.com/careers/

One Medical is the fastest-growing primary care system in the country with over ~100 locations in major cities nationwide such as Boston, Chicago, Los Angeles, New York, SF, DC, Atlanta, and most recently, Austin. Investment into our own technology represents one of our advantages so we expect you to have a significant impact on our members and hopefully our industry.

Security at One Medical: https://www.onemedical.com/careers/security/ - Senior Security Automation and Tooling Engineer - (SF or Austin) - Senior Application Security - (SF or Austin) - Sr. Security IR - (Austin)

Engineering and Corporate Roles: https://www.onemedical.com/careers/all-departments/

Our focus on technology and preventative healthcare has allowed us to help address aspects of the current covid crisis. We've been doing telemedicine for years, setting up outdoor covid stations across the nation, and as of last month we now offer antibody testing, outside of all of the other services we provide.

For a better understanding of One Medical, please check out our Instagram (https://www.instagram.com/onemedical), reach out to our Recruiting team (https://www.linkedin.com/in/sbunker) or me.

iblaine··on AWS Managed Workflows for Apache Airflow
Confluent seems to be thriving ever since AWS launched their own Kafka managed service. With that as an example, this could be a good thing for Astronomer.io.
iblaine··on Personal Data Warehouses: Reclaiming Your Data
>I believe all data warehouses are limited by the quality of their data model.

This is common and most organizations overcome this problem with a higher effort downstream by Analysts & BI. I personally prefer 3NF tables, event streams or Kimball, in that order, as solutions to create a better DW. The reality is ugly behavior in data modeling can be overcome by Analysts & BI tools. And that seems to be the norm these days...use a DW to structure data to make data accessible, then let Analysts figure out the metrics. With the recent emphasis on tools like Looker or Tableau, this pattern seems to be working.

iblaine··on Linux Command One-Liners
My most obscure cool one liner...

1. Pointing your webserver to your current directory. Simplifies copying files to your local box.

    The simple python 3.x way...

    python -m http.server 8001

    And the original way I used to do it...

    function sfiles() { x=`ipconfig getifaddr en0`;echo $x:8001;ruby -run -ehttpd . -p8001; }

    And please use this with *extreme* caution because any files you've got access to on your remote box will have no authentication when they're exposed via HTTP.
My most frequently used one liner...

2. Regex searching files in directories.

    grep -inR <keyword> *

    The Silver Searcher, aka ag, is better but not natively installed so...
iblaine··on AWS releases Glue Databrew, a visual ETL tool
Having used GUI ETL tools for years (SSIS, Informatca, Talend, Appworx) and now using Airflow(Prefect is an excellent alternative btw), I hope to never go back. Great to see Glue improving and for the industry’s sake I hope it doesn’t catch on. Most ETL should be treated as code. As code, ETLs are easier to write, maintain, and manage complexity.
iblaine··on Ask HN: Who is hiring? (November 2020)
One Medical | San Francisco, CA or Austin, TX | Full-time | https://www.onemedical.com/careers/

One Medical is the fastest-growing primary care system in the country with over ~100 locations in major cities nationwide such as Boston, Chicago, Los Angeles, New York, SF, DC, Atlanta, and most recently, Austin. Technology has been a key to our success and we believe in building as much of it in-house as possible. Depending on your department, these are some of the core technologies you can expect to interface with: Angular, React, Python, AWS, Spark, Ruby, MySQL, Android, GraphQL, Java, Cypress, Hadoop, PySpark, Tensorflow, AI/ML

- Senior Security Automation and Tooling Engineer - (SF or Austin) https://www.onemedical.com/careers/security/

- Senior Application Security - (SF or Austin) https://www.onemedical.com/careers/all-departments/2298809/

- Security IR - (SF) https://www.onemedical.com/careers/all-departments/2247894/

- Sr. Security IR - (Austin) https://www.onemedical.com/careers/all-departments/2326174/

- Infrastructure Security (SF or Austin) https://www.onemedical.com/careers/all-departments/2344612/

- Engineering jobs (SF) - https://www.onemedical.com/careers/engineering/

- Clinical/IT/Doctors/Legal/Marketing/Operations/Sales/etc - https://www.onemedical.com/careers/all-departments/

Our investment in technology and focus on preventative healthcare has allowed us to help address the current covid crisis. We've been doing telemedicine for years, setting up outdoor covid stations across the nation, and as of last month we now offer antibody testing, outside of all of the other services we provide.

For a better understanding of One Medical, please check out our Instagram (https://www.instagram.com/onemedical), reach out to our Recruiting team (https://www.linkedin.com/in/sbunker) or me.

iblaine··on PySpark Style Guide
Could not agree more. Similar to the The Principle of Least Privilege [1], I prefer to use SQL over pyspark if possible. [1] https://us-cert.cisa.gov/bsi/articles/knowledge/principles/l...
iblaine··on Data engineering learning path with recommended resources
Here's a worthwhile guide that shows a learning path, includes more skills and seems easier to comprehend https://github.com/datastacktv/data-engineer-roadmap
iblaine··on Ask HN: Who is hiring? (October 2020)
One Medical | San Francisco, CA or Austin, TX | Full-time | https://www.onemedical.com/careers/

One Medical is the fastest-growing primary care system in the country with over ~100 locations in major cities nationwide such as Boston, Chicago, Los Angeles, New York, SF, DC, Atlanta, and most recently, Austin. Technology has been a key to our success and we believe in building as much of it in-house as possible. Depending on your department, these are some of the core technologies you can expect to interface with: Angular, Python, AWS, Spark, Ruby, MySQL, Android, GraphQL, Java, Cypress, Hadoop, PySpark, Tensorflow, AI/ML

- Senior Security Automation and Tooling Engineer - (SF or Austin) https://www.onemedical.com/careers/security/ - Senior Application Security - (SF or Austin) https://www.onemedical.com/careers/all-departments/2298809/ - Security IR - (SF) https://www.onemedical.com/careers/all-departments/2247894/ - Sr. Security IR - (Austin) https://www.onemedical.com/careers/all-departments/2326174/ - Infrastructure Security (SF or Austin) https://www.onemedical.com/careers/all-departments/2344612/ - Engineering jobs (SF) - https://www.onemedical.com/careers/engineering/ - Clinical/IT/Doctors/Legal/Marketing/Operations/Sales/etc - https://www.onemedical.com/careers/all-departments/

Our investment in technology and focus on preventative healthcare has allowed us to help address the current covid crisis. We've been doing telemedicine for years, setting up outdoor covid stations across the nation, and as of last month we now offer antibody testing, outside of all of the other services we provide.

For a better understanding of One Medical, please check out our Instagram (https://www.instagram.com/onemedical), reach out to our Recruiting team (https://www.linkedin.com/in/sbunker) or me.

iblaine··on Why is Snowflake so Valuable?
Snowflake seems to be part of a segment of companies that are defying economy reality during covid: $TSLA, $ZM, $SNOW. These companies have huge P/S ratios and a higher volume of options activity relative to their underlying stock. A rational person may look at metrics, but we're not in a rational market. The strange behavior of the market may explain these valuations more than anything else. And maybe 3 digit P/S ratios are the new normal.

[1] https://www.ft.com/content/b330e091-2a59-4527-b958-9213731a5...

iblaine··on Why is Snowflake so Valuable?
Snowflake owes much of its performance benefits to "micro-partitions" [1]. BigQuery is a worthwhile comparison. MySQL, PostgreSQL, SQL Server, and Vertica are not close equivalents.

[1] https://www.infoq.com/presentations/snowflake-automatic-clus...

iblaine··on Ask HN: Who is hiring? (September 2020)
One Medical | San Francisco, CA or Austin, TX | Full-time | https://www.onemedical.com/careers/

One Medical is the fastest-growing primary care system in the country with over ~100 locations in major cities nationwide such as Boston, Chicago, Los Angeles, New York, SF, DC, Atlanta, and most recently, Austin. Technology has been a key to our success and we believe in building as much of it in-house as possible. Depending on your department, these are some of the core technologies you can expect to interface with: Angular, Python, AWS, Spark, Ruby, MySQL, Android, GraphQL, Java, Cypress, Hadoop, PySpark, Tensorflow, AI/ML

- Senior Security Automation and Tooling Engineer - (SF or Austin) https://www.onemedical.com/careers/security/

- Senior Application Security - (SF) https://www.onemedical.com/careers/security/

- Security Incident Response - (SF) https://www.onemedical.com/careers/security/

- Engineering jobs - https://www.onemedical.com/careers/engineering/

- Clinical/IT/Doctors/Legal/Marketing/Operations/Sales/etc - https://www.onemedical.com/careers/all-departments/ Our investment in technology and focus on preventative healthcare has allowed us to help address the current covid crisis. We've been doing telemedicine for years, setting up outdoor covid stations across the nation, and as of last month we now offer anti-body testing, outside of all of the other services we provide.

For a better understanding of One Medical, please check out our Instagram (https://www.instagram.com/onemedical), reach out to our Recruiting team (https://www.linkedin.com/in/sbunker) or me.

iblaine··on Snowflake S-1
Snowflake is the best all around DW product out there. It commonly gets compared to Redshift, but Amazon built Redshift on top of ParAccel's technology. Snowflake built its database from scratch. Most of the Snowflake founders have PhDs with an emphasis on distributed systems and I think you see that in the product.

I can't say enough good things about snowflake, and I have plenty of criticism to throw at hadoop, redshift, asterdata & vertica.

iblaine··on Launch HN: Hubble (YC S20) – Monitor data quality inside data warehouses
What does the tech stack look like?

Is there any caching for those situations where you may read the same historical data over & over?

iblaine··on Netflix Conductor: Open-source workflow orchestration engine
Since this has come up several times, Airflow is an orchestration tool for ETL jobs (long running complex processes) and Netflix Conductor is an orchestration tool for micro-services (short running simple processes).
iblaine··on Show HN: Dropbase 2.0 – Turn offline files into live databases
Gitlab has a similar project that will load google sheets as a dataframe.[1]

At my current company, we load google sheets into s3, then mount those files as external tables. There has not been a commit in years, meaning it has worked out well for us.

What seems to be missing in these solutions, and what Dropbase provides, is a UI to guide users through the process.

[1] https://gitlab.com/gitlab-data/analytics/tree/master/extract...

iblaine··on My best SEO tactic so far
> Create what you hoped your search would find

This is the content is king mantra. Ideal SEO is the point that delivers the best results with the fewest consequences. That’s a subjective rule, so it’s hard to say creating content works for everyone.

Point of saying this is you should strive to appease search engines by creating useful content, but the real world is messy. Some industries need to bend the rules to survive.

iblaine··on Ask HN: Who is hiring? (August 2020)
One Medical | San Francisco, CA or Austin, TX | Full-time | https://www.onemedical.com/careers/ One Medical is the fastest-growing primary care system in the country with over ~100 locations in major cities nationwide such as Boston, Chicago, Los Angeles, New York, SF, DC, Atlanta, and most recently, Austin. Technology has been a key to our success and we believe in building as much of it in-house as possible. Depending on your department, these are some of the core technologies you can expect to interface with: Angular, Python, AWS, Spark, Ruby, MySQL, Android, GraphQL, Java, Cypress, Hadoop, PySpark, Tensorflow, AI/ML

- Senior Security Automation and Tooling Engineer - (SF or Austin) https://www.onemedical.com/careers/all-departments/2135833/

- Senior iOS Engineer - (SF) https://www.onemedical.com/careers/all-departments/2113005/

- Engineering jobs - https://www.onemedical.com/careers/engineering/

- Clinical/IT/Doctors/Legal/Marketing/Operations/Sales/etc - https://www.onemedical.com/careers/all-departments/

Our investment in technology and focus on preventative healthcare has allowed us to help address the current covid crisis. We've been doing telemedicine for years, setting up outdoor covid stations across the nation, and as of last month we now offer anti-body testing, outside of all of the other services we provide.

For a better understanding of One Medical, please check out our Instagram (https://www.instagram.com/onemedical), reach out to our Recruiting team (https://www.linkedin.com/in/sbunker) or me.

iblaine··on NASA Astronauts Return Home in SpaceX's Crew Dragon Spacecraft [video]
The SpaceX crew land in Florida at 2:48 p.m. ET on Sunday.
iblaine··on Data Consistency Checks
There seems to be a trend where companies are creating tools to monitor data quality...

* Great Expectations: https://github.com/great-expectations/great_expectations

~2 years old, does data profiling

* Data Sentinel: https://engineering.linkedin.com/blog/2020/data-sentinel-aut...

It's not open sourced however.

* Data Anomaly Detector: https://www.slideshare.net/iblaine/using-airflow-for-tools-d...

Something I built at One Medical that is not open sourced...but this deck has some interesting concepts in it. The approach here is to minimize the effort, lines of code, number of clicks needed, to test your data.

* Amundsen: https://github.com/lyft/amundsen

This is open sourced, and some contributes are building features for data quality analysis. Amundsen seems like it has a bright future.

iblaine··on Systems Design for Advanced Beginners
I don't. I came across this in 2018 and recall the course being for sale for $80.
iblaine··on Systems Design for Advanced Beginners
Systems Design is a formidable topic because it can go in so many directions. This is a good guide. Might I also recommend something that includes caching, partitioning, indexing, and NoSQL vs SQL. More detail can be found here https://www.educative.io/courses/grokking-the-system-design-...
iblaine··on Ask HN: What's the worst piece of software you use everyday?
Hadoop was never meant to be user friendly. Which is why you have commercial versions of it like Cloudera & AWS EMR, meant to fill that void in the industry.

FWIW, I feel your pain on a daily basis, but I do like Hadoop as a low cost massively distributed db.

← PreviousPage 3 of 14Next →