HNHacker News
TopNewBestAskShowJobs

halfeatenpie

256 karma · joined May 2, 2017

PhD Climate Risk Management focusing on Resilience and Adaptation. Climate Risk Lead for a Big Tech company.
submissionscomments
halfeatenpie··on I was wrong about spreadsheets (2017)
See that's the thing.

You don't need Access Control because it's just an excel file and whoever has the file has the planning model. In most organizations like this, there's only like 5 or 8 people who all work together on a team, so it's not like 50-some people who are separated. Excel is a tool, a tool that you use to get things done. I mean even programming solutions have issues such as Knight Capital Group's $460 million loss due to mistakes in their code and deployment processes. Planning modules does not mean Operational modules. It's a big difference/jump between the two.

Again, these are all tools. What's important is how you use it and how do you properly validate these calculations. the thing you also need to understand is that no critical infrastructure planning is perfectly to-the-dot numbers. Our systems are far too complex, built off of human operational decisions, and have so many unknown losses that planning modules are based on scenarios. Also we're talking about thousands of these agencies around the world and their critical infrastructure planning teams are mostly 5 to 8 people. The current agency I'm working with has hired around 3000 employees to operate the systems but only 5 or so are really in-the-deep running these models and building these plans.

These people are not programmers. They spent their time learning about resource planning, mathematics, operation theory, physics, and engineering. Excel is an incredibly user-friendly tool that lets them automate a tremendous amount of their tasks. They're all very intelligent and can definitely learn how to code, but that takes their time away from more critical skills and tasks they need to accomplish.

The most important solution to the problems you've stated is the workflow. You setup a proper workflow, and risks of these problems should be minimized.

halfeatenpie··on I was wrong about spreadsheets (2017)
What I specifically handle is Water. Starting from the natural system (weather generator + Hydrology) through to the human system (Demand model (showing increase over time and temperature increase) + system operations (dams, reservoirs, treatment facilities, pipes, etc.) to delivery to customers.

The big problem we answer is "How can we continue delivering water to people while under climate change uncertainty? What policies or new infrastructure can we build so we remain robust and resilient under these new conditions?"

halfeatenpie··on I was wrong about spreadsheets (2017)
Honestly, people handle sharing via Google Drive/Dropbox/Enterprise solution plus file naming convention.

The current enterprise network I'm on? They block Box, Dropbox, and a lot of other solutions. They instead use Citrix's ShareFile and a NAS setup with file naming.

Also remember, surprisingly most of these critical infrastructure planning isn't handled by hundreds of people, at most it's a team of 5 people who then present their findings and suggestions to the C-suite decision makers. It's not as big of a problem as many people think.

halfeatenpie··on I was wrong about spreadsheets (2017)
I work in critical infrastructure planning. My organization builds software in R, Python, and other programming languages customized for these major organizations.

So many critical infrastructures, billions of dollars in planning, and just systems are built out of Excel. It's amazing. You'd assume something that services millions of people a day would have some more sophisticated and customized solution, but you're wrong.

The reason is because most people know how to use Excel. Most people know how to handle it, use it, modify it, build up from it. You don't need to get a regular support contract from a company for your custom-built spreadsheets. If something critical in that "Excel software stack" breaks (aka Microsoft Excel), most of the time you just need to reset Excel. It's amazing. You don't need a new server, you don't need support contracts, it's just like riding a bike.

In my opinion, the limitation of Excel is actually when it comes to big-data analytics. The way Excel handles large quantities of data is slow (due to the nature of it's software structure). That's where we come in and build out these models and systems. However, in the end, our data outputs will be fed back into Excel, because that's what most people are used to.

I have deep respect for Excel. After all, Excel empowers so many users who don't know how to code to provide amazing plots and perform major calculations with ease.

halfeatenpie··on Principles and Techniques of Data Science
I'd assume because you have the --no-install-recommends flag on your apt-get call. Maybe something you're doing requires the recommended (but not dependent) packages. I haven't done it yet, but that's my assumption at first glance, so take it with a grain of salt.
halfeatenpie··on American Employers Are Hung Up on Hiring PhDs
I think the discussions here are missing a critical point. Just like those who mention that their colleague or friend who only has a BS degree or only graduated from high school are able to run circles around those with PhDs, there's a vary level of PhDs as well.

PhD means you've had and accomplished that formalized level of training. There are those who formalized training works very well for, there are also those who it's just not a good fit. That's fine. Just like there's a varying standard and quality of people around in this world, there's a varying standard and quality of people in PhD programs. Just by having a PhD degree simply states that you've gone through the program and learn how to do research. There's a major difference between how academia approaches a problem and how the industry approaches a problem, but what I've understood is that PhDs (in engineering) know how to expand and innovate in a much easier and more structured way than those in industry.

When generalized, most research I find in the industry is mostly "look I changed this thing and it's improved accuracy this bit more in my use scenario!" It's a bit less common to find papers or research done (Engineering-wise) in the industry that fully understands the benefits and limitations of the method improved upon, instead it's mostly "see, here I did it and now try it on your own and see if it works for you!".

All tools and methods can be improved upon or all models are inherently incorrect. I want to know the proper "surface area" that the method can be applied to and what the proper parameters are. Too many times (especially with some international papers) I've seen methods applied to problems that had to be "shimmied in" to make it work. It's like using pastebin.com as a CDN to store your ASCII data of a 3D model for your game (seriously, I've seen this happen...). I mean, it's not the right tool, that's not how it's supposed to be used, but I guess ok?

Source: I'm getting my PhD in engineering in Massachusetts. I've also co-founded a startup and am currently going through another similar experience right now with a "project" that we'll also be transitioning into a product. PhD is something I wanted after seeing what's in the industry.

halfeatenpie··on We Pay You to Learn to Code
I just realized I totally misread it. Now it makes sense.

I just can't do math apparently.

halfeatenpie··on Show HN: Baxx – Unix-friendly backup service
Yeah as an individual who doesn't use the euro, I didn't know it was euros until I read it in the thread here.
halfeatenpie··on We Pay You to Learn to Code
That's true but they wrote a maximum of $30,000 total.

Is that total the monthly max cap or total net "payment" back to the organization? If it's 2000 a month salary for 21 weeks then yeah that's fine, but I guess it's the semantics that I failed to understand.

halfeatenpie··on We Pay You to Learn to Code
I don't seem to be understanding the finances behind it.

The agreement is by completing the program, you pay 15% of your salary for 2 years as long as you're receiving more than $40,000. However, it also says it's capped at $30,000 total over that period.

If you pay someone a $2,000 a month salary for 21 weeks, that's already $42,000. Where's the extra $12,000 coming from?

I'm just trying to understand since if I agree to quit my "current job" and rely on another company to help pay for me, I'm relying on that company to not pull the rug out from under me when I'm half-way through the program. How do I know this is a reliable and sustainable program?

Interesting concept! That's just the first question that comes to mind for me.

halfeatenpie··on The Comdyna GP-6 analogue computer that I use daily at work
From what I've gathered and read from OP's comments, it sounds like their analog model makes sense and is precise enough for them to work with. People talk about how much more powerful a "32-core Xeon machine" is and they're not wrong, but just because new technology is available you also have to look at is it worth it to transition over/update it.

I have an example from my daily work. We were hired to update a company's model from 6 years ago. Their model is an Excel VBA model that we're rebuilding in Python with an increase resolution of detail and accuracy. However, for our "more accurate" model to be accepted, we have to also prove the following:

- We have to show that our model's outputs are the same as the VBA's model for the same input/starting condition (makes sense at a higher level, but at a higher resolution this means you have to spend more time looking at why a specific value is in a specific way and can be different from the lower resolution model)

- Train the new users on the new model (knowledge transfer)

- Ability to update and modify the new model wherever they need to change it or understand it

- Understand the limitations and the constraints of the new model

This basically boils down to how much time is needed in training the new model, how much money and time is needed to invest in building a new model, and is it really worth it to build a new model when the old model has worked good enough for now (and you understand the limitations of said model and is within your needs)? There's only so much time in the day and you obviously have a ton of other tasks that also needs your attention, and getting these newer models verified and accepted by the board is great but it's also additional meetings and time that you need to setup with them and get everything OK-ed... doubly hard when everyone else is also super busy.

I'd say modern computational platforms are better when it comes to raw power, but the logic and just resources required to build, verify, accept, and train (human resources-wise) the new model might just not be fully worth it.

Remember, there are also other industries other than tech that uses computer hardware to get jobs done.

halfeatenpie··on Predict the future with Machine Learning
I'm a PhD candidate related to statistical forecasting and projection related to natural science (hydrology/streamflow projection under climate change uncertainty).

The article seems to be a decent introduction article that shows what Machine Learning is about (which is great) and shows how it can POSSIBLY be applied to forecasting and prediction. However, I think it would be even better if there was a simple example or two with each method being applied and showing different outcomes and then the significance of each methodology through those examples.

Also, I'd like to add a comment to this.

This article is great when you look at it from the Machine Learning perspective. However, when you look at it from a forecasting perspective it only shows a very small portion of what forecasting/predicting really is.

Algorithms you develop through machine learning is something known as a black-box model. You know that the input data and the output data you're matching up with are related somehow, but don't know exactly how they're matched up. That relationship is established based on a performance index determined from a trial-and-error method (of course depending on what actual method you use).

There are different methods available such as ARMA and ARIMA based models. In regards to physical science, there are models that focus on the physical interaction between the input data to simulate what is happening inside the system. ML methods are simply just a taste of other methods available.

Regards to programming use (as I'm sure most of you folks here are used to), ML is a good tool to use for forecasting if you're really interested in it. But just like any forecasting model you use, you should probably determine the performance of your model based on not one index but multiple indices which consider different parts of your "needs". Percent accuracy only shows how accurate you are, you should probably also consider how frequently you're over-estimating vs under-estimating, how many series of overestimation there is, etc. The most important one though in my books is bias correction. When working with ML Algorithms most of them do not consider for bias. So you, as the modeller need to prepare for bias correction. However this article kinda glossed over it by saying "right amount of data" and "combination of data" (which I understand is an introductory post, but I think this is very important).

Maybe look into applying a method like the K-Fold Cross Validation to make sure the final output parameters aren't AS biased. It really depends on the modeller and the model and your performance indices you use.

halfeatenpie··on Ask HN: Do you charge your webdesign clients for ongoing hosting and support?
A fairly sizable percentage of our product line is web hosting. Of course to note we started out as a hosting company and people focused on infrastructure.

I'd say hosting is a good way to retain clients and to continue to work with them. It's great for clients as well since we do host their website and if problems happen then we can help them fix it. The hosting fees are actually packaged "for free" to our maintenance fees which includes very small changes (text changes and minor updating). Major changes would be considered a new project.

Regarding time investment, our maintenance is almost none as most of our infrastructure code has been developed and put on autopilot including monitoring. There are minor blips we do have to worry about and handle, but overall everything works. We also have a person on-premise (inside the datacenter) who helps us with quick turnaround to hardware or physical problems.

Biggest time investment is working with the client directly on their projects.

halfeatenpie··on What every PhD should know before interviewing with a tech startup
That is true. Like in any project, research, or work, the devil is in the details. I personally think everyone could have been more understanding of the situation and clarified specifics. However, in the end no one won. The talent has left and the company still needs to fill a position with a capable candidate.
halfeatenpie··on What every PhD should know before interviewing with a tech startup
Seriously this. Biggest thing I learned when I started my PhD was that communication skill is key. The interviewers in the blog post weren't looking for cutting-edge best theoretical approach, they were just interested in his process of thinking.
halfeatenpie··on What every PhD should know before interviewing with a tech startup
Ignoring the tone of the piece, the article does make valid criticisms regarding those entering the workforce with higher education. Most people usually get these experience and lessons from working in the workforce, but those in higher education sacrifices that gain further professional education.

I think the interviewee did make a mistake during his interview, in that he didn't read the room. They weren't looking for anything "cutting edge" and they know that they can't understand the in-depth and complicated work you did for your thesis. So what the interviewee should have done was focus on simpler and easier methods to start off with and kept it general. They already know you're smart from your degrees and from your CV, the focus of that segment of their interview was to see if the candidate had the communication skills available to express their ideas.

We have the same meetings with our stakeholders and during our project meetings. I don't explain the complex methods and algorithms used in-depth. I just keep it general and tell them how everything fits together (if I have performance indices then I also add that in to show accuracy). If the stakeholders want more detailed information like specifically how it breaks down and why I used X method instead of Y, then I can answer their questions and (if they showed continued interest) be willing to put together a report for them detailing the literature reviews and information related to that topic.

20 minutes isn't a reliable indicator of how someone can do their job. However, as the interviewee or the person in the "hot seat", it is your job to know how to sell yourself and to use your time effectively to maximize your chances of getting an offer. The decision makers (or in this case, the interviewers) has the hard task of figuring out which candidate is right for them. Your job is to make that easier. Once you get the offer, then you get to decide if you want to accept it or not.

And then after you get the offer, you reflect on your experience with that company so far. If the interviewers seemed to be focusing on different things than you're interested in or if it seems like they're going to be miserable to work with, then you politely decline the offer and move on to the next thing. If it seems like they have a condescending tone towards you and your work, then you can always just say "thank you for your offer but I'll be declining".

halfeatenpie··on What every PhD should know before interviewing with a tech startup
While I agree with the base message in the post (speed is important, keep it simple, discuss before expanding, etc.), I find the post itself made assumptions about PhD that can be clarified upon.

I'm an engineering Ph.D. candidate focused on optimization and forecasting (tldr: large amounts of data analytics). I also code extensively in R. As a Ph.D. candidate, you're supposed to be the expert with the cutting-edge knowledge in your field. You're supposed to know what options are currently in development, the pros and cons of each methodology, and the significance of the final output and how it fits in to the big picture.

Most PhDs focuses on developing and implementing best methods (theoretically) and continuously improving accuracy of the final results. Methods that haven't matured enough to be implemented today, but will tomorrow. Timing and speed is still important in academia, but (and this is my opinion) more focus is spent on implementing the theoretical approach correctly.

From my observations, industry environment focuses on implementation speed (not the theoretical approach development but rather just "coding it in") and making it work to "solve" the problem. Focus is spent on implementation speed over maximizing accuracy. Therefore, simpler and less computationally intensive (and in most cases, easier) methods are preferred over more complicated methods.

I guess to sum up my point, academia focuses on cutting edge technology and continual development of some of the best methods available. That's how you get more of your research published and further your academic career. Private industry would love to implement cutting edge technology, but focuses more on implementation speed which can sometimes impact model accuracy. From my understanding of the article, it seems like the candidate they interviewed wanted to show the extent of his knowledge whereas the interviewers were looking for the general "overview" and discussion approach and thought process (his mistake probably for not reading the situation right, but just what I gathered).

Oh also to add as a P.S., most PhDs spend time focusing on theoretical problems and methods. Which usually means that knowledge regarding full stacks, infrastructure design, supplemental technology (e.g. D3), etc. can be a bit lacking at first compared to someone who spent the equivalent 4/5 years working in the industry and who has gathered experience using those technologies. Also PhD is really a "solo" mountain/task to conquer. While you do have colleagues and lab mates with you, everyone is usually doing their own research. So unless you spend time outside of your "work time" contributing to team coding projects (which is highly improbable as you'll spend most of your waking time working on your thesis), you probably won't get too many people with coding collaborative experience.

Overall, the post is pretty standard to what everyone needs. I'd chalk this up to inexperience in the workforce.

halfeatenpie··on Cloud Firewalls
Vultr has the cheaper side down I will admit, but their network hasn't been to great for me. It's great for development work but for production-level stuff I wouldn't put anything on it.

Linode is like that old king on the block. They've had security issues in the past (multiple) and since they do store your credit card information those did get released (If I recall). They're decent and they work. They're a bit slower than the other two in my opinion, but they work and that's something I need.

DigitalOcean has been fairly solid and reliable. Yeah you can say you get more resources for how much you spend elsewhere, but DigitalOcean has been more reliable and solid than Vultr. I mean I remember opening support tickets with them and their response being "We took care of someone else on that node. Go ahead." Took me a solid 2 hours just to install a Ubuntu image on one of their (Vultr's) storage nodes. DigitalOcean has never had to give those kinds of responses to me and overall performance and the composition of their nodes has been fairly solid and reliable.

Yeah Vultr and Linode are cheaper and you get more, but I really feel DigitalOcean is solid and more reliable than Vultr. I mean Vultr's SLA is 100% uptime, but their credit return policy is to the effects of "we'll just give you your cheap money back". I don't care about credits and I just want my service online and not having to worry about anything, and most of Vultr's answers has usually been "wait" or "we did it" (but no real long-term solution to the problem). DO has been focusing on providing long-term solutions to problems I've had and no "bs" excuses. Linode has been solid as well, but don't have many of the features DO is starting to roll out with (which fair enough, it's their decision). DO has nothing but praise from me.

halfeatenpie··on Venture investors blast US decision to withdraw from Paris Climate Accord
Climate scientist here.

I'd love to see more companies and private businesses help solve the climate problems.

However (and this is all just my experience and opinion so YMMV), on an economic level private corporations and businesses are focused on maximizing profit and minimizing costs, whereas public and government projects focus on maximizing benefit and minimizing costs. Not saying corporations are unethical but there could be times when solutions and research interests could conflict with profits or income generation. A public research institute is supposed to provide a "disconnect" between the need for income generation and research so that your results are really focusing on maximizing public benefit.

halfeatenpie··on Show HN: VPS Comparison – Automated tests to compare VPS by yourself
Oh wow. That's something new.

We currently have multiple dedicated servers out of OVH BHS Canada (bare metal servers) and I have never ran into that problem. Most of the time I believe OVH gives you all the tools necessary. If I recall correctly, I believe their power-cycle feature is sending a SNMP command to the UPS, so their online control panel should technically work, but mileage may vary.

We've started moving on to Colocated hardware though. Unless you specifically need OVH's VAC system or their network (or free 256 IPs with setup fees), I've found OVH's availability of hardware and configurations to be fairly lacking. Glad it works out for you guys though!

halfeatenpie··on Show HN: VPS Comparison – Automated tests to compare VPS by yourself
My experiences with OVH's cloud platform has been a bit dated. I've been using their cloud platform since 2012, especially when their OpenVZ "Cloud" VMs were nested VMs inside their VMWare containers and had regular network outages (due to noisy neighbors) or restarting the VM every few hours. I switched to their VMWare VMs and they work like a charm. Granted some see it as a bit pricey for a small project but it's still cheaper than some other options.

My experience with OVH's KVM Openstack line was when it was still maturing as a RunAbove product. I remember they had problems out the gate and at one point had regular service restarts and I think at one point had a storage problem becoming read-only. I stopped using them since then and just chalked it up as another problem with using OVH's cheapest cloud platform. They've never worked out for me.

However, for those who stuck around and also bought KVMs from OVH after they "graduated" from RunAbove, I believe it works to an extent. It's not the best, it's obviously focused on minimizing costs and maximizing possible resources and reliability, and (this is my own speculation now) I think their server density is fairly high (VMs per host node). I haven't heard of any major problems lately with their KVM platform but again, I wouldn't put anything intensive on there either.

If cost is a major factor for you, then go with OVH's Openstack KVM line. If you still need fairly solid reliability, I think going with AWS or GCP might be worth another look. If you're fine with a VPS starting out without Ceph, NVMe, etc, then I'd really suggest you hit up Linode. Linode offers some of the best resources per cost ratio while still providing a decent reliability factor. If you really do need reliable storage then I'd look at DigitalOcean's Block-Storage (also known as network storage). Once you've grown your operation a bit more and have more resources and funds, then I think it'd be worth re-visiting the idea of moving to AWS or GCP or any other cloud platforms.

YMMV as I am just one person.

If you want a more detailed response feel free to let me know.

Best of luck!

halfeatenpie··on Show HN: VPS Comparison – Automated tests to compare VPS by yourself
That's a very true statement. There are things outside our control (and sometimes in it) that can result in downtime. Like a datacenter downtime event in 2013 where a road construction crew were performing maintenance and ended up cutting their fiber (the location fiber actually was didn't line up with where it was supposed to be on paper). Since then they've built another fiber connection out of the building.

In the end, what matters is how they respond to these kinds of events and issues and what actions they take to prevent those issues in the future. While I can't say for certain what issues were avoided through proactive measures, I will state that Linode has been great with setting up fixes that hopefully prevents problems in the future. DigitalOcean recently had a problem twice with their billing and control panel system going down (in addition to their API being down). They've also taken action to prevent this in the future and have been fairly open about it (one was their SFO datacenter being down if I recall correctly).

My only gripe about Vultr is that their support responses are very basic. Limited to "try again now" and "we've turned off a neighbor". Honestly I was hoping a solution that would work long-term and communication on how they're working to ensure that. DO and Linode focus on relaying that information and I feel really works with me. This is why I'm really fine with buying through them even if they're technically "twice" as expensive as Vultr. That peace of mind is really worth it.

halfeatenpie··on Show HN: VPS Comparison – Automated tests to compare VPS by yourself
Haha yeah...

Singapore is a big problem. The network there is so fragmented. Last I checked, Singtel owns a large portion of Singapore's networks but won't peer with certain ISPs or anyone peering with certain ISPs. This meant a Singapore user trying to communicate with a Singapore server sometimes needs to be routed through Hong Kong or another exchange before returning back to Singapore.

Don't forget SoftLayer's network in Asia. It's a real hit or miss on residential ISPs. One time a client of ours in Seoul on SK was being routed through Japan to hit SoftLayer Hong Kong. What is usually 40ms latency from Seoul to Hong Kong turned into 90ms of latency with a higher chance of packet loss during peak hours (for those who aren't familiar with Asia network, all countries are connected via submarine cables which really makes it expensive to expand bandwidth between countries. During peak internet usage you really can sometimes experience the packet loss unless you pay more. Expect higher operation costs in Asia especially since many central IX countries are currently experiencing power shortages).

halfeatenpie··on Show HN: VPS Comparison – Automated tests to compare VPS by yourself
Great start to a comparison review.

I've used all the service providers available and these opinions include my historical experience with them (I've used all providers for many years, my account with DO is since 2013). Most of my service uses were focused in those service provider's North America locations, Asia locations, and Oceania locations. Note I did do a short comparison between DigitalOcean and Vultr back in 2014 here: https://vpsboard.com/threads/digitalocean-vs-vultr-the-asian...

Most service providers here are fairly decent.

From my experience, OVH's KVM VPSes weren't worth the time, but their VMWare VMs were absolutely top notch.

Linode has had two major security events over the last few years, one of which was in early 2016 which included release of credit card information. However, they also offer paypal payments now and are fairly top notch in terms of performance. (They've taken approaches to fix their problems from 2016 with this blog post here: https://blog.linode.com/2016/02/19/security-investigation-re... ).

DigitalOcean is fairly reliable. Their Singapore location early on had regular packet loss events, but they've been shaping up pretty nicely over time.

Vultr has been a major problem for me. Vultr for a while (on their storage instances in Japan) had terrible I/O which took a solid two hours just to install Ubuntu 14.04. When contacted support, they stated they "stopped a noisy neighbor and to retry" but no real improvements were seen.

We had a service running on a VC2 that irregularly required a large amount of CPU power (higher than 0.5 load on a 1 CPU instance for a period of 1 hour a night). The same task was completed faster with VMs from DigitalOcean and Linode. But the biggest problem I've experienced is their network.

For the longest time their Japan location was single-homed and routing was funky. We'd regularly see any traffic going outside of Japan being routed back through their Los Angeles location, so adding 300+ ms latency for any of our users in Korea, China, and anywhere in South-East Asia. Today it's much better with 40 ms latency from their Japan location to our Korea client, and a solid 80 ms latency to our Singapore client.

A colleague of ours had very important clients who required 100% uptime (which Vultr agreed to) but over a period of a month had regular packet loss events or network loss (regularly 40 minutes per event and around one event every few days monitored via external monitoring systems such as uptimerobot, statuscake, and a GCP server running Icinga2). Didn't receive much help from Vultr's engineers nor could they find the issue. They ended up moving to Google Cloud Platform. They did end up getting SLA credit from them but no reason to have SLA credit when you're not on their platform anymore.

I regularly return to Vultr every few months to every year or so to see how it's going and they've come a long way with the services they're offering. I'd recommend them for a dev environment and testing over a period of an hour a day or so. However, I would caution with production systems. Your mileage may vary, but I'd suggest reviewing the quality of the product in addition to the price tag per resource.

halfeatenpie··on Show HN: VPS Comparison – Automated tests to compare VPS by yourself
Last I recall Linode and DigitalOcean were VPSes. They're on a single node with local storage. You can use block storage to set up network storage, but they're all VPSes.
← PreviousPage 2 of 2