Hacker News Reader

Comment by hobbescotch | Hacker News Reader

I struggle with how to handle this at work. I am a data engineer and while writing new scripts to pull data, the testing process sometimes involves waiting 15 mins to an hour (or more) for all the data to pull. Some times I'll start a new task in the middle of the pull or, if I expect it'll only be a few minutes, I'll flip out my phone. But once the data is finished pulling, I'll then need to jump back it and validate it one last time.

I'm curious what others do during this sort of downtime?

View on HN

You need to shorten the feedback loop to become productive. Maybe you could optimize your environment to operate on smaller data sets during the development? Introduce sampling, local caching etc. Then once you feel confident about the code you run it over the final data-set.

I often find myself stuck staring at screen when some lengthy process occurs. What helps is adding progress indicator, ETA clock, some audible OS notification on finish. Or better take a walk and get a SMS once the process finishes :)

I read in a book, the idea is to break away from a similar kind of task and get back to it after break. If you are still at the computer doing other things while the code is being executed, you are in the same zone different task. But if you use the breaks to do a completely different activity you will be able to come back with a fresh mind and do the task well.

So read a paper book , take a walk, print research papers and read them when you are waiting,

The idea is to treat each of the tasks between waiting as individual task.

Edit: book mind of numbers

I usually try to have a single main task that requires my attention, and fill the gaps with menial tasks that don't require a lot of focus, clean up the code, improve documentation, add examples, spell check the wiki, stuff like that. If I run out of things I'll search for some article or tutorial about something related like a library or a tool I'm using and see if there's anything useful in there that I can apply to my code. If that's still not enough I'll consider starting something else or just browse HN for a while if I just need wait something like 10 or 15 minutes.

Why not automate the last step and send a message on success/failure (or automatically retry based on error type)?

https://xkcd.com/303/

Reply on news.ycombinator.com