dm03514

I'm really excited about the state of data infrastructure and the emergence of the data lake. I feel like the technical aspects of data engineering is reduced to getting data into some cloud storage (s3) as parquet. Transforms are "solved" using ELT from the data lake, or streaming using kafka/spark.

I think executing this in orgs with legacy data technologies is hard but it is much more a people problem than a tech problem. In orgs that have achieved this foundation it's really cool to see the business and analytic impact to the company.

Ahh, TOO Many fees :(

I incorporated through Stripe Atlas for 2019 hoping to start and begin to develop the business, but dissolved the company after the first year because of these fees. Will try again when idea/product is more mature.

* version control

* static typing (golang)

* unit tests

I created an opensource tool to generate performance reports based on Github pull requests (think gitprime, codeclimate, gitclear).

I've made $250 in a single year on a single consultation.

The first iteration was a service that listened to build and repository actions. I switched it up to generate static reports. It queries github at a point in time for raw PR data. Then it generates a basic PR report based on that data. It generates CSV so visualization falls on the end user :p

- https://github.com/ImpactInsights/valuestream

- https://medium.com/valuestream-by-operational-analytics-inc/...

* Function "sprawl". Managing lots of small functions

* AWS Support Dependency. IMO way better to have amazon managing more but it still increases reliance on amazon and support tickets

* Limited set of input events available. For example it's difficult to use rabbitmq but lambda orchestrates batching and orchestration for SQS

I didn't really mention these because I think they are better than the alternatives :)

I would personally rather manage a bunch of small functions leveraging amazon's documentation, community, clear interfaces, and SDK, than a random OSS async worker framework or homegrown processing system.

Companies that have committed to AWS are already dependent on AWS support, I haven't really found an issue with being more dependent on AWS support :p

The limited set of events has also been a net positive. There are few clearly defined options. I have found the limited choices helps delivery and simplifies architecture; less discussion. Also once a company learns SQS and kinesis, that knowledge is applicable everywhere, since those are the constrained choices.

-----

I would love to hear any experiences where lambda was not a good fit, or did not work out

thank you, this is amazing

Thank you, Ha i will!

all i was considering is what I'm going to do with the $$ :)

good points, i was already massively leaning towards taking it 95% just wanted a quick sanity test from HN

That's one of the first things I thought about :) Not much friction for him, his customers will def follow him around, the only friction will be from the legal part of it and the rebranding ahh

Our library (Eldersburg branch in carroll county MD) is vibrant!

I work from it M-F to get out of the tiny apartment and away from the kids. The library runs "story times" a couple times a week for the kids. There is a small kid play area that we frequent during winter or rain when we want to kill an hour.

The coolest part is that there is a STEM center with robot, 3d printing, Legos & Julia/Scratch machines. I also work from the "teen" section (because it allows for talking and I have zoom meetings all day) which has: comics, xbox one, and study rooms available.

There are always people tutoring out of the rooms and kids hanging out gaming with switch/xbox fortnite. It's really cool.

When I was younger, I believed that my spirit and thoughts and personality were somehow separate from my body. Like something uniquely me that was put in my body to animate it.

As I get older I began to think the opposite; that our thoughts, personalities, and everything original about us are a product of our physical composition and brain structure.

I began to really notice this after reading Principles where Ray Dalio observes that the way people think and approach and solve problems are largely fixed. I notice this in myself and the people around me.

There was a sort of inflection point for me after getting my first massage. My body was completely relaxed, resulting in my mind being completely relaxed. After the massage I went to a trader joes in columbia maryland during a saturday. This trader joes is a logistic nightmare that always stresses me out, so much so that I would usually avoid going during peak shopping. But this time I was completely unbothered, nothing about the experience was bothering, my thoughts were completely at ease. At that moment I realized that it wasn't stress making me tight, it was tightness making me stressed. The dependency was completely backwards from the way I thought. Being self centered i thought my thoughts informing my physical, but it was actually my physical causing my thoughts.

I have genetically high blood pressure, and I have to consume low sodium in order to keep it under control. While I was learning this and my blood pressure was high, I was literally angry about everything. Road rage, meltdowns at home, being short and explosive with my wife and son. I thought it was stress or affect from work I thought it was thoughts/personality -> body. As soon as I experimented with reducing sodium, getting exercise and my blood pressure dropped, my mood completely rounded out and fixed itself. IT was actually my body -> thoughts.

The same thing happens with hiking spending time outside. I used to feel like a connected, or spiritual relationship with the outdoors. I enjoyed drinking and camping, I thought that there was a spiritual relationship with the outdoors (personality -> body) when, once again, it's actually just physical -> personality, dopamine is released, my blood pressure probably dropped my thoughts became more relaxed, my body was tired and relaxed from hiking.

I think that we really are a lot less conscious than we think we are, and that the way we think, our personalities, and pretty much everything about us is largely predetermined by our physical composition. I think we're just more complicated animals and our personalities are just a response to physical input and reactions.

Greetings! My name is Danny and I'm the creator of ValueStream. ValueStream's goal is to help organizations identify and remove process inefficiency in order to save time and money.

I would love to hear your impressions and/or feedback and am open to integration (event source) requests and contributions!

Thank you

https://github.com/ImpactInsights/valuestream

Author here, would love to hear your thoughts and feedback. Are there any specific features that you'd like to see? Would you consider using a tool like this? What features would be required for you to do so?

I appreciate your time and Thank you

Danny

- https://medium.com/@dm03514/valuestream-devops-metrics-obser...

- https://medium.com/dm03514-tech-blog/debugging-devops-using-...

Greetings! Author here, I was a bit frustrated at the state of DevOps metrics and how difficult and time intensive it is to capture basic delivery metrics across systems. I created ValueStream in order to help address this.

https://github.com/ImpactInsights/valuestream

It is able to connect and capture metrics from any system that emits events through webhooks. Would love to hear your thoughts and/or feature requests.

How does this compare to telegraf?

https://github.com/influxdata/telegraf

Biggest thing that pops out to me is LUA engine (seems amazing :) )

Hi, Author here, I would very much love to hear your experiences capturing these "core" DevOps metrics. Are you able to do it successfully? What approach do you take?

The examples are interesting in that why is it the Shopping carts responsibility to report the results of the discountService?? If the metrics have to be surfaced, someone, somewhere needs to surface them: and I've been finding it achieves the cleanliness and doesn't break encapsulation when the service (discountService) is responsible for its own hooks, or monitoring adapter. Like mlthoughts2018 i've been finding success using decorators for this because it keeps the domain logic clean of metric/monitoring code:

(shameless plug i recently wrote about how decorators can be used to keep the domain metric/plumbing free)

https://medium.com/dm03514-tech-blog/designpatterns-consider...

(also just disclaimer: Team's i've been on have had lots of success using the approach outlined in the article, injecting a metrics/instrumentation object, it cleans up the domain logic, is easy to provide a stub implementation during tests, and abstracts the from metric implementations.

------

A decorator version might compose the production code by wrapping the domain logic in metric adapters:

  metrics = InstantiateMetricsClient()

  ObservableShoppingCart(
    ShoppingCart(
       ObservableDiscountService(
         DiscountService(

         ),
         metrics=metrics,
       )
    )
    metrics=metrics,
  )

----

Another option that I've seen be used successfully is to completely decouple domain events and the surfacing of domain events by having the domain generically emit events, and the logging/metrics subscribe and surface those however it decides.

----

I feel like for normal logging statements and metrics it only borders on being an issue, but when metrics include many timing wrappers, or when tracing (which involves much more reporting) is involved explicit adapters help to really keep things clean.

Principles by Ray Dalio, completely changed the way I think.

Currently in the middle of thinking fast and slow, and I think it may be happening again :p

I can imagine a future where what's true generally describes itself (like terraform on drugs for software :p) Imagine software that is fully self descriptive and would no longer require engineers to individually interpret what's happening because the software would tell us. The system would be a graph of every single component and all possible connections between them, and all variants of each component and state that it could be in. When we introduce a change we would be aware with perfect information about the affect to the states and the paths between them.

In the example the Mental Model was at a level too shallow, it should have only affected the paths between the autofocus and the user. But the bug necessitated a larger mental model (the author was considering too small subsection of the graph).

I'd hope in the future we could reach a state where the program could have detected that the frame refactor would have an affect on the autofocus and all other components instead of being an implementation detail.

This scene played out almost every night, except I learned to be quiet so no one would know I couldn’t open the door because of a mysterious feeling that if I didn’t do it “right” something horrible would happen.

I didn't quite realize it, but how many people suffering from OCD are motivated by "something horrible" happening? Like I wonder how many people have compulsions that aren't motivated by something bad happen?

OMG I too am a "chronic checker". When my first son was born and my OCD was aggravated by exhaustion, and living in a relatively unsafe part of baltimore, I would have these 10 "check loops". I just couldn't be 100% certain the door was locked. Even at the time some part of me realized the absurdity of it.

I feel like it has a lot to do with the ambiguity involved in debugging, and the fact that it is hard to teach.

I have found debugging to be fuzzy and largely based on experience, mental models, and internal (scientific method based?) approaches.

There seem to be a relatively few debugging methodologies or frameworks. (ie in performance/systems realm there are USE, RED, ...?). I have only started to recently create a formal framework (a google doc I update after each incident) after 9 years of engineering experience! Not because I wasn't interesting in debugging, but it literally took 9 years worth of debugging incidents for patterns to start to become apparent to me (i work devops/system so when there is a general feeling of a problem I'm usually tasked with finding root causes).

On a side note: so far the framework has been successful within my org and I hope to formalize it enough to write about it in the near future :)

End to end test framework nearing first stable release! Would love to hear feedback on feature requests, documentation, or general direction.

Thank you,

https://medium.com/dm03514-tech-blog/introducing-test-engine...

I'm personally in line with the OP where I don't have any fully formed side projects that I would feel very comfortable as a reflection of my current ability. I really like stackoverflow for this exact issue. Spending time with family is just more important than coding, but I'm able to easily/quickly contribute to stack overflow during work or in the off time. If someone is interested in seeing examples of how I think, help or code I can say, look at the 1000 answers on stackoverflow :p

I just bought a roomba:

https://www.amazon.com/gp/product/B008LX6OC6/ref=oh_aui_deta...

Highly recommend it to anyone with a large amount of hardwood floors. We have a couple area rugs, one is pretty thick and it handles the transitions fine.

Floors have never been cleaner, no longer need to vacuum animal fur everyday.

I've been massively recommending this book. I think it's very unlikely a backend engineer can avoid having their applications on multiple servers. This book does a wonderfully clear and practical description of the issues with distributed systems, and tools and techniques to address those issues. It's written in a way that is very accessible to someone without a traditional comp sci background.

I was excited about this book because there is a gap in distributed systems books. I feel like there's are a large amount of blogs but most of the books available on amazon are text books and/or include heavy math.

Author here.

This project is aimed at making end-to-end testing easier.

It's at a usable stage and was built out by implementing a couple of my employers functional tests.

I would really like to hear people's experiences with tools like these and any requests for features, or what would make this useful for your work.

Thank you!

SimpleReach | Senior/Lead Rails Engineer | NYC | ONSITE | $120-160k + equity

https://angel.co/simplereach/jobs/208333-senior-rails-engine...

I'm a backend engineer there for almost 3 months now. Tight knit team, tons of leadership opportunity, overall great place to work with lots of opportunity for an individual to guide the direction of the engineering! My interview process consisted of:

  - phone screening ~30 minutes
  - tech screening with CTO ~45 minutes
  - tech/culture screening with engineering manager ~ 60 minutes
  - onsite technical interview with Lead Engineer, Chief Architect,
 non-technical with VP of Engineering ~ 4 hours

i was in NYC for work during the holidays, etsy had a pop up exchange in Chelsea market and it was absolutely nuts. After seeing it, the need to create a food market to connect, and incentivize, local producers and consumers was obviously apparent.

I think garden tech is a wide open space with room for even small application of tech mentality to have a huge impact on yield