bruth

My team and I had the exact use case.. here is what we came up with: https://github.com/chop-dbhi/diff-table#diff-table

Supports delimited files and Postgres tables. Outputs a large JSON-based summary of record diffs or it emits an event stream as changes are found (and thus does not hold the summary in memory).

Prebuilt binaries: https://github.com/chop-dbhi/diff-table/releases

Edit: add link to releases and description

Hi Scott. I did not listen to the podcast (yet), so sorry if you answered this already, but what, in your opinion, could be the greatest impact quantum computing has in society? I know its a broad question, but curious of a few bullet points.

The Children's Hospital of Philadelphia | Arcus Data Platform Engineer | Philadelphia PA | ONSITE Please visit: https://careers.chop.edu/job/Philadelphia-Arcus-AnalystProgr...

The Children’s Hospital of Philadelphia (CHOP) Research Institute is recruiting a new team to build a data and informatics program called “Arcus” that will link clinical and biological data and provide world-class computational tools to solve the most challenging problems in child health. Recognizing the central role of data to the future of pediatric research, CHOP leadership and the Board of Trustees committed to a funding plan, and Arcus was launched in July 2017. The Arcus team integrates with major scientific initiatives in the Research Institute Strategic Plan: Lifespan, Rare Diseases, Novel Devices and Therapeutics, and Precision Health. We seek mission-oriented professionals with interest and expertise in the areas of biomedical science, library science, data education, data science, cloud computing, data privacy, and security.

This role will work on a small team focused on architecting and implementing a cloud-native “data platform” to support the goals of Arcus. We are looking for highly creative people who share our mission to advance child health and who will thrive in a continuous learning environment, acquiring and applying both new technical skills and biomedical domain knowledge.

More info about CHOP and our work: http://www.chop.edu/ | https://dbhi.chop.edu/ | https://github.com/chop-dbhi

What workloads do you prefer pubsub over nats and vice versa?

The Children's Hospital of Philadelphia | Arcus Data Platform Engineer | Philadelphia PA | ONSITE Please visit: https://careers.chop.edu/job/Philadelphia-Arcus-AnalystProgr...

The Children’s Hospital of Philadelphia (CHOP) Research Institute is recruiting a new team to build a data and informatics program called “Arcus” that will link clinical and biological data and provide world-class computational tools to solve the most challenging problems in child health. Recognizing the central role of data to the future of pediatric research, CHOP leadership and the Board of Trustees committed to a funding plan, and Arcus was launched in July 2017. The Arcus team integrates with major scientific initiatives in the Research Institute Strategic Plan: Lifespan, Rare Diseases, Novel Devices and Therapeutics, and Precision Health. We seek mission-oriented professionals with interest and expertise in the areas of biomedical science, library science, data education, data science, cloud computing, data privacy, and security.

This role will work on a small team focused on architecting and implementing a cloud-native “data platform” to support the goals of Arcus. We are looking for highly creative people who share our mission to advance child health and who will thrive in a continuous learning environment, acquiring and applying both new technical skills and biomedical domain knowledge.

More info about CHOP and our work: http://www.chop.edu/ | https://dbhi.chop.edu/ | https://github.com/chop-dbhi

Edited: fixed link

Interesting so adding new fields even.. I suppose there is quite a bit of planning and iterating on the interface before making it public to reduce churn? I understand doing that for backwards incompatible changes.

Thanks for the response! These are very good references. I like the breakdown of message and/or service definitions into separate files so it is easy to browse through the repository. I am assuming the version bump (v1 vs. v2) occurs when you introducing breaking changes to the API? I am aware that protobuf does a good job at allowing forwards and backwards compatibility at the message level..

Slightly off-topic, but related.. I have read that a common practice for managing proto files (or any schema definitions really) is to put them in a separate repo/package to share. It seems pretty straightforward in my head and provides several advantages. However I still ask about any trade-offs when doing this in practice?

The Children's Hospital of Philadelphia | Arcus Data Platform Engineer | Philadelphia PA | ONSITE

Please visit: https://careers.chop.edu/job/Philadelphia-Arcus-AnalystProgr...

The Children’s Hospital of Philadelphia (CHOP) Research Institute is recruiting a new team to build a data and informatics program called “Arcus” that will link clinical and biological data and provide world-class computational tools to solve the most challenging problems in child health. Recognizing the central role of data to the future of pediatric research, CHOP leadership and the Board of Trustees committed to a funding plan, and Arcus was launched in July 2017. The Arcus team integrates with major scientific initiatives in the Research Institute Strategic Plan: Lifespan, Rare Diseases, Novel Devices and Therapeutics, and Precision Health. We seek mission-oriented professionals with interest and expertise in the areas of biomedical science, library science, data education, data science, cloud computing, data privacy, and security.

This role will work on a small team focused on architecting and implementing a cloud-native “data platform” to support the goals of Arcus. We are looking for highly creative people who share our mission to advance child health and who will thrive in a continuous learning environment, acquiring and applying both new technical skills and biomedical domain knowledge.

More info about CHOP and our work: http://www.chop.edu/ | https://dbhi.chop.edu/ | https://github.com/chop-dbhi

The Children's Hospital of Philadelphia | Arcus Data Platform Engineer | Philadelphia PA | ONSITE

The Children’s Hospital of Philadelphia (CHOP) Research Institute is recruiting a new team to build a data and informatics program called “Arcus” that will link clinical and biological data and provide world-class computational tools to solve the most challenging problems in child health. Recognizing the central role of data to the future of pediatric research, CHOP leadership and the Board of Trustees committed to a funding plan, and Arcus was launched in July 2017. The Arcus team integrates with major scientific initiatives in the Research Institute Strategic Plan: Lifespan, Rare Diseases, Novel Devices and Therapeutics, and Precision Health. We seek mission-oriented professionals with interest and expertise in the areas of biomedical science, library science, data education, data science, cloud computing, data privacy, and security.

This role will work on a small team focused on architecting and implementing a cloud-native “data platform” to support the goals of Arcus. We are looking for highly creative people who share our mission to advance child health and who will thrive in a continuous learning environment, acquiring and applying both new technical skills and biomedical domain knowledge.

Please email your CV to ruthb AT chop DOT edu

More info about CHOP and our work: http://www.chop.edu/ | https://dbhi.chop.edu/ | https://github.com/chop-dbhi

Great read. I agree that Kafka is overkill if even necessary for most applications. I have been using NATS Streaming [1] which has the same properties as Redis Streams for some time and it is wonderful.

https://github.com/nats-io/nats-streaming-server#nats-stream...

ML with human-in-the-loop (HITL). There is a feedback loop that learns based on human expectation. Peloton[1] is already learning based on data access patterns, but doesn't consider subjective feedback. Maybe there could be a query planner that learns which query plans are better via reinforcement learning (the reward is given by the human).

[1]: https://github.com/cmu-db/peloton

Should be noted it is from 2012 in the title.

AI has been a field since the 50s (https://en.wikipedia.org/wiki/Artificial_intelligence#Histor...). Methods have been developed over the years, but the primary blocker has been computing power and data sizes in order for them to be effective. That is the primary reason they appear "in vogue" now because we have access to cloud computing and large datasets (or synthetic ones can be generated).

The best thing you can do as a developer to determine if AI is relevant to you, is read about how AI is being applied (the use cases) and if they sound interesting, learn the methods. Eventually you will discover if and how they apply to your work.

I can't speak to blockchain but that is orthogonal to AI.

Nice work. You definitely should get in touch with the Dat Project folks. There are several of them on the core team and in the community who are actively scraping government websites for open data.

Wow! My friend and I had this idea years ago.. possibly pre-GitHub. I am glad someone finally made it! Well done. My particular interest was seeing how a recipe deviates from the original.. a recipe graph of sorts. That would a fun way to visually explore and find related recipes that are similar in ingredients.

Yep. Just don't do it. https://genomebiology.biomedcentral.com/articles/10.1186/s13...

I have roughly 400+ voice recordings that I have taken during my commute. As a few have stated, simply speaking out loud typically validates an idea or not simply due to the medium.

SE Daily podcast about the company: https://softwareengineeringdaily.com/2016/12/16/scale-api-wi...

Along those lines (and possibly what the OP was looking for as well), what are the conventions around key names including namespacing? Or what are the common environment variables you capture for a given service, i.e. the boilerplate stuff.

Nice work! This would be great if it could be pluggable for relational schemas. The interactivity is nice for explaining the schema with domain experts.

Shameless plug: https://github.com/chop-dbhi/scds (stands for slowly changing dimensions store). A prototype, but versions individual records and tells you what changed if anything. Also there is preliminary support for JSON schema.

Not a direct answer, but Max Ogden talked about that a bit on Changelog's RFC: https://changelog.com/rfc/6

I can recommend this book as well! It is easy to read and understand and the choice of Go as the host language is also great.

I am in the process writing a semi-natural query language (boolean logic) targeted at domain experts (in this case biomedical researchers) that compiles to either database queries (e.g. SQL) or a function call for arbitrary logic. The goal of the compiler bit is to abstract away where the data ultimately lives (similar to GraphQL) or how it is structured.

Andy Pavlo recently gave a talk about it at DataEngConf (https://www.youtube.com/watch?v=mzMnyYdO8jk).

> That is not the case. Binary data is also allowed as the keys of objects

What is the value of binary key? A key is just the name for a value, it should not contain any data itself.

> I think it would make more sense to place the type tags on the values, not the keys, both visually and semantically.

Tags on keys are like types for columns or any other schema. I would rather not have to pre-process the values. To be pedantic, this would require copying all string-based values just to add a prefix.

All of the keys in JSON must be strings, so they should not need tags for themselves. Instead why not put the tag of the value assigned to the key in the key:

    {
        "s:string":"Hello, world!",
        "b64:binary":"SGVsbG8sIHdvcmxk",
        "i:integer":42,
        "f:float":42.0,
        "t:timestamp":"2016-11-02T02:07:30Z"
    }

This prevents having to mess with the values in general and integers don't need to be encoded as strings.

EDIT:

I see this constraint:

   Member names in TJSON must be distinct. The use of the same member name more than once in the same object is an error.

which is still satisfied, however you could have `i:foo` and `s:foo` which would result in redundant keys in the resulting JSON document. This constraint could be clarified that, untagged key names must be unique.

Another question, is a mimetype planned for this? `application/tjson`?

This is a great start. I too got frustrated with the overwhelming complexity of the "boilerplate" or "starter kits" that have all the bells and whistles. Having something very simple that you can exit if necessary is nice. The approach I took was just to document each tool or plugin that I may want to add to my project since it is generally very quick to do so: https://github.com/bruth/modern-javascript-starter

Nice reflection. I enjoy reading about how open source projects grow. Other than the attraction of solving #monitoringsucks, were there other significant factors that has led to the success of the project?

For that matter do others have links to or stories for other projects?