substack

Not for transferring large files. A centralized architecture is the most expensive, slow way to do that.

You can sort of do these operations with couchdb, but you've got to fight against some bad assumptions for this use case of offline edits that could span weeks or months. From https://wiki.apache.org/couchdb/Replication_and_conflicts:

    With CouchDB, you can sometimes get hold of old revisions of a document.
    For example, if you fetch /db/bob?rev=v2b&revs_info=true you'll get a list
    of the previous revision ids which ended up with revision v2b. Doing the
    same for v2a you can find their common ancestor revision. However if the
    database has been compacted, the content of that document revision will
    have been lost. revs_info will still show that v1 was an ancestor, but
    report it as "missing".

That is a very dangerous feature for us. Also, the conflict avoidance algorithm means that users can't work on branches in parallel like in git because couchdb obstructs that workflow with 409 responses. This is a very bad feature when you're collecting data in remote areas and replicating with other databases and don't have the time, battery life, or the expert skills to resolve a "conflict". A database should never have states where it rejects new information.

The focus is offline, but the underlying techniques work just as well across the public internet. For example, you could use https://www.npmjs.com/package/webrtc-swarm to sync the hyperlogs:

    var wswarm = require('webrtc-swarm')
    var signalhub = require('signalhub')
    var swarm = wswarm(signalhub('p2p-map', ['https://signalhub.mafintosh.com'])

    var osm = require('osm-p2p')()
    swarm.on('peer', function (peer, id) {
      peer.pipe(osm.log.replicate()).pipe(peer)
    })

This salary-focused data collection (aside from issues with self-selecting mentioned elsewhere here) is missing a huge number of freelancers who have much more variable income from contracts and other kinds of unpredictable work. Most discussions of wages in tech ignore this rather large cohort that is difficult to count and often forgotten.

    > Imagine being able to download just one build tool that by 
    > default allows you to write something like
    > ‘jsc /some/js/dir -o /some/build/dir –enable-jsx’

    $ npm install --save browserify babelify babel-preset-es2015 babel-preset-react
    $ browserify -t [ babelify --presets [ es2015 react ] ] \
      main.js -o public/bundle.js

To incrementally recompile when a file changes: `npm install --save watchify` and replace `browserify` with `watchify` above.

There are sets of tools that compose well and are easy to set up. I think developers hear about all these new techniques and think they need all of them, but it's much more important to know what you don't need and how to incrementally add pieces in small steps only when you need them and not before.

Webpack doesn't have a monopoly on hot reloading, you can do this in browserify too: `npm install browserify-hmr`

With watchify:

  watchify -p browserify-hmr -o bundle.js

With budo:

  budo main.js -p browserify-hmr

If you're using react, you'll only need a tiny wrapper around the root element to opt in to hot reloading: https://github.com/substack/react-starter-hmr/blob/master/ma...

All of this should fit comfortably in a Makefile (or npm run script) and each piece you add is small and comprehensible.

I think that's preferable to remembering whether any given method is in List::Moreutils, List::Util, List::AllUtils or some other package entirely. You already know the name of the method you want, why can't that be the name of the package?

I'm not sure which xhr module you're referring to but the `xhr` package doesn't have this problem:

    $ npm install xhr
    $ browserify -r xhr | wc -c
    11147
    $ browserify -r xhr | wc -l
    390
    $ browserify -r xhr | grep Buffer
    $

There are extensions to distributed protocols like bittorrent that are already deployed to address mutable, non-static content. The approaches I know of address content under the hash of the public key. One of these approaches is http://bittorrent.org/beps/bep_0044.html and ipfs supports this technique too.

If you have a single mutable pointer, you can build a feed of data that points at immutable content by its hash, which could replace the data model of twitter, facebook, or many other social networking web services. The benefits to decentralized distribution are huge: native offline functionality, trivially transferable identity, longevity and robustness against providers shutting down, direct commerce without middlemen.

Payments, or perhaps ISP-style peering arrangements may help with the spam/large binary problem. A big part of distributing the data model will also involve distributing the costs, but this is somewhere non-profits like the Internet Archive can play a very important role.

I'm a big fan of virtual-dom and the ecosystem of very sharp single-purpose libraries that people have built up around it, usually not even coupled to virtual-dom at all.

For me, virtual-dom + main-loop is a pretty great combo for front+back universal javascript. I made some repos about setting up virtual-dom, main-loop, routing, and browserify/watchify with npm run scripts:

* https://github.com/substack/virtual-dom-starter - bare-bones main-loop + virtual-dom setup * https://github.com/substack/virtual-dom-universal-starter - universal (front+back)end js example with shared routing and single-page pushState

I usually use an ordinary EventEmitter to rig up the unidirectional flow (but I dislike how "flux" makes this a big deal). Here's an example of that: https://github.com/substack/virtual-dom-unidirectional-examp...

If jsx is your thing, I made an example of setting up jsx in virtual-dom land too: https://github.com/substack/virtual-dom-starter-babel-es6-js...

As far as the research questions go for 1-3, I don't think sampling people about their subjective opinions will lead to any interesting findings, but for question #4:

I chose virtual-dom and main-loop because each piece has a singular concern and performs that concern well. I don't want to use a tool that does a bunch of unrelated things because I want to be confident that in the future I can discard pieces of my tooling at will as my requirements change. With bigger tools with larger scope that is not so easy.

You can configure different registries on a global and per-scope basis:

https://docs.npmjs.com/misc/scope#associating-a-scope-with-a...

Some of the other commercial npm hosts and open source offerings already support scoped packages.

Charging per-module like github would encourage people to author fewer packages and to lump more functionality into the same package, which goes completely against the ethos of npm and the spirit of tiny abstractions that do one thing well.

The way that npm has structured things, programmers pay once for membership in a commercial tier, where presumably money is already changing hands to work on private code. I think this makes much more sense and won't bias the code itself in a negative direction.

If you're dealing with vanilla node.js streams, you can just do:

   one.pipe(two)

Here's an example of that kind of transient information sharing working quite well over a long period of time:

https://oaklandwiki.org/Why_are_there_helicopters_right_now%...

The proposal for forward secrecy in the spec (https://darkmail.info/downloads/dark-internet-mail-environme...) is not great and does not reflect the current state of the art:

> PFS for message objects, as the description above suggests, is far more difficult, and contrary to the nature of email.

It recommends just rotating public keys every few days with a paranoid mode. A much better solution is to implement the Axolotl Ratchet pioneered by Open Whisper Systems for TechSecure:

https://www.whispersystems.org/blog/advanced-ratcheting/

https://github.com/trevp/axolotl/wiki

With Axolotl Ratchet, you get:

* forward secrecy - disclosure of private keys doesn't let an attacker in the future go back and read encrypted communication

* future secrecy - disclosure of ephemeral private keys doesn't disclose (much) future content

I also get the feeling from a glance reading the spec that way too much trust is being placed on service providers. That there is even such a category as "trustful" where the server has access to your private keys is a huge red flag and that was exactly the problem with lavabit: http://www.thoughtcrime.org/blog/lavabit-critique/

For node at least there's:

https://twitter.com/nodesecurity

http://nodesecurity.io/

> If we really want to move into The Future, we need to start > versioning individual modules or functions, instead of > whole suites of software. But the cognitive burden of doing > this is very high, and the software to do it hasn't been > written yet.

I don't think this problem is related to any cognitive burden, but is an artifact of how the package manager works and how easy it is to liberally publish packages. Consider npm, where people publish packages with little to no coordination and everything is mostly fine (whatever the author of the parent article says). Because it's so easy to make an npm account (anybody can `npm adduser` and start publishing immediately) and so easy to publish (just pick an unused name, make a package.json, and `npm publish`), people routinely make very granular packages, often just a single function. Importantly, these functions don't need to fit into a taxonomy namespace like in haskell which prevents namespace hierarchy turf wars for the lesser price of namespace landgrabs.

Better still, you can have multiple different versions of a library in the same application. In your example, libB would just get the version of libA that its constraints satisfy but elsewhere other libraries could depend on a version of libA that do not satisfy the constraints on libA that libB requires because every package gets their own copy of their dependencies, with automatic deduping when the constraints are satisfied to save space.

No.

You probably meant:

    var tr = require('trumpet')();
    tr.createReadStream('article > span')
      .pipe(process.stdout);
    
    var request = require('request');
    request.get('http://www.echojs.com').pipe(tr);

Bonus: I just noticed a simple bug in the selector engine from running your intended code that I just fixed in trumpet@1.5.6.

I have hundreds of open source modules and I really hate this idea. It creates a monetary incentive for developers to keep adding more features even when they are bad ideas. Also, what happens when a paid issue/feature was a mistake and should really be removed later? The economic incentive does not align very well with creating the most appropriate set of features for the task at hand and you end up with bulky enterprise kitchen-sink software.

I hope that nothing will "win" because "front-end" programming is such a nebulous concept that it's much better for each project to find a niche and cater to an increasingly narrower set of trade-offs and use-cases. The idea that we need such a thing as a "framework" is I think mistaken but understandably so: we've only had viable module systems and package management for a short amount of time and they are still evolving.

But module systems and package managers change everything about how open source ecosystems work! When you've got a package manager that is trivially easy to publish to, you remove pressure on libraries upstream to include extra functionality into their cores because users have too hard of a time fetching the functionality for themselves otherwise. With pervasive package management and a culture modularity, modules can be smaller because they can depend on other modules instead of inlining functionality or otherwise being overly cautious about dependencies. Convergence under this kind of fitness landscape tends toward narrowly-focused single-purpose libraries that do just one thing and do it well.

Popular frameworks have already severely dwindled in size from something extjs, mootools, or yui scale towards something backbone-sized. I sincerely hope that frameworks dwindle away into nothingness so that focused, reusable libraries that take simple inputs and produce simple outputs can fill the gap because the severe orthogonality and opportunities for novel recombination are so much greater!

browserify is a javascript bundler that recursively analyzes all the node-style require() calls in your program to create a single javascript bundle for serving up to browsers.

This kind of static analysis is really easy to do with falafel:

https://gist.github.com/substack/4735826

Program:

    var falafel = require('falafel');
    var fs = require('fs');
    var src = fs.readFileSync(__dirname + '/src.js', 'utf8');
     
    var output = falafel(src, function (node) {
        if (node.type === 'IfStatement'
        && node.test.type === 'MemberExpression'
        && node.test.object.name === 'Meteor'
        && node.test.property.name === 'isServer') {
            node.update('');
        }
    });
    console.log(output);

Input:

    if (Meteor.isClient) {
        console.log("I'm the client!");
    }
    
    if (Meteor.isServer) {
        console.log("I'm the server!");
    }

Output:

    if (Meteor.isClient) {
        console.log("I'm the client!");
    }

This won't catch fancy things like Meteor['isServer'] or compound expressions but should be a good place to start.

edit: I had them backwards. Fixed now.

Click on any of these demos: http://voxeljs.com/#gallery

And another new voxeljs module for portals: https://github.com/substack/voxel-portal http://substack.net/projects/voxel-portal/

What a crazy coincidence, I just factored out my company's payment widget code into a reusable module a few days ago: https://github.com/substack/pricing-widget

Basic example: http://substack.net/projects/pricing-widget/basic/ Fancy example: http://substack.net/projects/pricing-widget/browserling/

The service registry I wrote (http://github.com/substack/seaport) is fundamentally version-aware to avoid this problem. You register services with a semver such as `web@1.2.3` and then clients or other services query for registered services with semver query strings such as `web@1.2.x`. This way you can specify your breaking changes much more explicitly and not need to port over the clients to the new service all at once.

     and it DOES work with everything

... so long as you wrap all the files in the particular variant of define() function that works with requirejs (but not with other competing AMD function signatures). And then you might want to use some modules written for commonjs or node with `exports`, so you'll need a tool to transform those files. And then you'll probably want to compile the application into a single javascript bundle in production for performance.

I'm not saying bower is any better in any of these respects, just that everything is terrible.

No. If anything, it will probably just get even more fragmented before the situation gets better. As we understand the features and architectures of package management systems better it becomes even easier to write package managers. This is a good thing long-term because we can more thoroughly explore the solution space before settling on a few "winners". However, at the same time package repositories are fundamentally a network effects problem because packages depend upon other packages, so the repository with the most packages offers the most benefit for each additional package because new packages can build on everything that already exists instead of bundling that functionality themselves.

The reason why I am skeptical of new entrants into the javascript package management arena is that there is already a package manager with over 14k packages and a vibrant community that is producing packages at a very high rate: npm. npm might not be the best package manager as-is for browser development, but tools like browserify (disclaimer: I wrote browserify), ender, or many others can let your browser code use much of the value that has already been created for npm, even though npm is primarily about node.js packages. A surprising number of modules written for node will just work in tools like browserify.

Any upstart package systems would be wise to have a clear and simple path for unlocking the value created in the node community because there is too much value being created in npm to ignore. Whether that path involves adding fields to the package.json, tools to bridge the gap between package repositories, fancy in-browser trickery or compile steps remains an open question but it would be really pointless to require people to publish their modules that they've already put on npm to buy in to yet another package system, especially since there seems to be a new package manager every month lately.

Actually that code snippet that computes a summation can be done without any loop or recursion at all:

    function sum (m, n) {
        return m * (m + 1) / 2 - n * (n - 1) / 2
    }

I only know this because I remember from math competitions in high school learning that you can compute the sum of the digits from 1 to n quickly from `n * (n + 1) / 2` which naturally follows from the equations regarding binomial sums.

This example somewhat agrees with the author's main point that we are more a product of our experiences than endowed with certain innate capabilities.