Show HN: GitMap – Interactive map of top 10,000 repos clustered into ecosystems | Hacker News Reader

Show HN: GitMap – Interactive map of top 10,000 repos clustered into ecosystems

oracleofnj.github.io

79 pointsoracleofnj19 comments

This is really cool, and props for making it at least dynamic-ish by rendering JSON data with D3.

That said, some of the clustering (or at least the clustering visualization) seems strange, such as AngularJS within jQuery, PouchDB within React, EmberJS within Rails, redis-doc being included within Go, oh-my-zsh within homebrew, etc.

FWIW, there's also a bit more info available here: https://github.com/oracleofnj/gitmap

Thanks for the feedback! I have a few ideas for how the clustering could be improved at the higher levels. The re-clustering step I'm doing right now is pretty simplistic.

For some other clustering algorithms (albeit based on terms and semantics ie document clustering and not necessarily relationships) one might be interested in Carrot2[1]. Carrot is neat because you can change/tune the clustering algorithms on the fly something I think would make this gitmap more interesting.

It would be interesting to see document clustering applied to code (I have not seen it but it seems it might even be more reliable than affinity). That being said ... a lot of memory will be needed since most of the clustering algorithms implementations need to run in memory :)

[1]: http://project.carrot2.org/

Amazing to visually see React in such a short amount of time is nearly as large an ecosystem as Jquery?

Seems so difficult to believe, I question if I'm reading the data correctly!

The pace at which new things happen that you have to know is incredible.

I wouldn't take it too literally since as I mentioned in another comment, the higher-order clustering I'm doing is fairly simplistic. But interestingly enough, the initial reason I started working on this was to try and figure out what other ecosystems on GitHub had as active a community as React.

The interesting thing to do would be the things that go from nothing to many quickly and then learn them :-)

(hint, elixir)...

Yes, when you put all the scala/akka/play/spray/netty/typesafe/typelevel/gloovy/gradle/etc project inside the React cluster - I don't see the connection.

Well, React.js _was_ made by Facebook.

and not of the other mentioned technologies was made by Facebook, so I don't see the connection

By any chance could you post repos/stars data? (I guess downloading it would be much faster than crawling it.)

In any case - thank you a lot for sharing the source code! :)

I added a file "starcounts.json.gz" to the repository.

Thanks, but I meant more "cached_repos.json.gz" and "cached_users.json.gz". It took 2 nights of fetching and it is nowhere near there (~2100 crawled repos, and users).

This is cool.

TIL paragonie/awesome-appsec is in the top 10,000, but other projects like paragonie/random_compat aren't :(

It's kind of interesting that almost all of the top projects are programming tools. Programmers building better programming tools to help build better programming tools. I wonder what nice things we could have if we diverted half of that effort into end-user projects with the tools we've already got.

I think that's simply an artifact of the fact that client projects using these open source projects are often not themselves open sourced, or hosted on GitHub.

Well who knows programmers better than programmers themselves? And, if programmers decide to give their result back to the community, who are they going to give it back to before giving back to their own community?

There's plenty of open source projects that are not aiming for the programmers, but since they're not aiming them, how are they going to become popular on a site that's mainly for the programmers if the programmers themselves cannot use it in their own advantage?

I think this was one reason for PHP and MySQL's initial success. There were simply a lot of useful open source end user projects using it (WordPress, phpMyAdmin, MediaWiki, Drupal, SugarCRM, phpBB, Gallery, Magento) . Say what you will about the coding style and issues with these applications, but millions of people built real businesses on top of them. And I don't remember feeling like I was falling behind if I didn't write my application an entirely new way every few months. I was mostly working on solving a problem for a paying customer.

I really want to be able to run this for a specified repository

I was surprised to see a giant Jekyll cluster.

Reply on news.ycombinator.com