Hacker News Reader

Comment by prepend | Hacker News Reader

This looks interesting but impractical for me to mirror hundreds of repos here just to get better search.

Has anyone played with this or similar tools that can provide search by indexing at source. It’s a little surprising that Google doesn’t just index everything public in Github, gitlab, etc.

I would like a good code search solution especially because I have projects across lots of different repo servers but want developers to be able to find code regardless of repo home. A colleague indexed it with solr and that was ok, but had limited semantic ability and definitely nothing like what Google describes where your recent search history and activity is displayed to each user.

View on HN

> It’s a little surprising that Google doesn’t just index everything public in Github, gitlab, etc.

GitHub excludes a few patterns in robots.txt that means a lot of the public data is never seen by crawlers that respect robots.txt, for example:

    Disallow: /*/*/tree/*
    Disallow: /*/*/blob/*

and

    Disallow: /*/*/commits/*/*
    Disallow: /*/*/commits/*?author
    Disallow: /*/*/commits/*?path
    Disallow: /*/*/branches
    Disallow: /*/*/tags

https://github.com/robots.txt

They probably want people landing mainly on

- user profiles

- repo roots

- individual issues

- individual pull requests

And I am glad they do, because when I search on Google, the above mentioned kinds of results are exactly what I am looking for, not individual source files etc.

https://about.sourcegraph.com

Author of the blog post here.

Thanks for taking a look. I'm glad you find Cloud Source Repositories interesting. We'd love to Cloud Source Repositories more practical to use. Can you explain what is impractical for you? Is the initial process of setting up multiple mirrors? Or the routine of having to go to another site for search?

(Disclaimer: I work at Google and am the PM on this product)

Thanks for responding. I work in an organization (happy to PM) with a few Github orgs and about 300 repos. Dozens of project teams work in them and use various GitHub features.

I have no easy insight or overview into code across these repos without scripting. The impractical part is in cloning mirroring all of these repos to get insight from your new function because user behavior is baked into all the workflow around the Github repos.

The go to another site isn’t so bad as this is a new function and not available in Github, so programmers will likely check it out when they need.

But manually forming everything and syncing. It would be neat if I can import an org in one fell swoop.

That being said, I do have scripting forking everything over on my list of things to do when I have cycles on the weekend.

Reply on news.ycombinator.com