Hacker News Reader

Comment by Dolores12 | Hacker News Reader

Ok lets say i want to program a webcrawler in lisp & c# and cant figure out how to do it. I simple go and google it, right? how to crawl web page in lisp - returns unrelated websites. i still have no clue how to do it. how to crawl web page in c# - second link returns me a solution to my question.

So why would i waste my time if i can get things done faster in more popular programming languages? English or Esperanto?

View on HN

If you need to create a webcrawler out of pre-made pieces, you need to know what to look for. In this case, it would be a high-level networking library and an HTML parser. I haven't looked, but I'm sure you could find these libraries in Lisp.

If you don't know what a webcrawler entails, then you might not be the right audience for Lisp as it exists at the moment.

If I don't know how a webcrawler works, I would look at an programming language independent source like Wikipedia. After that I know what components I have to search for.

For common lisp, the first thing I would search for is a HTTP Client and a HTML Parser. https://duckduckgo.com/?q=common+lisp+http+client https://duckduckgo.com/?q=common+lisp+html+parsing

CLiki.net is always a good resource.

If you've finished building a webcrawler, please create a website where you explain webcrawling for (common-)lisp. So the next one will not have the same problem :-)

Two roads diverged in a yellow wood,

And sorry I could not travel both

And be one traveler, long I stood

And looked down one as far as I could

To where it bent in the undergrowth;

... ...

I shall be telling this with a sigh

Somewhere ages and ages hence:

Two roads diverged in a wood, and I—

I took the one less traveled by,

And that has made all the difference.

This poem was actually written satirically by Frost about going for walks with a friend who always complained afterward that they should have taken some other route and acted as if it would have been significantly better... when in reality it would have made no real difference:

Frost's biographer Lawrance Thompson is cited as saying that the poem's narrator is "one who habitually wastes energy in regretting any choice made: belatedly but wistfully he sighs over the attractive alternative rejected." According to the Thompson biography, Robert Frost: The Years of Triumph (1971), in his introduction in readings to the public, Frost would say that the speaker was based on his friend Edward Thomas. In Frost’s words, Thomas was "a person who, whichever road he went, would be sorry he didn’t go the other."

I think that makes the poem especially apt, considering how some programmers fret over language choice.

If you want to make your own DSL for crawling web sites, and you know Lisp, you'll probably be able to get it up and running pretty quickly. That's the kind of thing Lispers do for fun. Otherwise, just look at the language and what people say about it and decide for yourself if you're interested.

lisp has many dialects. try searching "webcrawler clojure"

Reply on news.ycombinator.com