Putting Freebsd on little Wi-Fi router devices | Hacker News Reader

Putting Freebsd on little Wi-Fi router devices

adrianchadd.blogspot.com

84 pointsfcambus35 comments

My primary use-case in small border firewalls like this is running sshuttle on them as a lightweight VPN.

The problem is, FreeBSD cannot properly run sshuttle - it can do it for regular traffic, and it works well, but it cannot do DNS lookups for the entire network over the tunnel. This shortcoming is documented by the sshuttle README @ github.[1]

I am very interested in having this fixed / worked around and am willing to PAY SOMEONE TO FIX IT. If anyone out there is interested in (finally) getting sshuttle working on FreeBSD, please email info@rsync.net and we can discuss.

[1] https://github.com/apenwarr/sshuttle

You sure you linked the right repo? There are forks with newer activity, and the linked readme doesn't say anything about DNS not working? (Or I'm to tired right now to find it) Is any of the forks a new "official" version?

If this is still a problem then yeah, join #freebsd on efnet and ask for some help. :)

Where in the README does it state that it is unable to do DNS?

Avery hasn't put much effort into sshuttle for some time - you probably want Brian May's fork, which is now https://github.com/sshuttle/sshuttle

In any case, since 2011 sshuttle has supported DNS interception via a divert socket, so although it might not work today I suspect it should be somewhat straightforward to fix. I haven't looked into this yet as my only use case for sshuttle involves running it on the end host itself (typically my laptop).

Yes, you're right - I lazily linked to the original project.

However the new project has the same issues with DNS over VPN on FreeBSD. I have personally corresponded with the current maintainer, etc., and since they build and test only on Linux, it's not something that ever got fixed.

If you read the original HOWTO/README from the original project, it details the issues and I can confirm that no amount of settings changes or workarounds will allow you to get DNS over the tunnel on FreeBSD.

I see this issue has been discussed on the FreeBSD lists on occasion, but I can't find it on the sshuttle list in the four years of archives I've got. I also don't see a reference to this in the README.md in the original project.

The threads I have managed to find largely amount to "it doesn't work." Are you aware of anywhere the specific details of the issue are posted (debug logs from sshuttle, packet captures, etc.)?

that's awesome. now for those who have a plethora of other MIPS & ARM devices but next to zero kernel hacking skills, what can they do?

If you cannot physically donate them, perhaps you can offer remote access: hook them through a serial line to a computer that other people can ssh into.

You'll need a small contraption to be able to reset the device remotely. If you have a friend who knows a thing or two about hardware, he can come up with one in a few hours (it just needs to momentarily interrupt the connected device's power supply).

jump in and ask questions. :)

I believe this was more about "how to make it scale for the masses", and not so much about "how to make it work for one more person with less technical knowledge".

love how you had some downvotes here because people don't realize you're the blog post author

Thanks for working on this. My friend has been looking at doing some consulting and wanted to be able to distribute low cost FreeBSD-based routers. Now the framework is here...

> love how you had some downvotes here because people don't realize you're the blog post author

Not sure I get this criticism.

To my understanding, downvoting is a judgement of a specific comment. It is neither about the person who wrote it, nor about whatever else that person wrote.

In particular, I don't see how these donvotes diminished the author's work on that article in any way. The 50 upvotes (currently) on that article tell an entirely different story.

Why would anyone downvote "jump in and ask questions :-)"? What is there to disagree about there?

Nothing.

It was downvoted because it looked like a stranger telling someone what to do.

ok. how do I add support for other similar Atheros MIPS SoCs? I have a few, and a bit of FreeBSD user/admin experience; but the kernel config files in the wifi git repo might as well have be conjured up using magic, to me.

more directly: how'd you do what you did? :)

Aha! Jump into #freebsd on IRC (on efnet, don't ask) and poke me (adri) and/or ask for help.

If it's the same SoC in a different box, then it isn't too hard:

* Attach serial console, boot to uboot. If you're using a d-link then you likely can't without some hardware hacking (and even then they sometimes disable serial RX in u-boot. Grr.) If you're using tp-link then you mash 'tpl' at the console whilst uboot is counting down from 1 second -> 0 seconds to drop into the prompt. * Chances are an existing kernel will boot - you can test using tftpboot and go. So, build an existing image, grab kernel.BLAH from ../tftpboot/, put it somewhere tftp'able, and:

uboot> setenv ipaddr ip-for-AP uboot> setenv serverip ip-of-server uboot> tftpboot 0x80050000 kernel.BLAH uboot> go 0x80050100

Some uboots don't have 'go', only 'bootm'. In that case you can grab the kernel.BLAH.lzma.uboot file - it's a compressed u-boot application. You have to ensure you load it /above/ where FreeBSD boots from, as uboot will decompress it for you into the right spot (here it's loaded into 0x80050000.) I typically put it 16MB above (and thus assume you have at least 32MB RAM):

uboot> tftpboot 0x81050000 kernel.BLAH.lzma.uboot uboot> bootm 0x81050000

It should decompress the kernel and start running. If it's the same SoC, then it should boot.

There are a few things you'll need for the board, which typically you lift from the openwrt board config files :) :

* The ethernet setup (what mode each port is in, PLL setups, etc) * the ethernet switch setup (which switch, internal/external, whether ports are hooked up pass-through or to the switch itself, the port config, LED config, etc.) * The atheros onboard wifi and where in flash the calibration data is. * The MAC addresses - where in flash are they. * if there's PCI/PCIe devices, where that is and where in flash the calibration data is * What the GPIO setup looks like. * If there's any custom required GPIO hacks - like say, "wifi RX LNA on gpio X", "enable the serial UART GPIOs explicitly", etc.

That part will take a little more to describe. Maybe I should also write a blog post/wiki article about it. But if you get it to the point of tftpboot'ing on a new piece of hardware then I'll absolutely help you with the rest on IRC and we can write it up.

If it's a different SoC - that's a different article. I have support up to the QCA955x - I think I have two or three more newer SoCs and ethernet switches to write support for, and some more wifi support to port from their reference driver. If you really want to do /this/ part then I'll write that up separately. :)

Can't help myself, why is there an official efnet and freenode channel? What's the deal there?

FreeBSD's older than freenode. Inertia is.. inertia. A lot of FreeBSD developers only hang out on efnet, so that's where a lot of stuff happens.

ah, good, was worried there was a split of the community or something.

> Others have added ipfw support to do NAT and firewalling - I'm going to add configuration rules for NAT, IPFW and routing soon so it's all integrated.

I thought that most of the BSD world had transitioned to PF at this point. I checked FreeBSD handbook and indeed all three firewalls are supported: IPF, IPFW, PF. Are the older ones still really relevant?

ipfw's popular, well supported, and not going anywhere. It's still the default FreeBSD firewall[1], PC-BSD recently switched their default to it[2], and DragonFlyBSD recently rewrote it[3].

1: https://svnweb.freebsd.org/base/head/etc/rc.firewall?view=ma...

2: https://forums.pcbsd.org/showthread.php?tid=19683

3: http://www.dragonflybsd.org/docs/ipfw2/

pfsense is using PF, and I'm not going to try and take any steam away from them.

ipfw is small, lightweight and fine for what I'm experimenting with.

If people want to include pf and send me a pull request with that as an option then by all means, please do!

pfSense sucks. PF sucks too.

glare

How little is little? Can I put it on the "original" linux router, the Linksys WRT54GL with 16MiB RAM and 4 MiB flash?

I believe it's impossible to squeeze relatively recent (9 or 10) FreeBSD versions down to that size. The linked git repo makes mention of needing at least 8MB for some builds. And you don't get IPv6 at that size.

Yup. OpenWRT uses (patched!) Linux, but with a much smaller set of libraries/utilities to fit in small storage footprints. I'm using ye normal FreeBSD libc, libcrypto, etc - things are pretty big.

It could be fun to take something like the BSD kernel and an alternative BSD licenced userland to make a much smaller footprint device, but I don't have the energy to /also/ do that.

If someone would like to do that then please do step up and do it! I wouldn't even mind if the FreeBSD wifi stack and Atheros MIPS support was ported back to NetBSD and NetBSD was used for this purpose.

Can you build a BSD on a case-insensitive file system? I wanted to look at modifying openwrt to fit on my routers but I was shocked to discover that linux needs a case-sensitve file system to build.

Anyway, it was a good read, thanks.

nope, it only builds inside freebsd for now. that's a bit of a sore point which I'd like to fix, but .. you know, ENOTIME.

NetBSD has some support of some kind for some of the Atheros chipsets. I understand even less about MIPS in NetBSD than I do in FreeBSD; but I'm very interested and financially motivated.

I'll have to get on freebsd-mips and start getting annoying with my simpleton questions :)

He mentions pfSense and though it's sort of tangential, I have used pfSense on a Netgate device for years and years and been very happy with its performance. Highly recommended.

Netgates are nice, but not cheap. What do you do when you want to run FreeBSD on a $50 wifi router you have laying around?

Not $50 and you probably don't already own one, but it can be done on the Uniquiti EdgeRouter Lite ($100, runs Debian MIPS + Vyatta as the default officially-supported Ubiquiti image but very hackable): http://rtfm.net/FreeBSD/ERL/

to get the performance people are constantly raving about, you need the very proprietary binary blobs that take full advantage of the Cavium hardware acceleration.

otherwise it's just another beefy MIPS machine.

Yes, this is an important bit of subtext that most people don't realize.

Reply on news.ycombinator.com