Hacker News Reader

Comment by andremedeiros | Hacker News Reader

Hi all! I wrote this code, and for educational purposes, I’d be more than happy to answer any questions.

A gentle reminder that the whole history is here for show. This is very vulnerable — you can see the times I was in a rush, or when I took my time. You’ll find my mistakes. Either way, all I ask is please be kind. I know there are 50 things I could have done better, but this is a product of the reality at the time — a balance between something that works and something that was cost effective for Christian.

View on HN

You made something that people got a lot of value out of. That’s incredible. The actual code and commits are just the journey. They’re not the product. I’m rather delighted every time someone shows a realistic journey full of all the things we all experience, but some of us pretend don’t happen. Thanks for being vulnerable and sharing this.

This is an amazing way to look at it. Thank you for your perspective <3

Any developer should be proud of the value that code/backend provided millions of users.

Amen to that.

When I first started coding (some 15 years ago) I wrote the ugliest, buggiest and "owasp-ridden-est" PHP spaghetti code. Turns out that code went viral and was used by millions of users for a few years.

Yet, I came to the conclusion that that's the best code I've ever written, for it's been the most used code I released so far.

Real software development is far different than the ideal one.

Software managers take note… Your MVP isn’t necessarily your perfect code. It’s the messy, “get it out the door”, value it brings to others. This is your product, refine it from there but don’t hold it back for sake of polishing a turd. Myth busters proved you can indeed, polish a turd. Don’t waste the sprints.

No, that's just wrong. It's totally fine to put scrappy code up to get a product going, but that time has long since passed if you're already at a scale to have dedicated managers.

At that point you're likely going to cause more damage to the brand/corp with the half-assed implementation then the value you're providing with said code.

I tend to agree with this. My spouse is currently entrenched in an over-budget, under-funded, mismanaged project that was recently announced by the CEO.

That’s at a Fortune 500. It all started with a “well we can do an MVP and get that out the door first.”

It turns out that was a massive mistake.

You'll note I said MVP. To get the product going. I agree that once it's going it's best to refine and polish tested features but software managers are still usually hands on at launch phase. Once you have your fit then follow best practice and have a proper process in place (agile, waterfall, whatever). What scale does one have to be to have a manager? Do companies operate without management?

This. A thousand times this. I work on a household-name product with millions of users. It works pretty well most of the time but also has frequent outages, bugs, and usability issues, during which times the 1-star reviews pile up.

By far our biggest problems are:

1. Leadership at the time just wanted an MVP, and they wanted it yesterday.

2. Leadership now believes "You don't need more people, and rewrites are a waste of time. Look what we've built with a skeleton crew and tight timelines!"

Fellow engineers, beware: if you train your leaders to believe your team can deliver with no budget and unrealistic timelines, they will continue to give you no budget and unrealistic timelines.

Those are indeed leadership failings. After-market processes should be put in place to ensure quality. If that means slowing the cadence of releases, fine, so long as the value delivered is there. If you have millions of users, it sounds like it is. In which case my point about just getting it out the door isn’t applicable.

What a completely lovely way to express what I expect many of us are thinking. Thank you.

What an awesome comment.

Thanks for your work. Apollo made using Reddit a decent experience, and clearly you had a very large part in that.

User rapnie mentioned the code in the repository doesn't have a license [0]. You might want to review that.

[0] https://news.ycombinator.com/item?id=36256566

I’ll check in with Christian and see what he wants to do. Got any particular recommendations?

You probably want MIT

Got it, thanks!

The advice in https://news.ycombinator.com/item?id=36256665 is more thorough, I wouldn’t just skip to this one.

You may want to also check out Apache.

Here is a good comparison of licenses.

https://gist.github.com/nicolasdao/a7adda51f2f185e8d2700e157...

This gist doesn't even mention AGPL which as other users mentioned is much more appropriate for server-side apps than GPL.

GPL will ensure that the source is not used by any projects not affiliated with you who are trying to profit off of your work.

AGPL will do that, no GPL

GPL on server side software doesn't really matter, that is why the AGPL exists

>GPL on server side software doesn't really matter

That's maybe too general. There are cases where it does matter.

Maybe "GPL on software-as-a-service code has limited usefulness"

Even then, people do sometimes choose it on purpose for SaaS, where they want to ensure any redistributed code, where the entity doing it makes that choice...is bound by the GPL.

Some options spring to mind:

2. AGPL, to ensure that if someone does take this forward, no part of it can be closed source without your agreement.

3. MIT, which is extremely permissive and effectively you don't care what someone does with it, commercial or not.

Depending on your comfort level I would start with the middle option, I don't think people tend to mind you re-licensing more liberally but if a community has formed and you decide to go commercial further down the road it can cause problems.

Gives those who wish to learn and stops from anything taking the future glory.

Apollo was one of the apps that was basically giving up due the API changes, right? If they aren’t doing anything with the code, might as well let people fork it.

Just not MIT or BSD, I like these sorts of licenses normally, but it would be a bummer if Reddit got to steal their code after all this bad behavior.

Agree, would be awful if Reddit themselves just decide to lift and reuse this!

Just add them as a specific exception to whatever license you use, if that's your wish.

Why would Reddit add another backend layer?

If they’re unhappy with their own API, this could provide an abstraction layer for the app. They rewrite their client to use this API and it frees up two teams to work (almost) independently. The UI team can work on their roadmap while the backend team fixes the internal API or retires the logic and implement it into this new layer.

I’m not saying they’ll do this or that it makes sense. But I’ve definitely done it in much smaller projects.

This here is the superstar comment. There are other options but this is an excellent start.

Take these recommendations with a grain of salt and make sure you understand the implications of each license.

I wouldn’t add a license unless you specifically want to allow people to copy and use this code, otherwise just ensure that the appropriate copyright notice is present. No license is better than a badly written or badly chosen license — no license already means that all rights are reserved and no rights are given.

Having to ask people be kind about your code on HN is such a shame of reality to me. Half of brilliance is having your working version, used by many people. You had working code that brought reddit to so many reddit users every day that was better than what the official app of reddit provided.

Some of the 'code' running massively profitable departments in finance are spreadsheets, on shared drives, with thousands of linked cells thought up by a brilliant employee who left the company five or six years in the past.

I wish I was kidding.

Thanks for how great you kept apollo.

I would love if we all agreed to value the joy of building on here. Not saying it needs to be nauseating positive, but just a default expectation of comments like “wow! This is cool, here are some things I might have done differently”

Show HN is always really close to to that ideal. One of the reasons I like this place.

It’s hard to put oneself out there like that.

Christian is very visible in the community, but I want to say thank you to you too.

Thank you for all the work you did! This made a lot of our experiences great.

This right here is why I did it. To see a friend succeed, and to help make a community happier <3

That's really wholesome which makes this incident much worse IMHO - it's the evil greedy corpo growth bs mindset vs the good will of few people that care about their community.. dystopic world

This sentiment is what made Reddit a great community for smaller and specific subs.

It’s just people talking around a topic and helping each other. Subreddit mods doing everything they can to keep their area clean for people reading. I’ve been on Reddit over for over a decade and this is the power behind it. Just wanting to help and bring good member.

Thank you Andre. For your work and commitment to Christian, Apollo, and the community.

Your code was production ready for millions of users, that's all that matters

Do you think Christian would be willing to open source the app (under a non-commercial license) and allow us to build it with our own Reddit API keys?

That way we could still use Apollo, but pay for it ourselves

So am I right in thinking that the backend was needed to poll the user account in order to generate notification? Was that one of the issues that reddit had with how apollo was using their apis?

That’s how it worked, yes, but we went to great lengths to ensure that call had minimal impact on their end.

The TLDR here is that we only ever requested new notifications (give me anything that came after this thing I know about), which funnily enough then broke for some users, because if a reply you got notified about gets deleted and you use it for reference, Reddit will never return anything new to you, so I had to go and write a 2nd service that runs much more infrequently but does some checks on each account to make sure this isn’t the case.

> Was that one of the issues that reddit had with how apollo was using their apis?

I doubt reddit had any actual issues other than it impacting profits and no optimization to Apollo would fix it.

Maybe a description of what we're looking at in the Readme would be helpful. Why wouldn't the Apollo client just directly call reddit? Or does it partially do that and this server component is additional for certain functionality? What functionality does this offer?

That’s great feedback. Any particular areas I can walk you through?

Since I have no idea what's even here, I can't really answer that. Maybe just a few sentences on what this code does in general?

I’ll be happy to once I sit down in front of my computer.

A naïve question, but why is a backend needed and why couldn’t the app call the APIs directly? Just curious to learn!

Sibling comment above says it was mostly for notification, but flowing calls through their server also meant they didn't need to resubmit to Apple most of the time when something broke on Reddit's side.

Yeah. I’d imagine the less client side code the better when you have somebody getting paid $1/hour to randomly reject apps as the final step in hour launch process.

This was for push notifications to IOS - to tell the user when (for example) someone had replied to one of their comments - even when Apollo wasn't running.

Why not point Apollo at a Reddit alternative, or use the open-source code for Reddit (or code like this https://github.com/libertysoft3/saidit) instead of shutting it down?

If Apollo's users (or a good percentage of them) moved over to an alternative platform, that would be poetic justice, at least.

To go even further, all the app devs should get together and create a new platform or pick one Reddit alternative and point all their apps at it.

That way it would be even more likely to get critical mass quickly. 3 weeks should be enough time to come up with a coordinated plan. Don't let Reddit push you out -- push them out and let your users stick with the app they love.

Reddit is only its users, nothing else. They don't own the users' loyalty -- the apps do. Take advantage of that and teach them a lesson by pointing the apps and users elsewhere!

Don't be ashamed of code you wrote that wasn't intended for an audience.

Most would hate to see what a sausage gets made out of, but nearly everyone will eat that same sausage.

I would demolish a couple hot dogs right about now…

FWIW, that just sounds like real code to me. You can spend your time polishing code until it shines and jerking off over the perfect crystal of logic you've created, or you can spend your time building something impressive that does cool stuff.

Especially when you're solo or on a small team, technical debt is actually something you can lean into. In this domain, refactoring is cheap and cutting corners can help you get where you need to go faster.

It's a too common mistake to think that because technical debt is a tar pit that kills many large products that this is universally true. The real trick is lowering the tolerance for flaws as the project grows and matures.

Your commit messages [1] are better than of many projects and seniors I have worked with. Nothing to be ashamed. You made something that "just worked" and delivered value. You should be proud!

Sample of latest commit messages:

batch size wants to be smaller for less locking time, not higher

increase scheduler batch size

bump locks again

make pool sizes customized

increase prefetch

[skip render] remove old redis from infra

increase poolsize a bit

bring new redis up to line in blueprints

reduce pool size

bump pg client count

tweak pool sizes and consumer count

double the amount of consumers

small tweaks

[ci skip] update pgbouncer github url

clean allow lists

add new redis instances

make connections to the correct redis url envs

add new env variables

spam prevention

update reddit http client

feat(ops): add distributed tracing (#109)

I’m a long time paid customer of apollo. I have only used Reddit because apollo exists. What an amazing app. Thank you for all the hard work.

> A gentle reminder that the whole history is here for show. This is very vulnerable — you can see the times I was in a rush, or when I took my time. You’ll find my mistakes. Either way, all I ask is please be kind. I know there are 50 things I could have done better, but this is a product of the reality at the time — a balance between something that works and something that was cost effective for Christian.

The way I took Reddit's comments were there were things that could be optimised to reduce costs. I've not seen a codebase yet where that isn't true.

Sadly, in the hands of Redditors who think you can host an API on AWS for $1 per million requests comments like come across as "Oh they did shoddy work".

What I think is we're seeing production quality code like the majority of us have running.

Christian talked about this.

A payload of 25 comments vs a payload of 100, takes less time to generate and deliver.

So sometimes, to optimize his users experience, he might request 25 to show something quick to the user and then do a request for 100 and append it.

He also mentioned that he’s being compared to other apps, _not_ to the API limit (which he was under anyway).

So, can it be less?, of course. Heck, he could make it 1/minute or even 1/day. But his quota was 60/min. So why not use the 60?.

The Reddit API usage is as efficient as it can be, server wise, although I’d like to be proven wrong by them.

As far as the app goes, an average of ~330 reqs/user/day is pretty good, considering they built the platform budgeting for 86,400.

Yeah, every time I see that number trotted out I’m just like “and the point is…?”

If there’s an issue with the cost to process that many calls I’m much more suspicious of inefficiency on the API side than the client. Those numbers seem very reasonable.

I think the point of inefficiency was that Apollo could say money and make the costs less than they would be by doing less requests. If someone is saying X is too much it's a valid rebuttal it wouldn't be X but Y if you made a few changes.

I don't think Reddit cares about the number of requests. They just want paid. They weren't caring before when it was free.

That's just the reality of writing code, we all make mistakes and they haunt us all :)

As another who will stop using Reddit when Apollo goes dark, I wanted to say thank you for your work with Christian!

Apollo saved my Reddit use after they bought and killed Alien Blue. I have gotten so much out of and out so much into the more obscure technology subreddits. It’ll be missed.

I appreciate all your work. For what it’s worth, I’ve never had server-side issues with Apollo (that I’ve noticed anyway!)

Edit: typo

Sorry you feel on the defense. Developers can be very aggressive with feedback and I know the feeling. To echo others, "Build something that works first", and you have something that works :)

IMHO the gesture of releasing the code shows how much integrity there is behind the devs, and I could care less if there's not-best-practice in there.

Why don't you and Christian ask Apple for a small donation, and replace the reddit API with your own? You must have talked about the idea?

I am waiting on delivery of an iPad pro, my first apple device in 10 years. And, even though I hate reddit, I was excited to check out Apollo, if for no other reason than to see what a great ios app looks like.

I know that reimplementing reddit API from scratch will take more than a few weeks. But, I'm sure loyal Apollo users will be happy with a much scaled-down version. Maybe, to begin with you put the app in Read Only mode, and point it to a reddit archive. Then work on making one sub read/write again and expand from there.

I mean, even just a good reddit archive browser has TONS of value. I'd happily pay a small sum for an awesome reddit archive explorer.

"Chatgpt, build the back end for this API: $(cat reddit_api_docs)"

I was diving into your code and noticed you were using golang-migrate for running your migration queries. I never heard of it before and the number of databases they support sounds interesting!

How was the experience overall with this tool? If you have experience with Flyway, how does it compare?

You wrote code that works and powers an app for millions every month and any bugs or 'bad' ways of doing it are clearly sufficient to maintain a large, healthy user base, so you should feel nothing but pride!

Also, thank you for making this code available, but would it be possible for you and Christian to choose a license for it so people can use it? I don't think which license matters all that much, personally, but I can also understand wanting to keep it proprietary, either way, thank you again for all your (and the rest of the Apollo teams') work.

Are you proud of your code ? Which design patterns do you use ? Any lessons that you've learned to be shared ? Thanks.

I’m never proud of my code haha. This could always be better. No specific design patterns either.

The one fun thing I learned was about databases. Given enough records, a query planner might decide to start doing things very differently and you will get woken up at 3am with a CPU usage alert. That was a fun one to debug for sure…

You shipped and found market fit on your own. A lot of funded teams haven’t been able to achieve what you’ve done.

I'm curious if ya'll have any plans to support a reddit alternative, like Tweetbot -> Ivory and other twitter clients. I love Lemmy. I think it would be an excellent middle finger to reddit, but I understand if you are ready for the next thing

My friend, that’s ALL of our code!

Making the code available is a wonderful thing to do. I love being able to see how others managed and structured a Go project that was actually meant to run in production.

Thank you.

Keep in mind that this isn’t rooted in any specific best practices. It just felt right.

Thank you for your contribution. This is a heroic act.

Wouldn't it be possible to change it to use Lemmy or some other alternative?

Honestly, Lemmy would probably not need half this code. We wrote this to work with an entity that exists in a “come and get it” model. Lemmy being open source means we could build the features right in the server and slash this code to a bare minimum (mostly the watcher stuff at this point.)

So, too much work to adapt it to hackernews?

Would love to have a decent client for this site…

HN is a whole other beast. A notification server would have to keep track of any replies to any of your comments / submissions in order to determine what to notify you about, purely because they don’t have a notification model.

You might want to put this in the readme

Reply on news.ycombinator.com