tabbott

Search engines are great at rendering content using JavaScript, they're just not able to explore the Zulip web application's URL structure, so they tend to index one page per Zulip installation.

The current solution is that projects that want their publicly available content to be search engine indexed set up https://github.com/zulip/zulip-archive, which usually involves hooking a GitHub action up to GitHub Pages for a 0-infrastructure deployment. Ultimately, that's the same model as an IRC indexing project like this one: a separate tool from the chat server is responsible for search engine indexing.

Lean runs one here: https://leanprover-community.github.io/archive/, but it looks like it hasn't updated in a year, so likely whoever in the Lean community needs to investigate why it isn't updating.

https://github.com/zulip/zulip/issues/21881 tracks our goal of making the server natively offer search engine indexing. The current separate archive tool model has some advantages (search engine load can't break the server, for example), but I think it'll be worth doing the native version when we can find the resources for it.

Source: I lead the Zulip project.

Without a clear explanation of methodology, this is meaningless. My guess is this statistic is generated using misleading techniques like classifying "code changes generated by existing bulk/automated refactoring tools" as "AI generated".

I hope the FTC staffs a large office for enforcement on this. There are surely many hundreds of companies in the business of selling fake reviews, many of them outside the US, and I don't expect much change in the consumer reality of "most reviews are fake" without a great deal of investigatory effort tracing money flows to shut these operations down.

Greppability is the reason why I feel it's really quite unfortunate that HTML's dataset attribute went with "canonicalizing" to camel case; if you're using any non-camel-case data attribute names in your project, then they immediately become not fully greppable when making use of `dataset`.

https://developer.mozilla.org/en-US/docs/Web/API/HTMLElement...

It is kinda amazing how consistently status pages show everything fine during a total outage. It's not that hard to connect a status page to end-to-end monitoring statistics...

Zulip is designed to use your OS/browser's timezone for this and all other displays of times (say, when messages were sent). This is usually what you want, since it means you just need to make sure your computer's clock is correct and not manually update every app you're using.

In the issue you linked, the root problem is that the anti-fingerprinting browser feature has the browser pretending that its preferred timezone is UTC, resulting in the user seeing all times displayed to them in UTC.

It has nothing to do with the "global times" feature -- that feature just renders an element in your message in the same timezone that it renders the timestamps.

That's effectively an API feature only: There's no button or typeahead for entering it. Zulip's version is a first-class feature with a compose box button and typeahead, designed to be convenient enough that humans routinely use it rather than typing "10 Pacific time".

I wonder if there's a story here involving a URL shortener service having hidden costs? I can imagine there being something in the abuse space that makes it feel more expensive than just the hosting costs to operate.

This is so cool!

It reminds me a bit of one my favorite debugging techniques: Running an individual test with coverage reporting enabled, and then clicking around the HTML coverage report to see exactly what code path the test followed, without needing to use either print statements or a specialized debugger.

Very helpful for answering questions of the form "Why does this test not exercise the code I thought it did?".

If you have a bit of time, I'd love it if you stopped by chat.zulip.org to discuss this more interactively.

I don't think we have heard any of them from other users, so feedback including a specific example of an organization or user who struggled with this would be helpful for us to debug your experience with this feature. For the thread, my understanding of the situation is:

- The one-time setup steps for a self-hosted installation to enable web-public streams are a couple minutes of work, and to me seem immaterial compared to the minimum realistic effort for getting a server, getting a domain, transactional email provider, and SSL certificate, and then actually installing and configuring the server in the first place.

- The default stream type when creating a new stream is web-public if the feature is enabled for an organization, for users with permission to create them.

We don't allow normal users to create web-public channels as an anti-abuse measure. Trust me, you don't want it to possible for a threat actor to be able to sign up for an account in your open community, create a secret web-public channel with just themselves as a subscriber, and start hosting malware on your domain.

I hate when anti-abuse concerns limit our ability to design features in the most convenient way possible, but that's life building products for the Internet.

If anyone has ideas for how we can make this better, we'd love to discuss!

Just a quick correction: Zulip supports logging in with your Google account (as well as GitHub, GitLab and Apple): https://zulip.com/help/configure-authentication-methods. In Zulip Cloud, those options are enabled by default, but chat.futo.org is a self-hosted system, and I assume they either haven't done the setup work (getting API keys, etc.) for those authentication methods, or don't want to for policy reasons.

As to why every organization has its own accounts, each organization has different people administering/controlling them, and separate accounts for different Zulip organizations is a lot cleaner of a security model, both for business customers but also for projects that are paranoid or opinionated about authentication options.

That said, we are planning to create a Zulip Cloud SSO option that would allow users in the many communities that aren't picky about authentication to use a single login for all their Zulip Cloud organizations. Feedback on how exactly you'd like to see that work is appreciated, especially in #feedback on https://chat.zulip.org, where it'd have the most visibility within our project.

Open organizations can also set any channel to be publicly accessible without logging in, which I'd expect to be useful for something like FUTO: https://zulip.com/help/public-access-option.

We (Zulip) also have a free Community plan that we sponsor for thousands of communities, that does not limit message history.

Yeah I think Slack's updated stated internal policies are about as reasonable as one can hope for from a tech giant, if one can trust them to stand by those policies. Your article was on my mind when writing this, I guess I should have linked it.

The crux of the matter is whether you can trust a big tech company to do what they claim they will. They all think AI is worth infinite dollars. In that world, without some very clear, painful, straightforward, contractual penalty ... well we've seen that the tech giant plan is that rules are meant to constrain your competitors' behavior, not yours.

If they wrote "If any of your data is discovered to have been in an AI training model, Slack owes you 10x your lifetime payments to Slack, and any involved whistleblowers get 1% of the total paid" in their terms of service, which means if Slack screws this up, the company is immediately bankrupt, that might prove effective. But a promise in a "privacy principles" policy that doesn't appear to actually be incorporated into the core ToS does not have a lot of teeth.

Zulip project leader here.

I think you're missing a big part of the point of the post: Which is that if you self-hosting, nobody can train models on your data, if you're going to use a cloud service, you should use one where you can move data to self-hosting, and where you can trust the vendor.

We do basically guarantee that you won't have your organization's data included in AI training in Zulip Cloud without consent. But yes, we're not ruling out the possibility of some sort of opt-in feature that might be useful in Zulip Cloud.

I'm humble enough to not pretend I know what will be possible/expected in terms of AI technology in 5-10 years, but one could easily imagine some sort of tool trained on web-public channel data in open Zulip communities being a thing that could make sense if done with appropriate consent. If such a thing were desired, I don't think it most of the concerns related to the slack controversy would apply.

I'd be curious to investigate this; do you have many tens of thousands of unread messages in the Lean community, or is it a new account?

I ask because we have a known issue in Zulip that the first minute or two of using the product is quite slow for folks with 100K for unread messages because it the web app's client-side data fetching system ends up fetching them all in order to make clicking around the app afterwards extremely snappy.

We've reworked that system over the last few months, and it'll be deployed to Zulip Cloud next week if all goes well. But if you don't have a lot of unread messages, I'd very much like to understand why -- I'd love for you to stop by chat.zulip.org and debug with us a bit if you can spare a bit of time.

(It could be that the Lean community Zulip involves rendering a LOT of LaTeX, for example)

I lead the Zulip project, feel free to stop by chat.zulip.org to discuss what you're trying to do.

Just to clarify for anyone here reading, Zulip can be scaled out, but largely our philosophy is that you should only scale out if you need to, since it comes with extra fiddly operational headaches and is expensive in dollars. We've designed Zulip with an efficient data model and well-indexes queries, and are constantly working as we add new features to maintain the property that only multi-tenant installations like Zulip Cloud should need to scale it out.

In particular, Zulip on a single server with 16GB of RAM can handle 10,000s of users, thousands of which are concurrently active. (That's what we use for chat.zulip.org, for example, and the machine is usually about 96% idle with only half its memory used).

(Keep in mind that Zulip's public access option means you don't need every user to create an account in order to access content).

I don't know your context, so I can't comment on whether zulip would be suitable. But there are very few communities today having complex conversations involving 100K+ people in a single organization/workspace, so I would expect it to be quite rare that autoscaling would be relevant. (Certainly there are Discords with over a million users, like the Midjourney one, but most of those users are just users the product, not having discussions at all; the company famously made the unusual decision to only offer a Discord bot as UI for their product).

https://zulip.readthedocs.io/en/latest/production/requiremen... has some basic guidelines on Zulip's scalability for a single-instance deployment, but those are fairly conservative -- the goal is for folks to be able to deploy with those settings and be safe even if their load profile per user is fairly heavy for whatever reason.

Indeed, Zulip has several views along the lines of "All threads", designed for different use cases and user preferences:

- https://zulip.com/help/inbox

- https://zulip.com/help/recent-conversations

- https://zulip.com/help/all-messages

Following up on this thread, we spent a bunch of time talking to users over the last few days and have done more or less exactly that:

https://blog.zulip.com/2023/12/19/self-hosting-without-comme...

Thanks for the thoughtful response! I really appreciate it. I'm doing my best to be as transparent as possible, so here's a very long reply.

This is not a change motivated by short-term needs.

The problem statement is not that we need to fund the costs of delivering push notifications. We need to fund the costs of building Zulip -- the server, apps for every platform, support for a vast range of different self-hosted configurations, etc.

Yes, we are extremely capital-efficient with a small core team who have taken some very big pay cuts to work on 100% open-source software. But even so, we need a business model that makes sense. A business model based on charging a small markup on top of the marginal costs of delivering notifications can't fund investments in the Zulip main experience or the self-hosted Zulip experience.

The main revenue source for Zulip thus far has been Zulip Cloud, and it's proven hard to grow that business, largely due to competition with the $0 price of self-hosting Zulip, which gets you most of the features of Slack Plus ($12.50/user/month).

Zulip needs a healthy model where businesses that self-host Zulip (and rely on the SaaS push notifications service) contribute to its development, just like those that use the SaaS Zulip Cloud service. This doesn't specifically require the same pricing for the two products. But it does preclude a model where Zulip Business only includes push notifications and is extremely cheap.

On the specifics of pricing, we of course have had a lot of conversations with users prior to this announcement. Most of the objections we heard were from system administrators who told us they would struggle to convince their company to pay even $1/user/month -- either because it was a big company whose bureaucracy had "already bought Microsoft Teams", or because they were a startup that had a very very high bar for paying for software at all.

Meanwhile when we talked to leaders who make purchasing decisions for their company, they tend to think about budgets in comparison with other applications that their business relies on, or else compare with fully-loaded costs per employee, which are along the lines of 1000x the price of Zulip Business. Reactions tend to be: "$6.67/employee/month? That's nothing, I pay several times that for X, Y, and Z, which we use way less than our primary team collaboration tool."

If you think it's priced too high, I'd really appreciate it if you chat with whoever at your business would approve spending money on your Zulip installation, get their feedback, and send us an email. There may well be some categories of customers that we missed in those conversations, and specific examples are very helpful for considering whether there's a gap in our pricing model that we can patch.

We always appreciate well-written bug reports!

But for what it's worth, we're including support with Zulip Business, not consulting. Consulting is building custom features or whatnot that are just for a single customer.

Support is a guarantee that when you have a problem, you can ask for help and a human will investigate it and get back to you with what to do. It sounds like you've already received free support from us in the past.

Most of the issues that one helps a customer with in a support context are not unique to the customer -- but the support is still essential for the customer to actually have their problem solved, and investigating, debugging, and deduplicating reported issues responsively is effectively support work done by our (small!) team of people who need to make a living.

As noted in our email to customers, if you are seriously considering leaving Zulip because of this pricing, we'd like to hear from you. This pricing is intended to be very affordable as a cost for team chat for a business using it for their paid team, and the full price is intended primarily for that class of customer.

But without context on how you're using Zulip, I can't comment on whether your organization would be eligible for a discount or our community plan, but see https://zulip.com/plans/#self-hosted for several documented situations where an installation would not pay full price.

https://news.ycombinator.com/item?id=38660799 may also provide some useful context.

We of course carefully looked at following Rocket.Chat's previous business model of metering notifications. A lot of incentive and predictability problems with a per-notification pricing plan that I mentioned in https://news.ycombinator.com/item?id=38660799 apply, though to a lesser extent, because a lot of potential customers will be far from the threshold.

(FWIW, 10k mobile push notifications a month sounds high, but it is not a very high limit; in practice, a lot of teams of fewer than 25 users exceed it.)

I will say that we put a lot of careful design and engineering effort into doing this transition in a way that cannot result in unpleasant surprises like upgrading to the latest release breaking a feature you're relying on.

Using a 100% open source product provides customers with a lot of protection against bad behavior by vendors. If you're using a proprietary product (including the paid/proprietary versions of open core products that market themselves as open source), the vendor has a huge amount of power to move a feature you need to their Enterprise plan that costs 5x as much, and if they do that, you're stuck either running an old version (which will accumulate serious known security vulnerabilities), upgrading and dealing with the changes, or migrating to an alternative, which in situations where the product has all your data in it, can easily cost far more in than the product itself.

I can't say I have all the answers for how commercial open source software should work. I can tell you that open core is a much easier business model.

The hope is that people do care about the very significant benefits that come with using well-engineered, no tricks, 100% open source software, and that businesses that are relying on open-source software are willing to pay for the services and support that they need to succeed with it. Time will tell.

Pricing per seat is a better model for all parties.

- It's more predictable for the business (you can model what you'll pay in advance), so the business can model their total expenses and make a budget. I don't see how a customer could accurately predict how many mobile notifications per month they're going to generate for a product they're not using yet -- it depends on the personal preferences of every employee what notification settings they choose. Any business with a procurement function is going to look very skeptically at unpredictable costs.

- It's more predictable for us -- metered per notification would have all sorts of weird seasonality.

- A per-notification fee would have perverse incentives for us, like "we made the product really spammy with push notifications and it doubled our revenue". We would try very hard to resist that, but incentive structures are powerful, and we want our financial incentive to be to maximize the number of people who are happily self-hosting Zulip, not some other weird metric.

- A per-notification fee would have perverse incentives for the business, like the manager asking people to toggle their settings to save money rather than to choose the settings that make them most productive.

Are you using Zulip exclusively for paid team members in a first-world country? It's hard to imagine this pricing being a significant expensive if you are.

If not, check the sponsorships and discounts section of https://zulip.com/plans/#self-hosted

(In any case, our sales team would be very happy to discuss your or anyone else's individual situation -- specific situations are very helpful for figuring out gaps in policies where it might make sense to introduce a new plan)

Since there's a lot of discussion of it feeling expensive for "just push notifications", I think it's probably worth my addressing that directly as a top-level comment.

Zulip Business costs $6.67/user/month. While the only Zulip feature that can't work without purchasing a plan (or doing a huge amount of work to publish your own mobile apps) is mobile push notifications for businesses with 10+ users, https://zulip.com/plans/#self-hosted details dozens of features for which Zulip Business includes expert support.

Zulip is 100% open source. In open core products, those specific features just don't exist at all in the open source version. For example, Mattermost requires the proprietary Mattermost Professional [1] at $10/user/month if you want to use SSO, LDAP sync, user groups, read receipts, etc.

(Note also that $10/user/month is the minimum price at which Mattermost offers push notifications "for use in production environments" [2].)

If you compare Zulip's pricing options to Mattermost's, our offering is substantially more generous. It would be even if we went open core and moved a few dozen features of Zulip to a proprietary version. But we're not going to do that: we really do care about being an 100% open source project.

If you instead compare Zulip Business to using Zulip for your business's mission-critical communications between paid team members without paying for something, I think that's the wrong question. Skilled humans' time is incredibly valuable, and people spend a LOT of time in team chat tools. It is rational for businesses to pay a tiny fraction of the fully-loaded cost of their employees for the insurance that comes with support and mobile push notifications for their main collaboration tool.

(If you're using Zulip for something other than a server full of paid employees and contractors, see the Community plan and discounts for other use cases).

Finally, on the topic of cost to us: Supporting a business that is self-hosting complex, mission-critical software like Zulip is more expensive for us than having that business using Zulip Cloud. Hosting is cheap compared to humans who can debug anything, and there are big economies of scales in terms of human time for managing a large multi-tenant cloud installation.

Helping thousands of different people with varied skill levels self-hosted your application successfully is not cheap, even for a project like Zulip that is very focused on making self-hosting Just Work.

[1] https://mattermost.com/pricing/features/ [2] https://docs.mattermost.com/deploy/mobile-hpns.html

Hi, I lead the Zulip project.

I agree the existing React Native mobile apps have been the biggest pain point for Zulip. Our mobile team's main focus for the last year been rewriting our mobile apps in Flutter, which will result in apps that are faster, dramatically more correct in terms of how faithfully they implement Zulip's API and business logic, and with a cleaner design. Here's the section of the Zulip 8.0 blog post about it:

https://blog.zulip.com/2023/12/15/zulip-8-0-released/#apps

Curious readers may enjoy the main chat.zulip.org conversation it links to, about our evaluation of Flutter. But in short, our experience has been that it has proven to be a much much better platform than React Native for a small but very capable team that wants to build a high quality mobile app.

I should mention that the upstream Flutter project is awesome: it's been an absolute pleasure that when we find significant problems with the platform for our use case, like the Android scroll physics not feeling native, we have been able to contribute a series of pull requests to fix the issue for everyone. As a result, we've contributed a few dozen PRs to upstream Flutter already.

Thanks for the kind words!

For those curious, Zulip is deep into rewriting our mobile apps in Flutter (https://github.com/zulip/zulip-flutter). It's hard to predict the timing for that type of project, but the goal is to be able to replace the existing React Native mobile apps around the end of the year. We've been thrilled with the Flutter platform, both the technology and the community -- it's been just incredibly pleasant to contribute fixes for issues that are important to us to the upstream project, so they're fixed for everyone.

I'd be curious to hear what you don't like about Zulip's attachments work -- I'm not sure what you have in mind.

Just to clarify something: Zulip decorates streams with icons hinting the size of the audience for the stream, which can be helpful for considering the level of care that you might want to use in writing your messages.

In particular, Zulip uses hash icons for public streams open to all members of the organization, lock icons for private streams, and globe icons for web-public streams (https://zulip.com/help/public-access-option). So if you're seeing only globe icons, it's because you're in an organization where you only have access to web-public streams.

See https://zulip.com/help/stream-permissions for more details.

Thanks for the report! It's intentional that if you're logged in that you get to the plans page showing the current plan for your organization, but the redirect for the case where you're not logged should go to the root domain.

You can do search engine indexing via https://github.com/zulip/zulip-archive; it defaults to archiving your public access streams.

We will eventually support search engine indexing without the extra overhead of running a separate archive tool (likely as an organization-level settings checkbox, since not everyone who wants public access wants search engine indexing).

We're very happy to help debug that sort of thing in the Zulip development community: https://zulip.com/development-community/. Unfortunately, basically no modern team chat product provides satisfactory documentation of their data export format (E.g. Rocket.Chat is just a MongoDB database dump). I don't blame them -- all the data of your instance is a pretty complex thing to document, tends to change often, and doesn't get a ton of use to motivate investing hundreds of hours into writing and maintaining good documentation for it.

But it does mean the quality of import tools is limited by how many people have run it and reported all the issues they ran into so that we can fix them. It is not uncommon for our data import tools to need to work around things that are essentially data corruption in the original database.

Our Rocket.Chat import tool was indeed improved greatly in 6.0 last November thanks to a user with a very large and old Rocket.Chat instance submitting a very helpful pull request last Fall, so you may have better luck if you try again now.

For those readers who are skeptical whether this curriculum is really in widespread use, it definitely still is.

I have young kids and before listening to this podcast, my wife and I spent a bunch of time last Fall looking at schools for our eldest to start Kindergarten, because San Francisco's lottery system means there's no default school.

A lot of schools, even fancy private schools, used materials designed for this three cues reading curriculum. We were really puzzled why the reading curriculum featured stupid books where each page had a picture of a mouse and another object, and you were supposed to read "Mouse sees bear".

These books had two problems that were obvious to us as parents: You don't need to know how to read in order to appear to read the book (just be shown the pattern for what to say on each page), and the books are _incredibly boring_ -- even our 2-year-old wouldn't be interested in reading such a book. The teachers we talked to said things that in retrospect echo the marketing lines for this curriculum that you here about in this podcast.

We Googled why these stupid books are being used in schools at all, and found https://www.apmreports.org/episode/2019/08/22/whats-wrong-ho... (An earlier article the by the host of the Sold a Story podcast). I recommend the article for folks who want to read something on the topic, though the podcast is the result of her considerably deeper investigation into why this is happening, and I highly recommend it as well for those who have the time.

We ended up sending our daughter to a Montessori school; we picked it for several reasons, but one observation is that because they have their own philosophy about how to teach reading and writing, they're relatively resilient to this sort of fake "science" based fad.

(Other things we saw visiting schools that made me sad included plentiful posters about the CUBES math strategy, which felt like an algorithm for being able to produce the correct answer to a word problem without actually reading and understanding the words -- i.e. an adaptation of the math curriculum to a world where many children struggle to read).