confluence_perf

Would you be willing to share what type of additional export formats you're looking for, and for what purposes?

Hi - I'm coming from the Confluence team working on Performance, but I can say that we (all of Atlassian) are aware of performance is an area needing improvement, and we're working on it. Customer feedback (including logs, recordings, and exports of slow pages) is always very helpful to us. If you're open for followup over email to provide specific information, please let me know.

Thanks for mentioning that -> honestly I keep forgetting to ask because I work on the Cloud side.

I will see if I can find some server folks to ask around, but obviously can't promise any movement.

Hi Slackwise,

This is interesting -> I don't think Ctrl+S should be exiting editing in any form, can you describe this a little bit further? When you hit this is it:

a) moving to Preview (you can see the changes you made, but in preview mode not page view mode (the page tree won't be visible)

b) browser returns to the 'view page' (page tree visible), but your changes are not published (you may see a tag at the top "UNPUBLISHED CHANGES")

c) something else (please describe)

Second question: what does the browser's forward/back buttons display after the 'ctrl+S' result?

Hi t-writescode,

For the most part, each individual top line products' team operates out of a different site, with Confluence operating in Silicon Valley (mostly). I assume this is public information (or derivable) so I think it's safe to share.

Internally we have a few Confluence Cloud instances we use, with a single main one shared by the entire company (basically) which is located in a single area (somewhere?), though the VPN connection points are different.

So from a network topology standpoint our experience shouldn't be too different from most customers (at a very gross level) -> summary is, we definitely have users representative of 'bad networking', but you're right maybe I should be trying to intentionally degrade mine (right now i think its two cross US hops, but could be wrong).

I'll make sure our team looks at this dimension -> I don't think it's possible within our telemetry data, but maybe simulating it will get some interesting results.

I do recall reading once about something about AWS infrastructure (endpoints? edges?) having some configuration that causes omething like this: 1 - first returned packet is some some small size 2 - next packet (after ack is returned) can be double the size 3 - same 4 - same 5 - until max packet size

And though (1) is configurable (in theory), and I think the doubling/size increase is configurable (in theory), AWS does not allow for this configuration in their services.

But I can't find what I was reading so if anyone knows about that (a) being a problem and (b) how to work around it, let me know!

Both right

a) the public facing jira.atlassian.com is Server/DC instance - but this instance is only used for customer/outside world facing tickets (I think)

b) internally for our own development we use a few (several?) cloud instances -> but I can only speak to those I interact with (Conf Cloud and Jira Cloud primarily).

I appreciate the well wishes, honestly it means a lot - good guess too, I am in the US (maybe you knew, but most of Confluence Cloud is based out of the West Coast offices).

I don't know what the overlap is between Atlassian users and IT admins though - my previous job was on the vSphere UI and if you happen to know about the death of the Flash based client, this is not too far off.

Hopefully users stay willing to engage with us so we can improve the product as fast as possible.

Not offended - as an employee I have no specific insight into the actual headline term in the ToS - honestly I'm planning on tracking down someone in legal to help clarify this, since it seems like it currently as written (and currently as interpreted in worst case) unecessarily impedes me from doing my job.

I would never encourage anyone to violate a ToS of another product and apologize for anyone that was considering doing it due to my ask.

I think these are other possibilities: 1) (as stated) other products don't have such ToS 2) other products may have published their own metrics and made them available for consumption 3) from a more legal in depth standpoint, maybe other companies have such ToS terms but have clarified them to some point that makes them more clear about when they apply and when they don't

I don't think I know what that might exactly mean - I know we have synthetic traffic generation tools (and thus, measurements generated from the synthetic traffic), but I think those exhibit the same variance as production -> the backend for them are the same cloud IaaS systems and SW, so there's no 'sandboxed from all outside variance'.

If it means something else then I'm not aware if we do it or not.

Hi acdha,

Thank you for the advice - how about on the page load side? Our biggest problem is probably the variance issue (mentioned in other subthreads) -> we can't easily tell what is the difference between a slow and a fast load in many cases.

Even if we compare things that are available from the metrics like CPU, Mem, and Network speed, those are not very granular metrics (for example, to understand someone with 16 threads was actually at 95% Mem used during that page load), and have little correlation at a wide level with page load speed.

Best I can say here (as a Confluence PM not a Jira one, and in a public forum) is that customers are not the only ones that experience this pain, and the appropriate folks are notified on a regular basis.

I think the genesis of this is historically many different editable fields might had different modules behind them (not every single one a different one, but maybe a handful of common ones). It looks like for whatever reason we (Atlassian) haven't fully migrated all of them to the newest common editor modules -> I don't know anything for sure though

Thanks for the link! Yes this is the general guidance we're using too (0.1/1/10s), and one that we're reinforcing at every level of the company. This link does have more detail than I've seen in other places though, so it's an interesting read.

However I've not seen guidance on whether these should be P90 or P95 or P99 measures for example though. We've selected something internally, but obviously selecting amongst three 'measurement points' could drastically change general user's experience.

(HN is throttling my replies so apologies for delay)

I guess I meant that in a more general sense: prioritization is always about tradeoffs, and sometimes you're improving one (paint faster) or improving the other (interactive faster), sometimes both, sometimes trading off one versus the other.

We have looked into the network issues and some of it is similar to what you stated, we do have a known minimum given our chosen cloud infrastructure (separate from our software performance) - we obviously recognize we're not at that limit yet either though.

I have not tried what you mentioned above (load from disk), but I will give it a shot -> it may also give us a clue on how to make our performance testing lower variance, come to think of it......

(apologies for slow replies, HN is still throttling me due to downvotes)

Thank you for coming back and clarifying. Do you happen to have links to any public testing results of other tools, or guidance to this specificity - would love to use them to build a case internally

Most of what we've seen online are nowhere near this level of detail (X-ms for Y-%ile for Z-type of load)

(edit: clarified request)

"High variance as a consequence of poor software" is an interesting point and not one I'd considered -> I will take this to engineering and see if we can do anything about that (some components maybe, but we see high network variances too which seem unlikely to be fixable).

Showing that we're making progress isn't as much of a problem - similar to what you stated, the fixes themselves target large enough value that it's measurable at volume for sure, and even in testing.

The main issue is "degradations" -> catching any check-ins that can degrade performance. These are usually small individually (lets say, low double digit MS) within the variance noise), but add up over time, and by the time the degradation is really measurable, its complicated tracking down the root cause. Hopefully I described that in a way that makes sense?

Any suggestions welcome.

(Edit: downvoted too much and replies are throttled again) ----@lostdog Thanks for the detail! will definitely take this to eng team for process discussion.

Thank you so much for the details! This is very helpful. I will pass this along to my Jira Perf colleagues (there's multiple of them, since they know Perf is such a big issue).

Just to clarify on Search though, which search are you talking about:

a) quick search (top bar) b) issue search (the one with the basic/JQL switcher) c) something else

Trying to narrow down the latter "even if clicking into any of them is not" part to understand which view that is

Hi BillinghamJ,

You're right, I apologize for not being clear. We're targeting 1s for "Initial loads" on new tabs/new navigation, which I assume you're referring to. Our target for 'transitions' is different.

If however the numbers you're referring to are "initial load" numbers, then I'm not sure.

(edit: and action responses again are also a separate category. Our largest number of complaints are about 'page load times' in Confluence, so most conversations center around that)

Hi lostdog,

Thanks for the understanding! Indeed I haven't been at Atlassian that long, but that's not a good excuse: it's my problem to own.

I appreciate the reinforcement of "fix everything", and I assure you we're trying our best to do so. As a PM it is my natural instinct (and literal job) to prioritize, so I'm always looking for more details to do so.

I can understand that my request for details can imply that I'm either not listening or not believing the feedback, but that is not the case -> I do understand everything is slow and needs fixing.

As recommended by another poster to take advantage of the technical community here, I have one question and one comment if you can provide more insights:

question a) My understanding is that performance numbers fluctuate a LOT, even at sampling in the tens of thousands. Do you have any recommendations of tools or methods to reduce this variance?

comment b) we're definitely trying to do this but we're not there yet - most of our metrics don't meet goals we set. Instead the blocking goals must be 'don't make it any worse', which is doable -> but it doesn't necessarily make anything better yet (thus all the questions about what is most annoying that we can fix first).

Hopefully point (b) is clear - I'm not saying "our performance is great/good/acceptable", just the best I can do (as a PM) is try to figure out what to prioritize to fix.

Hi michaelt,

Thank you for the numbers -> I agree these are slow, and I can guarantee you that the Jira team is working on it (though I can't talk about details). These numbers are definitely outside of the goals.

I appreciate the call out of "page to complete rendering and the progress bar at the top of the screen to disappear" and "until the issue, comment and buttons have all loaded". In a dream world of course, everything would load in < 1s (everything drawn, everything interactive), but working our way down to that will take time.

We're currently looking at each use case to understand the '(a) paint faster vs (b) interactive faster' tradeoff and trying to decide which cases the user has a better experience with (a) or (b). In Confluence this is clearer in some places than in others, but in Jira it's less clear I think (I work on Confluence, I probably shouldn't speak for Jira specifics).

It always comes down to a limitation of resources though, which is why we're always hoping to get as specific feedback as possible.

Thanks for the detail! Point (1) is very interesting

on (2) -> it sounded like you're saying (a) 'across entire suite identically is good', but that also (b) 'if there's a clear separation of Dev vs nonDev apps/docs, having different support is okay', did I read that right? please correct me if wrong.

Thanks! I'm not sure this is the most doable thing (I'm not familiar with the technical aspects of the editor storage format) but can definitely discuss with that team.

The "reasonable subset of Markdown" is also a very useful specific detail, exactly the kind of specificity that helps us do our jobs.

Hi acdha,

We have metrics, but of course as with many such things you always want more insights than the amount of data you're collecting (so we're always trying to grow this as appropriate).

This data is what led to the above (added edit trying to reply to @rusticpenn) saying we can see that "some instances are slower than others", and "some users are slower than others". I can't share those numbers of course though.

However, privacy reasons does prevent us from collecting too much data, so differentiating why individual users might have different experiences (even when other known factors are similar/identical) is difficult.

Also I'd be happy to take any suggestions you have about what to look at back to my engineering team, if you're willing to share other ideas. I know we're tracking several of the ones you mention but more options is always better.

followup questions: what level of support are you looking for, and what would be sufficient?

1) markdown macro (limits markdown to the body content, and not interacting with other macros or styling)

2) copy/paste markdown -> autoconvert to WYSIWYG (limits markdown to copy/pasting, so no editing Markdown inside)

3) "markdown pages" (something like 'this page is Markdown only, no WYSIWYG)

I make no comments/promises on any of these becoming real but looking for what's most valuable

Hi ratww,

Thank you for reiterating this point, and I'll try to shed some light on this. We actually are working on systemic changes to try to make this lighter/better but I can't talk about specifics until the feature is available.

On the other hand, any level of specificity is great, for example: 1) full page loads are slower and more annoying than Transitions (or, vice versa) 2) loading Home page is slower and more annoying than Search Results (or, vice versa) 3) waiting for the editor to load is more annoying than X/Y/Z 4) etc....

Even systemic changes require individual work for applying to these different views, so any level of specific feedback would be helpful.

(also it looks like HN is limiting my reply rate so apologies for any slowness)

We're trying to collect and fix such occurrences, so if you have something with specific repro steps please send them to me and I'll make sure they get to the right team.

We have certainly heard from some customers that agree that 'everything' is slow, but we've also heard from other customers saying they have no problems.

We would love to fix "everything", and we have some longer term projects focused on this -> However, "everything" fixes seem to be a more incremental boost and also take longer time to complete.

If you have any feedback about "specific" items that are the most frustrating, we'd love to hear about those -> targeted fixes for specific can be much faster, much greater gains, and usually offer better user experience gain/engineering time returns.

If not, I can only say that we are definitely working on making 'everything faster'

(edit: trying to reply but looks like HN is limiting my reply rate)

(edit: maybe I can post my replies here and hopefully they'll get read)

------ @rusticpenn - This is definitely possible that 'some users are just used to it'. But we also see a very wide variance in individual customers' performance numbers (ie. some instances have consistently faster performance than other instances), and even within individual instances variance amongst users (some users have consistently faster experience than other users on the same instance) -> we're trying what we can to narrow down the causes in this variance.

Hearing from "users with slow experiences" is simply one of the ways we're trying to track this down, but it helps if users are willing to provide more info.

--------

@ratww - thank you for the suggestion! We have some amount of data that helps us see what might be different between instances, but haven't gone out of our way to 'interview a fast customer', I'll bring this up with the team to see.

The two biggest factors I think we've seen: slow machines can contribute (but not a necessity), and large pages (especially with large tables, or large number of tables) can contribute.

Is this also on Cloud products?

Right now on holiday (through Tuesday), but would like learn more. I see your contact info in your profile, if you give consent I'd like send you an email on Wednesday with some followup questions (the first question is "what is the URL for your cloud instance" so I don't want to be asking for it on a public forum)

(edit: typo)

Sorry to hear it's been a frustrating experience. I'm a PM for Confluence Cloud and we're always trying to make it better. Would you be willing to share more specifics, such as: - Pages with content X are the slowest - Trying to do A/B/C is annoyingly slow - etc ?

(edit: looks like HN is severely limiting my reply rate so apologies for delays)

We're trying to focus on frustrating pages/experiences rather than number of network calls and such, because while the latter is correlated, the former (frustrating slow experiences) is really the bar and the higher priority.

In terms of the ToS I'm not from legal so can't say (still looking into it), but have definitely had conversations with users on public forums about performance issues, and afaik no one has been accused of violating their ToS.

(edit: since I can't reply due to HN limits I'll try to add some stuff in this edit)

------- @plorkyeran "target things that are easier to fix than those with highest impact" -> this is a good point and something we're trying to do. Engineers know the former (easier to fix) pretty readily, but identifying "highest impact" requires some work, so I'm (as a PM) always trying to find out. It's of course some combination of these two (low hanging fruit, high impact items) that forms the priority list.

------ @igetspam (moved followup into a reply to trigger notification)

------@core-questions "perf to take over company for 6mo-1yr" I'm not in the position to make that level of decisions, but can certainly pass the feedback up the chain. The perf team is trying their best though, so any info anyone can provide us can help us apply our resources in the right place

Sorry to hear it's been a frustrating experience. I'm a PM for Confluence Cloud and we're always trying to make it better. Would you be willing to share more specifics, such as: - Pages with content X are the slowest - Trying to do A/B/C is annoyingly slow - etc ?

(edit: looks like HN is severely limiting my reply rate so apologies for delays)