All Boards etc pagination? (was: Scalability / Performance) #3788
Milestone
Comments
|
Just to document: For the moment I moved the mongo Don't know which index should help yet. |
|
Thanks! I added some info to Scaling. It is also at https://wekan.github.io / Download / Scaling (green button). And I also added it to wiki below Features. |
|
As mongo states increasing the The real solution would be pagination or a global parameter to set a limit to 1000. Who wants to search a board out of 1000 on the screen? Search tool is there for that. So a global parameter limitation could be a first approach easier than pagination. |
|
Some alternatives, kind of related: a) Pagination b) Infinite scroll c) Automatic categorization + Pagination, based on board color, etc, with tabs/tree/other categorization method #2796 #641 d) Swimlanes for All Boards page #2796 #641 e) Search board name and board description f) Some other ways, but not preferred to do those, because some features could be lost. |
|
Today, I'm looking for tools to profile wekan mongo. Pages rendering is too slow despite proc/memory/IO is all right (8 Vcores, 8 Go, 40 Go SSD). Mongo process is often 100% but I believe on one core only. Cards table is 1.5 Go, but I think, it's not involved. My page is slow to render a lot before any request are done to Cards. And while on page all the rest is fluent. I'm quite sure this it's slow because of my Board table. Board table is 80 Mo. If I set I'm rather new to Mongo. So any advices and tools are welcome. Here are the logs observed between the time I call the link and the one I get the board. I believe some 'conn' are not linked to my page loading but just to background requests. 20 secs is the classic time I need to see the board. Sometime, I even wonder, if it's not a kind of time out. I also see regular Regards, |
|
See #3345 (comment) and 9db3c93 |
|
Also see https://github.com/wekan/wekan/wiki/Debugging about debugging related stuff. Not everything is added to Wekan, and Wekan does not use Apollo or GraphQL, I prefer current Wekan REST API. |
|
Usually with any new optimization added to Wekan, it speeds up a lot. |
|
Great ! I was on https://github.com/wekan/wekan/wiki/Logs. |
|
For MongoDB GUI, there is also DBGate https://github.com/wekan/wekan/wiki/Backup#dbgate-open-source-mongodb-gui , you can test does it have some useful features. |
|
You see current indexes with: |
|
Here we are completely stuck. It already arrived yesterday. And this time, I think it's linked with db.cfs.attachments.filerecord. Suddenly, while working directly on the db, wekan go away. node process is still there, but nothing is listening on port 3001 anymore. The only thing we see in the log While restarting we have: When it arrived yesterday, the only solution we found was I don't know why mongo/meteor/wekan (I'm lost) keeps requesting this Regards, |
|
Meteor web framework does listen to changes at MongoDB and reacts to them immediately. For example, if some new card is added with REST API or directly to database, it will be visible immediately. |
|
@xet7 |
|
Currently like poll and diff: Currently Wekan still uses MongoDB 3.2.22: And it's started here: Snap also still uses Ubuntu 16.04 as base. I have some test files to try to sometime get changes to Snap Ubuntu 20.04 base: I have sometimes tried to get Snap to user newer Ubuntu base and MongoDB but I have not got them working yet. Docker uses newer Ubuntu base, and can use newest MongoDB 4.x. Wekan does not have oplog set yet, using oplog would be faster. If oplog would be used, For example in Snap package:
Even faster alternative would be to use Redis Oplog, although that would require also installing Redis: To send those database changes to users, Meteor uses DDP / Websockets. (Using websockets is faster than polling. If someone has restrictive F5 firewall, it could deny websocket traffic, those kind of environments prefer other polling based solutions). Some of these are also listed at Performance section of Awesome Meteor: Alternative to current Meteor Publications publish/subscribe for reactivity would be Meteor Methods this way: About inotify for checking filesystem changes: It's not so good on Windows filesystem, it can easily cause file and filesystem corruption, that I did notice when coding some inotify Go code some years ago. Using some atomic transaction database like SQLite or some multiuser database would be better for concurrent access. |
|
For "multitasking" of with Javascript and node, Meteor currently uses fibers: Sometime in future Meteor will migrate from fibers to async-await. Usually porting Wekan to other CPU is just building fibers on that platform: Wekan uses async-await for for some parts already, like here where Wekan page are combined to one .zip file for Wekan board static HTML page download: https://github.com/wekan/wekan/blob/master/client/lib/exportHTML.js#L15 |
|
@xet7 I will read and think about all this information, and without reading them all now, i think polling isnt the best strategy, the more queries which are watched for polling the more Meteor and MongoDB needs resources like CPU. But now, i will read all the informations first 🙂 |
|
I added it to here https://github.com/wekan/wekan/wiki/Developer-Documentation#stories-about-how-wekan-works-and-developing-wekan to first topic "How realtime board updates work". |
|
Storing attachments in database causes high RAM consumption etc: |
|
Both Wekan and RocketChat are made with Meteor and MongoDB. |
|
About storing attachments in filesystem: |
|
@xet7 |
|
At this article https://forums.rocket.chat/t/high-ram-consumption-on-ubuntu/11095/3 is mentioned this repo https://github.com/arminfelder/gridfsmigrate for RocketChat, but it has not been tested with Wekan. At Wekan v5.27 with this commit I did fork non-maintained CollectionFS repos to be at
|
|
Currently storing files in filesystem works this way, if I got info from some email correctly:
|
|
About that https://github.com/zcfs , how I forked it to wekan:
In those CollectionFS changes was already previous fixes that it could upload and download non-ASCII filename files. It's just that those changes were not released to packages at https://atmospherejs.com . |
|
Really, really, really interesting. We have 250 k attachments (representing 50 Go). This could explain a lot. We are not on Ubuntu but I believe in this context Debian Buster is the same. We have installed last MongoDB from the mongo community. Wekan snap comes with its own mongodb, but through snap we can set another mongodb port. If we duplicate the data. Would using a newer mongodb be as simple as changing this port? Reading your post, I've got the feeling it's not so simple. Other idea, is this behaviour fully intricate in the code or could it be easily disable by a setting. It's a wonderful feature and I understand it better now. I've seen Wekan was reactive to everybody actions on the same board but without imagining the constraints. I would love to have such a feature but in the meanwhile could it be disabled? We have one board for each prospect and there are no reason for operators to share the same board at any time. What we like with wekan is it's fast and easy to add a new card but we don't really use its collaborative feature for our CRM purpose. By the way, we have another instance of wekan that we really use in a collaborative way. Right now, I have:
Regards, |
|
Could you remind us about the reason to choose CollectionFS? What about Meteor-Files? Is there one better that the other? Is it just history? I found this regarding Meteor-Files to avoid deletion: https://github.com/VeliovGroup/Meteor-Files/blob/master/docs/constructor.md#use-onbeforeremove-to-avoid-unauthorized-remove. I suppose it's about the same with cfs. Regards, |
|
Sorry, for the noise. Onbeforeremove is the right place to check permission but it doesn't help you split delete db entries and unlink file on the FS. |
|
I added summary about CollectionFS and Meteor-Files here #3273 (comment) |
|
From @e-gaulue
Wekan can be run without Snap. Wekan only requires node+mongodb. Prebuilt bundle info is at https://github.com/wekan/wekan/Raspberry-Pi (includes info about startup script and systemd service). Or alternatively like at https://github.com/wekan/wekan/wiki/Offline . If you would like to build that bundle from source code, info is at https://github.com/wekan/wekan/wiki/Emoji steps 1-4.With building, bundle is at That redis oplog https://github.com/cult-of-coders/redis-oplog can be installed with buiding Wekan, and then: Then building Wekan again, and running with start-wekan.sh or as service.
No, it can not be disabled.
You can have read-only public board.
Snap has MongoDB logs disabled https://github.com/wekan/wekan/blob/master/snap-src/bin/mongodb-control#L92 , to see any logs you should use non-Snap install of Wekan. Also, with #3345 (comment) I did mean you look at step 3 at Nosqlbooster database profiler, to see are there slow queries that take a lot of time to run. |
|
Thanks @xet7, I'm going to try most of this. We manage to compile wekan 3 weeks ago, so we may manage to handle it. Wekan code looks really readable, all you say is quite understandable, but it looks hard for me. I've no skill I can rely on coming from the admin sys + PHP-MySQL and server driven world. I have to read a lot, so it's slow. But I'm pretty sure there is no reason why a 3 Gb database couldn't be serve fluently by a 8 Gb server. I also know now, that client is also involved and we have to be careful to the amount of data send to it. I've got the felling wekan was rather fluent before we added those 250 k attachments. But we have to try again. Knowing we already created users (+/- 20) and custom values, we then create the 30 k boards, 30 k swimlanes and 30 k x 3 lists all associated to all our users. At this time, we have the message mentioned at the beginning of this issue regarding "sort out of memory", but wekan works rather fast (not as fast as the other instance we use for collaboration). Then we add 430 k cards representing 2 Go in mongo. And then we add 250 k attachments. We have to redo and see where wekan really hangs. Anyway I can't stand using technologies I can't master. So it's a useful time as long as my boss doesn't say STOP. |
I already read this, but as I could see logs from mongodb I tough it had changed since. |
|
About Wekan code, also see https://github.com/wekan/wekan/wiki/Directory-Structure |
|
According go OpenHub https://www.openhub.net/p/wekan , coding Wekan took an estimated combined 17 years of effort (COCOMO model) of 263 contributors. When adding some big feature, it has required some time to get it it to fit into Wekan properly, fixing related corner cases, and getting all features to play with each other properly. |
|
So, I rebuild wekan with cultofcoders:redis-oplog, then we start wekan on the locally installed MongoDB 4.4 => OK I stopped wekan, I move db data with I installed nosqlbooster and connected it to localhost:27017, and I've been to profiling. I changed But in the mongolog wekan keeps requesting for |
|
I see those query: and then plenty of batch 1000 Looks like Regards, |
|
Moved to #4791 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
Not really an issue, but a question. Is there a place with all configuration recommandations for scalability / performance ?
I read #3585, #3345. Today we saw in the log the "Sort operation used more than the maximum 33554432 bytes of RAM. Add an index, or specify a smaller limit." while wekan was searching something in boards.
We have presently (under snap) 30 k boards, 30 k swimlanes, 90 k lists, 1000 cards and 200 attachments (outside of Mongo). It looks big numbers, but in fact it not so much, few Mo db. We wan't to go to 150 k boards, 150 k swimlanes, 450 k lists, 600 k cards, 250 k attachments.
We will never need the all board page view. But it looks the request is thrown from somewhere else anyway.
First, I would just like to check if similar things already happened to anybody, hence the question.
Best regards,
The text was updated successfully, but these errors were encountered: