Please help me spread the link to #swad

https://github.com/Zirias/swad

I really need some users by now, for those two reasons:

* I'm at a point where I fully covered my own needs (the reasons I started coding this), and getting some users is the only way to learn about what other people might need
* The complexity "exploded" after supporting so many OS-specific APIs (like #kqueue, #epoll, #eventfd, #signalfd, #timerfd, #eventports) and several #lockfree implementations based on #atomics while still providing fallbacks for everything that *should* work on any #POSIX systems ... I'm definitely unable at this point to think of every possible edge case and test it. If there are #bugs left (which is somewhat likely), I really need people reporting these to me

Thanks!

I now implemented a per-thread #pool to reuse #timer objects in #poser (my lib I use for #swad).

The great news is: This improved performance, which is an unintended side effect (my goal was to reduce RAM usage ). I tested with the #kqueue backend on #FreeBSD and sure, this makes sense: So far, I needed to keep a list of destroyed timers that's always checked to solve an interesting issue: By the time I cancel a timer with #kevent, the expiry event might already be queued, but not yet read by my event loop. Trying to fire events from a timer that doesn't exist any more would segtfault of course. Not necessary any more with the pool approach, the timer WILL exist and I can just check whether it's "alive".

The result? Same hardware as always, and now swad reaches a throughput of 26000 requests per second with (almost) perfect response times.

I'm still not happy with memory usage. It's better, and I have no explanation for what I oberved now:

Ran the same test 3 times, 1000 #jmeter threads each simulating a distinct client running a loop for 2000 times doing one GET and one POST for a total of 4 million requests. After the first time, the resident set was at 178MiB. After the second time, 245 MiB. And after the third time, well, 245 MiB. How ...?

Also, there's another weird observation I have no explanation for. My main thread delegates accepted connections to worker threads simply "round robin". And each time I run the jmeter test, all these worker threads show increasing CPU usage at a similar rate, until suddenly, one single thread seems to do "more work", which stabilizes when this thread is utilizing almost double the CPU as all other worker threads. And when I run the jmeter test again (NOT restarting swad), the same happens again, but this time, it's a *different* thread that "works" a lot more than all others.

I wonder whether I should accept scheduling, memory management etc. pp are all "black magic" and swad is probably "good enough" as is right now.

Getting somewhat closer to releasing a new version of #swad. I now improved the functionality to execute something on a different worker thread: Use an in-memory queue, providing a #lockfree version. This gives me a consistent reliable throughput of 3000 requests/s (with outliers up to 4500 r/s) at an average response time of 350 - 400 ms (with TLS enabled). For waking up worker threads, I implemented different backends as well: kqueue, eventfd and event-ports, the fallback is still a self-pipe.

So, #portability here really means implement lots of different flavors of the same thing.

Looking at these startup logs, you can see that #kqueue (#FreeBSD and other BSDs) is really a "jack of all trades", being used for "everything" if available (and that's pretty awesome, it means one single #syscall per event loop iteration in the generic case). #illumos' (#Solaris) #eventports come somewhat close (but need a lot more syscalls as there's no "batch registering" and certain event types need to be re-registered every time they fired), they just can't do signals, but illumos offers Linux-compatible signalfd. Looking at #Linux, there's a "special case fd" for everything. Plus #epoll also needs one syscall for each event to be registered. The "generic #POSIX" case without any of these interfaces is just added for completeness

Now that #swad 0.7 is released, it's time to prepare a new release of #poser, my own lib supporting #services on #POSIX systems, following a #reactor with #threadpool design.

During development of swad, I moved poser from using strictly only POSIX APIs (with the scalability limits of e.g. #select) to auto-detected support for #kqueue, #epoll, #eventports, #signalfd and #timerfd (so now it could, in theory(!), "compete" with e.g. libevent). I also fixed quite some hidden bugs, and added more base functionality, like a #dictionary using nested hashtables internally, or #async tasks mimicking the async/await pattern known from e.g, #csharp. I also deprecated two features, the periodic and global "service tick" (superseded by individual timers) and the "resolve hosts" property of a "connection" (superseded by a separate resolve class).

I'll have to decide on a few things, e.g. whether I'll remove the deprecated stuff immediately and bump the major version of the "posercore" lib. I guess I'll do just that. I'd also like to add all the web-specific stuff (http 1.0/1.1 server) that's currently part of the swad code as a "poserweb" lib. This would get a major version of 0, indicating a generally unstable API/ABI as of now....

And then, I'd have to decide where certain utility classes belong to. The rate limiter is probably useful for things other than web, so it should probably go to core. What about url encoding/decoding, for example?

Stay tuned, something will come here, maybe helping you to write a nice service in plain #C :

https://github.com/Zirias/poser

The next release of #swad will probably bring not a single new feature, but focus on improvements, especially regarding #performance. Support for using #kqueue (#FreeBSD et al) to handle #signals is a part of it (which is done and works). Still unsure whether I'll also add support for #Linux' #signalfd. Using kqueue also as a better backend for #timers is on the list.

Another hopefully quite relevant change is here:

https://github.com/Zirias/poser/commit/798f23547295f89fa0c751f0e707c3474b5c689c

In short, so far my #poser lib was always awaiting readiness notification (from kqueue, or #epoll on Linux, or select/poll for other platforms) before doing any read or write on a socket. This is the ideal approach for reads, because in the common case, a socket is NOT ready for reading ... our kernel must have received something from the remote end first. But for writes, it's not so ideal. The common case is that a socket IS ready to write (because there's space left in the kernel's send buffers). So, just try it, and only register for notifications if it ever fails, makes more sense. Avoids pointless waiting and pointless events, and e.g. with epoll, even unnecessary syscalls.

Hmm. Now that I have a working "generic" #signal handling in #poser, I'd like to optimize a bit by picking up @david_chisnall's suggestion to use #kqueue for the job if available. Would have a clear advantage: No need to fiddle with the signal mask around every call to #kevent.

I still had the doubt whether a signal delivered via kqueue would still remain pending when it is just blocked, so I wrote some little test code and the unfortunate answer is: yes. Unfortunate because I want my library code to restore everything as it was found (signal mask and handlers) on exit, but I certainly don't want a batch of spurious signals handled when unblocking them.

Kind of obvious solution: Set the signals temporarily to ignored when unblocking them, as shown in the screenshot. Now I have the next doubt: Is it guaranteed to have pending signals delivered instantly when unblocking them?

Still working on #swad, and currently very busy with improving quality, most of the actual work done inside my #poser library.

After finally supporting #kqueue and #epoll, I now integrated #xxhash to completely replace my previous stupid and naive hashing. I also added a more involved #dictionary class as an alternative to the already existing #hashtable. While the hashtable's size must be pre-configured and collissions are only ever resolved by storing linked lists, the new dictionary dynamically nests multiple hashtables (using different bits of a single hash value). I hope to achieve acceptable scaling while maintaining also acceptable memory overhead that way ...

#swad already uses both container classes as appropriate.

Next I'll probably revisit poser's #threadpool. I think I could replace #pthread condition variables by "simple" #semaphores, which should also reduce overhead ...

https://github.com/Zirias/swad

@pertho The only little drawback compared to epoll is the lack of atomic signal mask setting, so you need a bit more code and a thoughtful structure to handle signals in the same loop. Apart from that, indeed much better than #epoll.

Unfortunately, it's not beter than inotify (for a completely different purpose, #file #monitoring ... kqueue covers them all). With #inotify, you can for example set a #watch by path, while #kqueue requires opened file descriptors.