Mastodon

Felix Palmen :freebsd: :c64:<a href="https://mastodon.bsd.cafe/@jan" class="u-url mention" rel="nofollow noopener" target="_blank">@jan</a> funnily, the lack of <a href="https://mastodon.bsd.cafe/tags/POSIX" class="mention hashtag" rel="nofollow noopener" target="_blank">#POSIX</a> <a href="https://mastodon.bsd.cafe/tags/timers" class="mention hashtag" rel="nofollow noopener" target="_blank">#timers</a> in <a href="https://mastodon.bsd.cafe/tags/OpenBSD" class="mention hashtag" rel="nofollow noopener" target="_blank">#OpenBSD</a> inspired me to come up with a timer implementation supporting different backends. I was annoyed at first, but didn't regret it. The interface offered by POSIX timers is really clumsy, they can either send some signal or (even worse) launch a thread. My current code only uses them on <a href="https://mastodon.bsd.cafe/tags/illumos" class="mention hashtag" rel="nofollow noopener" target="_blank">#illumos</a>, which offers a better signaling mechanism on an "event port". Where available, <a href="https://mastodon.bsd.cafe/tags/kqueue" class="mention hashtag" rel="nofollow noopener" target="_blank">#kqueue</a> is used for timers. the next fallback is <a href="https://mastodon.bsd.cafe/tags/Linux" class="mention hashtag" rel="nofollow noopener" target="_blank">#Linux</a>' <a href="https://mastodon.bsd.cafe/tags/timerfd" class="mention hashtag" rel="nofollow noopener" target="_blank">#timerfd</a>. And finally, as a last resort, some manual multiplexing on top of <a href="https://mastodon.bsd.cafe/tags/setitimer" class="mention hashtag" rel="nofollow noopener" target="_blank">#setitimer</a> (cause it's not much worse than "vanilla" POSIX timers).tl;dr, might be an alternative to contribute code to upstream enabling them to use better platform interfaces for timers...

Felix Palmen :freebsd: :c64:Please help me spread the link to <a href="https://mastodon.bsd.cafe/tags/swad" class="mention hashtag" rel="nofollow noopener" target="_blank">#swad</a> 😎<a href="https://github.com/Zirias/swad" rel="nofollow noopener" translate="no" target="_blank">https://github.com/Zirias/swad</a>I really need some users by now, for those two reasons:* I'm at a point where I fully covered my own needs (the reasons I started coding this), and getting some users is the only way to learn about what other people might need * The complexity "exploded" after supporting so many OS-specific APIs (like <a href="https://mastodon.bsd.cafe/tags/kqueue" class="mention hashtag" rel="nofollow noopener" target="_blank">#kqueue</a>, <a href="https://mastodon.bsd.cafe/tags/epoll" class="mention hashtag" rel="nofollow noopener" target="_blank">#epoll</a>, <a href="https://mastodon.bsd.cafe/tags/eventfd" class="mention hashtag" rel="nofollow noopener" target="_blank">#eventfd</a>, <a href="https://mastodon.bsd.cafe/tags/signalfd" class="mention hashtag" rel="nofollow noopener" target="_blank">#signalfd</a>, <a href="https://mastodon.bsd.cafe/tags/timerfd" class="mention hashtag" rel="nofollow noopener" target="_blank">#timerfd</a>, <a href="https://mastodon.bsd.cafe/tags/eventports" class="mention hashtag" rel="nofollow noopener" target="_blank">#eventports</a>) and several <a href="https://mastodon.bsd.cafe/tags/lockfree" class="mention hashtag" rel="nofollow noopener" target="_blank">#lockfree</a> implementations based on <a href="https://mastodon.bsd.cafe/tags/atomics" class="mention hashtag" rel="nofollow noopener" target="_blank">#atomics</a> while still providing fallbacks for everything that *should* work on any <a href="https://mastodon.bsd.cafe/tags/POSIX" class="mention hashtag" rel="nofollow noopener" target="_blank">#POSIX</a> systems ... I'm definitely unable at this point to think of every possible edge case and test it. If there are <a href="https://mastodon.bsd.cafe/tags/bugs" class="mention hashtag" rel="nofollow noopener" target="_blank">#bugs</a> left (which is somewhat likely), I really need people reporting these to meThanks! 🙃

Felix Palmen :freebsd: :c64:I now implemented a per-thread <a href="https://mastodon.bsd.cafe/tags/pool" class="mention hashtag" rel="nofollow noopener" target="_blank">#pool</a> to reuse <a href="https://mastodon.bsd.cafe/tags/timer" class="mention hashtag" rel="nofollow noopener" target="_blank">#timer</a> objects in <a href="https://mastodon.bsd.cafe/tags/poser" class="mention hashtag" rel="nofollow noopener" target="_blank">#poser</a> (my lib I use for <a href="https://mastodon.bsd.cafe/tags/swad" class="mention hashtag" rel="nofollow noopener" target="_blank">#swad</a>).The great news is: This improved performance, which is an unintended side effect (my goal was to reduce RAM usage 🙈😆). I tested with the <a href="https://mastodon.bsd.cafe/tags/kqueue" class="mention hashtag" rel="nofollow noopener" target="_blank">#kqueue</a> backend on <a href="https://mastodon.bsd.cafe/tags/FreeBSD" class="mention hashtag" rel="nofollow noopener" target="_blank">#FreeBSD</a> and sure, this makes sense: So far, I needed to keep a list of destroyed timers that's always checked to solve an interesting issue: By the time I cancel a timer with <a href="https://mastodon.bsd.cafe/tags/kevent" class="mention hashtag" rel="nofollow noopener" target="_blank">#kevent</a>, the expiry event might already be queued, but not yet read by my event loop. Trying to fire events from a timer that doesn't exist any more would segtfault of course. Not necessary any more with the pool approach, the timer WILL exist and I can just check whether it's "alive".The result? Same hardware as always, and now swad reaches a throughput of 26000 requests per second with (almost) perfect response times. 🥳 I'm still not happy with memory usage. It's better, and I have no explanation for what I oberved now:Ran the same test 3 times, 1000 <a href="https://mastodon.bsd.cafe/tags/jmeter" class="mention hashtag" rel="nofollow noopener" target="_blank">#jmeter</a> threads each simulating a distinct client running a loop for 2000 times doing one GET and one POST for a total of 4 million requests. After the first time, the resident set was at 178MiB. After the second time, 245 MiB. And after the third time, well, 245 MiB. How ...? 🤯 Also, there's another weird observation I have no explanation for. My main thread delegates accepted connections to worker threads simply "round robin". And each time I run the jmeter test, all these worker threads show increasing CPU usage at a similar rate, until suddenly, one single thread seems to do "more work", which stabilizes when this thread is utilizing almost double the CPU as all other worker threads. And when I run the jmeter test again (NOT restarting swad), the same happens again, but this time, it's a *different* thread that "works" a lot more than all others.I wonder whether I should accept scheduling, memory management etc. pp are all "black magic" and swad is probably "good enough" as is right now. 😆

Felix Palmen :freebsd: :c64:Getting somewhat closer to releasing a new version of <a href="https://mastodon.bsd.cafe/tags/swad" class="mention hashtag" rel="nofollow noopener" target="_blank">#swad</a>. I now improved the functionality to execute something on a different worker thread: Use an in-memory queue, providing a <a href="https://mastodon.bsd.cafe/tags/lockfree" class="mention hashtag" rel="nofollow noopener" target="_blank">#lockfree</a> version. This gives me a consistent reliable throughput of 3000 requests/s (with outliers up to 4500 r/s) at an average response time of 350 - 400 ms (with TLS enabled). For waking up worker threads, I implemented different backends as well: kqueue, eventfd and event-ports, the fallback is still a self-pipe.So, <a href="https://mastodon.bsd.cafe/tags/portability" class="mention hashtag" rel="nofollow noopener" target="_blank">#portability</a> here really means implement lots of different flavors of the same thing.Looking at these startup logs, you can see that <a href="https://mastodon.bsd.cafe/tags/kqueue" class="mention hashtag" rel="nofollow noopener" target="_blank">#kqueue</a> (<a href="https://mastodon.bsd.cafe/tags/FreeBSD" class="mention hashtag" rel="nofollow noopener" target="_blank">#FreeBSD</a> and other BSDs) is really a "jack of all trades", being used for "everything" if available (and that's pretty awesome, it means one single <a href="https://mastodon.bsd.cafe/tags/syscall" class="mention hashtag" rel="nofollow noopener" target="_blank">#syscall</a> per event loop iteration in the generic case). <a href="https://mastodon.bsd.cafe/tags/illumos" class="mention hashtag" rel="nofollow noopener" target="_blank">#illumos</a>' (<a href="https://mastodon.bsd.cafe/tags/Solaris" class="mention hashtag" rel="nofollow noopener" target="_blank">#Solaris</a>) <a href="https://mastodon.bsd.cafe/tags/eventports" class="mention hashtag" rel="nofollow noopener" target="_blank">#eventports</a> come somewhat close (but need a lot more syscalls as there's no "batch registering" and certain event types need to be re-registered every time they fired), they just can't do signals, but illumos offers Linux-compatible signalfd. Looking at <a href="https://mastodon.bsd.cafe/tags/Linux" class="mention hashtag" rel="nofollow noopener" target="_blank">#Linux</a>, there's a "special case fd" for everything. 🙈 Plus <a href="https://mastodon.bsd.cafe/tags/epoll" class="mention hashtag" rel="nofollow noopener" target="_blank">#epoll</a> also needs one syscall for each event to be registered. The "generic <a href="https://mastodon.bsd.cafe/tags/POSIX" class="mention hashtag" rel="nofollow noopener" target="_blank">#POSIX</a>" case without any of these interfaces is just added for completeness 😆

JdeBP<a href="https://bsd.network/@meka" class="u-url mention" rel="nofollow noopener" target="_blank">@meka</a> One thing not strictly select()-related:EVFILT_PROC with NOTE_TRACK is another gotcha. It looks great at first glance.Until one really hammers it with lots of short-lived processes in a deep tree.Then it becomes apparent that NOTE_EXIT and NOTE_CHILD both use the data field for different purposes, but get merged, and grandchildren get lost from their parents.<a href="https://news.ycombinator.com/item?id=23549116" rel="nofollow noopener" translate="no" target="_blank">https://news.ycombinator.com/item?id=23549116</a><a href="https://tty0.social/tags/FreeBSD" class="mention hashtag" rel="nofollow noopener" target="_blank">#FreeBSD</a> <a href="https://tty0.social/tags/kqueue" class="mention hashtag" rel="nofollow noopener" target="_blank">#kqueue</a>

JdeBP<a href="https://bsd.network/@meka" class="u-url mention" rel="nofollow noopener" target="_blank">@meka</a> I'm currently working with an old kernel, and I don't know off the top of my head if/when this got fixed. But one of the things is that attempting to add an EVFILT_READ for a file descriptor that is /dev/null results/resulted in an ENODEV EV_ERROR event.It's generally edge case devices that didn't get attention because a lot of the relevant software still used select() and so no-one (apart from kevent()-everywhere people like me) really hit this.<a href="https://tty0.social/tags/FreeBSD" class="mention hashtag" rel="nofollow noopener" target="_blank">#FreeBSD</a> <a href="https://tty0.social/tags/kqueue" class="mention hashtag" rel="nofollow noopener" target="_blank">#kqueue</a>

JdeBP<a href="https://bsd.network/@meka" class="u-url mention" rel="nofollow noopener" target="_blank">@meka</a> It is always welcome to see more kevent(), if only because it lets other people share my pain, in the hope that that increases the push for kevent() to be fully completed and as good as select().There are a number of cases I have hit over the years kevent() cannot *quite* do what select() does.<a href="https://tty0.social/tags/FreeBSD" class="mention hashtag" rel="nofollow noopener" target="_blank">#FreeBSD</a> <a href="https://tty0.social/tags/kqueue" class="mention hashtag" rel="nofollow noopener" target="_blank">#kqueue</a>

Felix Palmen :freebsd: :c64:<a href="https://mastodon.bsd.cafe/@jhx" class="u-url mention" rel="nofollow noopener" target="_blank">@jhx</a> Regarding that, at least in theory, it's indeed "truly portable" as it works fine using only <a href="https://mastodon.bsd.cafe/tags/POSIX" class="mention hashtag" rel="nofollow noopener" target="_blank">#POSIX</a> compliant APIs.In practice, there can be issues with platforms that don't implement the *full* POSIX feature-set (which is in fact most platforms nowadays). There can also be nasty issues with how feature-test macros are handled (set by the compiler, interpreted by the system's headers) and sometimes with which libraries are needed (unfortunately, POSIX doesn't specify that, e.g. on illumos, you have to link a libsocket for any socket functionality 🙄).Once I started to add optional support for the platform-specific mechanisms <a href="https://mastodon.bsd.cafe/tags/epoll" class="mention hashtag" rel="nofollow noopener" target="_blank">#epoll</a> on <a href="https://mastodon.bsd.cafe/tags/Linux" class="mention hashtag" rel="nofollow noopener" target="_blank">#Linux</a> and <a href="https://mastodon.bsd.cafe/tags/kqueue" class="mention hashtag" rel="nofollow noopener" target="_blank">#kqueue</a> on <a href="https://mastodon.bsd.cafe/tags/BSD" class="mention hashtag" rel="nofollow noopener" target="_blank">#BSD</a> (because the POSIX standard select and poll have severe scalability issues), I wanted to also add support for /dev/poll as used on solaris, that's why I installed <a href="https://mastodon.bsd.cafe/tags/OpenIndiana" class="mention hashtag" rel="nofollow noopener" target="_blank">#OpenIndiana</a> (illumos-based) in a VM to do tests, and I quickly learned /dev/poll was superseded by "event ports", so that's what I added instead.

FreeBSDA Programmer-Friendly I/O Abstraction Over io_uring and kqueue This is an external post of mine. ... <a href="http://notes.eatonphil.com/a-friendly-abstraction-over-iouring-and-kqueue.html" rel="nofollow noopener" target="_blank">http://notes.eatonphil.com/a-friendly-abstraction-over-iouring-and-kqueue.html</a> <a rel="nofollow noopener" class="mention hashtag" href="https://mastodon.social/tags/databases" target="_blank">#databases</a> <a rel="nofollow noopener" class="mention hashtag" href="https://mastodon.social/tags/zig" target="_blank">#zig</a> <a rel="nofollow noopener" class="mention hashtag" href="https://mastodon.social/tags/io_uring" target="_blank">#io_uring</a> <a rel="nofollow noopener" class="mention hashtag" href="https://mastodon.social/tags/kqueue" target="_blank">#kqueue</a> <a rel="nofollow noopener" class="mention hashtag" href="https://mastodon.social/tags/linux" target="_blank">#linux</a> <a rel="nofollow noopener" class="mention hashtag" href="https://mastodon.social/tags/freebsd" target="_blank">#freebsd</a> <a rel="nofollow noopener" class="mention hashtag" href="https://mastodon.social/tags/darwin" target="_blank">#darwin</a> <a rel="nofollow noopener" class="mention hashtag" href="https://mastodon.social/tags/tigerbeetle" target="_blank">#tigerbeetle</a> <a rel="nofollow noopener" class="mention hashtag" href="https://mastodon.social/tags/external" target="_blank">#external</a> <a href="https://awakari.com/pub-msg.html?id=D9FNqhGgL0wIdlVDEjw3HD4xST2&interestId=FreeBSD" rel="nofollow noopener" target="_blank">Result Details</a>

Felix Palmen :freebsd: :c64:Unfortunately, I had to do a bugfix release: <a href="https://mastodon.bsd.cafe/tags/swad" class="mention hashtag" rel="nofollow noopener" target="_blank">#swad</a> 0.8Although I didn't observe any obvious misbehavior on my own installation for several days, I discovered two very relevant bugs just after release of 0.7 🤦‍♂️ -- one of them (only affecting <a href="https://mastodon.bsd.cafe/tags/kqueue" class="mention hashtag" rel="nofollow noopener" target="_blank">#kqueue</a>, for example on <a href="https://mastodon.bsd.cafe/tags/FreeBSD" class="mention hashtag" rel="nofollow noopener" target="_blank">#FreeBSD</a>) even critical because it could trigger "undefined behavior".Both bugs are regressions from new (performance) improvements added, one from trying to queue as many writes as possible when sending HTTP responses, one from using kqueue to provide timers and signals.See release notes for 0.8. Don't use 0.7. Sorry 🤪 <a href="https://github.com/Zirias/swad" rel="nofollow noopener" translate="no" target="_blank">https://github.com/Zirias/swad</a>

Felix Palmen :freebsd: :c64:Now that <a href="https://mastodon.bsd.cafe/tags/swad" class="mention hashtag" rel="nofollow noopener" target="_blank">#swad</a> 0.7 is released, it's time to prepare a new release of <a href="https://mastodon.bsd.cafe/tags/poser" class="mention hashtag" rel="nofollow noopener" target="_blank">#poser</a>, my own lib supporting <a href="https://mastodon.bsd.cafe/tags/services" class="mention hashtag" rel="nofollow noopener" target="_blank">#services</a> on <a href="https://mastodon.bsd.cafe/tags/POSIX" class="mention hashtag" rel="nofollow noopener" target="_blank">#POSIX</a> systems, following a <a href="https://mastodon.bsd.cafe/tags/reactor" class="mention hashtag" rel="nofollow noopener" target="_blank">#reactor</a> with <a href="https://mastodon.bsd.cafe/tags/threadpool" class="mention hashtag" rel="nofollow noopener" target="_blank">#threadpool</a> design.During development of swad, I moved poser from using strictly only POSIX APIs (with the scalability limits of e.g. <a href="https://mastodon.bsd.cafe/tags/select" class="mention hashtag" rel="nofollow noopener" target="_blank">#select</a>) to auto-detected support for <a href="https://mastodon.bsd.cafe/tags/kqueue" class="mention hashtag" rel="nofollow noopener" target="_blank">#kqueue</a>, <a href="https://mastodon.bsd.cafe/tags/epoll" class="mention hashtag" rel="nofollow noopener" target="_blank">#epoll</a>, <a href="https://mastodon.bsd.cafe/tags/eventports" class="mention hashtag" rel="nofollow noopener" target="_blank">#eventports</a>, <a href="https://mastodon.bsd.cafe/tags/signalfd" class="mention hashtag" rel="nofollow noopener" target="_blank">#signalfd</a> and <a href="https://mastodon.bsd.cafe/tags/timerfd" class="mention hashtag" rel="nofollow noopener" target="_blank">#timerfd</a> (so now it could, in theory(!), "compete" with e.g. libevent). I also fixed quite some hidden bugs, and added more base functionality, like a <a href="https://mastodon.bsd.cafe/tags/dictionary" class="mention hashtag" rel="nofollow noopener" target="_blank">#dictionary</a> using nested hashtables internally, or <a href="https://mastodon.bsd.cafe/tags/async" class="mention hashtag" rel="nofollow noopener" target="_blank">#async</a> tasks mimicking the async/await pattern known from e.g, <a href="https://mastodon.bsd.cafe/tags/csharp" class="mention hashtag" rel="nofollow noopener" target="_blank">#csharp</a>. I also deprecated two features, the periodic and global "service tick" (superseded by individual timers) and the "resolve hosts" property of a "connection" (superseded by a separate resolve class).I'll have to decide on a few things, e.g. whether I'll remove the deprecated stuff immediately and bump the major version of the "posercore" lib. I guess I'll do just that. I'd also like to add all the web-specific stuff (http 1.0/1.1 server) that's currently part of the swad code as a "poserweb" lib. This would get a major version of 0, indicating a generally unstable API/ABI as of now....And then, I'd have to decide where certain utility classes belong to. The rate limiter is probably useful for things other than web, so it should probably go to core. What about url encoding/decoding, for example? 🤔Stay tuned, something will come here, maybe helping you to write a nice service in plain <a href="https://mastodon.bsd.cafe/tags/C" class="mention hashtag" rel="nofollow noopener" target="_blank">#C</a> 😎:<a href="https://github.com/Zirias/poser" rel="nofollow noopener" translate="no" target="_blank">https://github.com/Zirias/poser</a>

Felix Palmen :freebsd: :c64:<a href="https://chaos.social/@Toasterson" class="u-url mention" rel="nofollow noopener" target="_blank">@Toasterson</a> <a href="https://mastodon.social/@astade" class="u-url mention" rel="nofollow noopener" target="_blank">@astade</a> I'm downloading an <a href="https://mastodon.bsd.cafe/tags/OpenIndiana" class="mention hashtag" rel="nofollow noopener" target="_blank">#OpenIndiana</a> image right now, we will see.For Linux, I now have support for <a href="https://mastodon.bsd.cafe/tags/epoll" class="mention hashtag" rel="nofollow noopener" target="_blank">#epoll</a> (fd events), and additionally <a href="https://mastodon.bsd.cafe/tags/signalfd" class="mention hashtag" rel="nofollow noopener" target="_blank">#signalfd</a> and <a href="https://mastodon.bsd.cafe/tags/timerfd" class="mention hashtag" rel="nofollow noopener" target="_blank">#timerfd</a>. On the BSD's that have <a href="https://mastodon.bsd.cafe/tags/kqueue" class="mention hashtag" rel="nofollow noopener" target="_blank">#kqueue</a>, it covers all these aspects. So, little question here: Are there any solaris / <a href="https://mastodon.bsd.cafe/tags/illumos" class="mention hashtag" rel="nofollow noopener" target="_blank">#illumos</a> APIs specifically for signal handling and individual timers I should look into? Or should I stick to classic async handlers (signals) and setitimer (timers) on that platform, and just have a look at /dev/poll?

Felix Palmen :freebsd: :c64:<a href="https://chaos.social/@Toasterson" class="u-url mention" rel="nofollow noopener" target="_blank">@Toasterson</a> <a href="https://mastodon.social/@astade" class="u-url mention" rel="nofollow noopener" target="_blank">@astade</a> Yeah, my main use case is socket notification, the lib offers <a href="https://mastodon.bsd.cafe/tags/select" class="mention hashtag" rel="nofollow noopener" target="_blank">#select</a> and <a href="https://mastodon.bsd.cafe/tags/poll" class="mention hashtag" rel="nofollow noopener" target="_blank">#poll</a> as a base working across <a href="https://mastodon.bsd.cafe/tags/POSIX" class="mention hashtag" rel="nofollow noopener" target="_blank">#POSIX</a>, and I added <a href="https://mastodon.bsd.cafe/tags/epoll" class="mention hashtag" rel="nofollow noopener" target="_blank">#epoll</a> for Linux and <a href="https://mastodon.bsd.cafe/tags/kqueue" class="mention hashtag" rel="nofollow noopener" target="_blank">#kqueue</a> for FreeBSD ... and as kqueue can do a lot more, I recently also implemented using it for signal handling.I normally prefer my own local infrastructure, already fixed my Linux VM yesterday to test thoroughly with epoll (and, MAYBE, add signalfd support). So I might just install e.g. <a href="https://mastodon.bsd.cafe/tags/OpenIndiana" class="mention hashtag" rel="nofollow noopener" target="_blank">#OpenIndiana</a> to also play around with /dev/poll. Had a look at an old SunOS manpage, seems simple enough to use 😉

Felix Palmen :freebsd: :c64:The next release of <a href="https://mastodon.bsd.cafe/tags/swad" class="mention hashtag" rel="nofollow noopener" target="_blank">#swad</a> will probably bring not a single new feature, but focus on improvements, especially regarding <a href="https://mastodon.bsd.cafe/tags/performance" class="mention hashtag" rel="nofollow noopener" target="_blank">#performance</a>. Support for using <a href="https://mastodon.bsd.cafe/tags/kqueue" class="mention hashtag" rel="nofollow noopener" target="_blank">#kqueue</a> (<a href="https://mastodon.bsd.cafe/tags/FreeBSD" class="mention hashtag" rel="nofollow noopener" target="_blank">#FreeBSD</a> et al) to handle <a href="https://mastodon.bsd.cafe/tags/signals" class="mention hashtag" rel="nofollow noopener" target="_blank">#signals</a> is a part of it (which is done and works). Still unsure whether I'll also add support for <a href="https://mastodon.bsd.cafe/tags/Linux" class="mention hashtag" rel="nofollow noopener" target="_blank">#Linux</a>' <a href="https://mastodon.bsd.cafe/tags/signalfd" class="mention hashtag" rel="nofollow noopener" target="_blank">#signalfd</a>. Using kqueue also as a better backend for <a href="https://mastodon.bsd.cafe/tags/timers" class="mention hashtag" rel="nofollow noopener" target="_blank">#timers</a> is on the list.Another hopefully quite relevant change is here:<a href="https://github.com/Zirias/poser/commit/798f23547295f89fa0c751f0e707c3474b5c689c" rel="nofollow noopener" translate="no" target="_blank">https://github.com/Zirias/poser/commit/798f23547295f89fa0c751f0e707c3474b5c689c</a>In short, so far my <a href="https://mastodon.bsd.cafe/tags/poser" class="mention hashtag" rel="nofollow noopener" target="_blank">#poser</a> lib was always awaiting readiness notification (from kqueue, or <a href="https://mastodon.bsd.cafe/tags/epoll" class="mention hashtag" rel="nofollow noopener" target="_blank">#epoll</a> on Linux, or select/poll for other platforms) before doing any read or write on a socket. This is the ideal approach for reads, because in the common case, a socket is NOT ready for reading ... our kernel must have received something from the remote end first. But for writes, it's not so ideal. The common case is that a socket IS ready to write (because there's space left in the kernel's send buffers). So, just try it, and only register for notifications if it ever fails, makes more sense. Avoids pointless waiting and pointless events, and e.g. with epoll, even unnecessary syscalls. 😉

Felix Palmen :freebsd: :c64:I think I'll "go <a href="https://mastodon.bsd.cafe/tags/kqueue" class="mention hashtag" rel="nofollow noopener" target="_blank">#kqueue</a> all the way" now, by also using it for providing my <a href="https://mastodon.bsd.cafe/tags/timers" class="mention hashtag" rel="nofollow noopener" target="_blank">#timers</a> if available. The current implementation multiplexes them on top of <a href="https://mastodon.bsd.cafe/tags/setitimer" class="mention hashtag" rel="nofollow noopener" target="_blank">#setitimer</a>, that's the portable variant that will stay as a fallback, but with kqueue, there's no need to have all these <a href="https://mastodon.bsd.cafe/tags/SIGALRM" class="mention hashtag" rel="nofollow noopener" target="_blank">#SIGALRM</a> generated, and I can avoid the slight imprecision by multiplexing in userspace 😎

Felix Palmen :freebsd: :c64:Ok, I forgot about restoring the previous state, cause I already broke this when implementing the generic signal handling via flags. And restoring everything to default must be good enough. Then, using <a href="https://mastodon.bsd.cafe/tags/kqueue" class="mention hashtag" rel="nofollow noopener" target="_blank">#kqueue</a> was kind of easy. Just this one weirdness that I'm not allowed to ignore <a href="https://mastodon.bsd.cafe/tags/SIGCHLD" class="mention hashtag" rel="nofollow noopener" target="_blank">#SIGCHLD</a>, so I have to block this instead ... anyone has any idea why?Still thinking whether I should also add support for <a href="https://mastodon.bsd.cafe/tags/signalfd" class="mention hashtag" rel="nofollow noopener" target="_blank">#signalfd</a> ... unfortunately different semantics, for that I should (according to its manpage) *block* signals, not ignore them. But maybe I should do some tests there as well.

Felix Palmen :freebsd: :c64:Hmm. Now that I have a working "generic" <a href="https://mastodon.bsd.cafe/tags/signal" class="mention hashtag" rel="nofollow noopener" target="_blank">#signal</a> handling in <a href="https://mastodon.bsd.cafe/tags/poser" class="mention hashtag" rel="nofollow noopener" target="_blank">#poser</a>, I'd like to optimize a bit by picking up <a href="https://infosec.exchange/@david_chisnall" class="u-url mention" rel="nofollow noopener" target="_blank">@david_chisnall</a>'s suggestion to use <a href="https://mastodon.bsd.cafe/tags/kqueue" class="mention hashtag" rel="nofollow noopener" target="_blank">#kqueue</a> for the job if available. Would have a clear advantage: No need to fiddle with the signal mask around every call to <a href="https://mastodon.bsd.cafe/tags/kevent" class="mention hashtag" rel="nofollow noopener" target="_blank">#kevent</a>.I still had the doubt whether a signal delivered via kqueue would still remain pending when it is just blocked, so I wrote some little test code and the unfortunate answer is: yes. Unfortunate because I want my library code to restore everything as it was found (signal mask and handlers) on exit, but I certainly don't want a batch of spurious signals handled when unblocking them.Kind of obvious solution: Set the signals temporarily to ignored when unblocking them, as shown in the screenshot. Now I have the next doubt: Is it guaranteed to have pending signals delivered instantly when unblocking them? 🤔<a href="https://mastodon.bsd.cafe/tags/FreeBSD" class="mention hashtag" rel="nofollow noopener" target="_blank">#FreeBSD</a> <a href="https://mastodon.bsd.cafe/tags/coding" class="mention hashtag" rel="nofollow noopener" target="_blank">#coding</a>

Felix Palmen :freebsd: :c64:Still working on <a href="https://mastodon.bsd.cafe/tags/swad" class="mention hashtag" rel="nofollow noopener" target="_blank">#swad</a>, and currently very busy with improving quality, most of the actual work done inside my <a href="https://mastodon.bsd.cafe/tags/poser" class="mention hashtag" rel="nofollow noopener" target="_blank">#poser</a> library.After finally supporting <a href="https://mastodon.bsd.cafe/tags/kqueue" class="mention hashtag" rel="nofollow noopener" target="_blank">#kqueue</a> and <a href="https://mastodon.bsd.cafe/tags/epoll" class="mention hashtag" rel="nofollow noopener" target="_blank">#epoll</a>, I now integrated <a href="https://mastodon.bsd.cafe/tags/xxhash" class="mention hashtag" rel="nofollow noopener" target="_blank">#xxhash</a> to completely replace my previous stupid and naive hashing. I also added a more involved <a href="https://mastodon.bsd.cafe/tags/dictionary" class="mention hashtag" rel="nofollow noopener" target="_blank">#dictionary</a> class as an alternative to the already existing <a href="https://mastodon.bsd.cafe/tags/hashtable" class="mention hashtag" rel="nofollow noopener" target="_blank">#hashtable</a>. While the hashtable's size must be pre-configured and collissions are only ever resolved by storing linked lists, the new dictionary dynamically nests multiple hashtables (using different bits of a single hash value). I hope to achieve acceptable scaling while maintaining also acceptable memory overhead that way ...<a href="https://mastodon.bsd.cafe/tags/swad" class="mention hashtag" rel="nofollow noopener" target="_blank">#swad</a> already uses both container classes as appropriate.Next I'll probably revisit poser's <a href="https://mastodon.bsd.cafe/tags/threadpool" class="mention hashtag" rel="nofollow noopener" target="_blank">#threadpool</a>. I think I could replace <a href="https://mastodon.bsd.cafe/tags/pthread" class="mention hashtag" rel="nofollow noopener" target="_blank">#pthread</a> condition variables by "simple" <a href="https://mastodon.bsd.cafe/tags/semaphores" class="mention hashtag" rel="nofollow noopener" target="_blank">#semaphores</a>, which should also reduce overhead ... <a href="https://github.com/Zirias/swad" rel="nofollow noopener" translate="no" target="_blank">https://github.com/Zirias/swad</a><a href="https://mastodon.bsd.cafe/tags/c" class="mention hashtag" rel="nofollow noopener" target="_blank">#c</a> <a href="https://mastodon.bsd.cafe/tags/coding" class="mention hashtag" rel="nofollow noopener" target="_blank">#coding</a>

Felix Palmen :freebsd: :c64:<a href="https://friedcheese.us/users/feld" class="u-url mention" rel="nofollow noopener" target="_blank">@feld</a> <a href="https://mastodon.bsd.cafe/@pertho" class="u-url mention" rel="nofollow noopener" target="_blank">@pertho</a> Doesn't feel like something I'd like to try though. It would be pretty much <a href="https://mastodon.bsd.cafe/tags/FreeBSD" class="mention hashtag" rel="nofollow noopener" target="_blank">#FreeBSD</a>-specific, but worse than that, you shouldn't rely on dtrace availability, as it's an optionally loadable profiling tool (so, it would also be a "misuse" regarding purpose). Related to that, I'm pretty sure it requires superuser privileges for everything, which would be another issue for some general-purpose application software.No, the "canonical" solution for filesystem watching on a BSD system is indeed <a href="https://mastodon.bsd.cafe/tags/kqueue" class="mention hashtag" rel="nofollow noopener" target="_blank">#kqueue</a>. And unfortunately, it does fall short a bit compared to <a href="https://mastodon.bsd.cafe/tags/Linux" class="mention hashtag" rel="nofollow noopener" target="_blank">#Linux</a>' <a href="https://mastodon.bsd.cafe/tags/inotify" class="mention hashtag" rel="nofollow noopener" target="_blank">#inotify</a>. For my <a href="https://mastodon.bsd.cafe/tags/xmoji" class="mention hashtag" rel="nofollow noopener" target="_blank">#xmoji</a> tool, I wanted notifications about any change on the runtime configuration file, and additionally to the pure <a href="https://mastodon.bsd.cafe/tags/POSIX" class="mention hashtag" rel="nofollow noopener" target="_blank">#POSIX</a> solution of periodically calling stat() (which is stupid, but still works for a single file), I implemented backends for both inotify and kqueue. For just a single file, kqueue's requirement of having an open file descriptor is just a minor annoyance, but you can deal with that. Note it's not as simple as it sounds in any case, e.g. when the file is deleted, you want to watch the directory of course, so you learn when it's re-created ... which with kqueue requires opendir() 🙈 ... still doable. But for scenarios where you want to watch a whole tree with potentially lots of files and directories, this is really bad and <a href="https://mastodon.bsd.cafe/tags/inotify" class="mention hashtag" rel="nofollow noopener" target="_blank">#inotify</a> really shines.

Felix Palmen :freebsd: :c64:<a href="https://mastodon.bsd.cafe/@pertho" class="u-url mention" rel="nofollow noopener" target="_blank">@pertho</a> The only little drawback compared to epoll is the lack of atomic signal mask setting, so you need a bit more code and a thoughtful structure to handle signals in the same loop. Apart from that, indeed much better than <a href="https://mastodon.bsd.cafe/tags/epoll" class="mention hashtag" rel="nofollow noopener" target="_blank">#epoll</a>.Unfortunately, it's not beter than inotify (for a completely different purpose, <a href="https://mastodon.bsd.cafe/tags/file" class="mention hashtag" rel="nofollow noopener" target="_blank">#file</a> <a href="https://mastodon.bsd.cafe/tags/monitoring" class="mention hashtag" rel="nofollow noopener" target="_blank">#monitoring</a> ... kqueue covers them all). With <a href="https://mastodon.bsd.cafe/tags/inotify" class="mention hashtag" rel="nofollow noopener" target="_blank">#inotify</a>, you can for example set a <a href="https://mastodon.bsd.cafe/tags/watch" class="mention hashtag" rel="nofollow noopener" target="_blank">#watch</a> by path, while <a href="https://mastodon.bsd.cafe/tags/kqueue" class="mention hashtag" rel="nofollow noopener" target="_blank">#kqueue</a> requires opened file descriptors. 😞

Recent searches

Search options

Administered by:

Server stats:

#kqueue