NetWyrm’s Litte Corner

In my last post I talked about patching LuaSocket to make it work with LuaLanes. The patch is there: lua socket acceptfd patch.

So what does it do?

It adds a new method “acceptfd” which works just like “accept” but instead of returning a socket object it returns  the underlying “file descriptor” (a number). A new optional parameter to “socket.tcp()” allows to create a socket object from a file descriptor.

How does it help us ?

Lua Lanes handles multithreading by running a new lua state in each thread, they do not share data and instead communicate using Lindas (see the docs on Lanes site). Linda is great but it cannot pass such userdatas as sockets. Si we use this patch to accept an incomming connection, getting its  file descriptor. Then pass it onto a new thread which creates its socket object from it.

Easy, fast and grooovy!

A little exemple:

require ‘lanes’
require ’socket’

local linda = lanes.linda()

function handler(fd)
require’socket’

local sock = socket.tcp(fd)
local line = sock:receive(”*l”)
linda:send(”order”, {line=line})
sock:send(”line: “..tostring(line)..” \n”)
sock:close()
return “done”
end

function main_thread()
while true do
local order = linda:receive(3, “order”)
if order then
print(”order”, order.line)
end
end
end

function run()
local s = socket.bind(”0.0.0.0″, 2525)
local threads = {}

lanes.gen(”*”, main_thread)()

while true do
local fd = s:acceptfd()
threads[lanes.gen("*", handler)(fd)] = fd

print(”threads:”)
for h, s in pairs(threads) do
print(” * “, h, ” :=: “, h.status)

local v, err = h:join(0)
if not v and err then
print(err)
elseif v and v == “done” then
threads[h] = nil
end
end
end
end

run()

(Badly formated sorry, I have not yet mastered the art of posting code with wordpress )

Ever wanted to make a network server in Lua?

You have come to the right place, after much experimentation on Tethys and Mailcatch.

When it comes down to how you can do a sockets server you have those options:

• A single process with lua coroutines. The main thread will handle the server socket, when accept’ing connections it starts a new coroutine to handle it. A coroutine scheduler is necessary to yield while waiting on sockets. Such a scheduler could be Copas or LOOP. I will write about it later but I have found this way to choke when handling many connections/seconds
• A forking server. The main process handles the server socket, when accept’ing connections it forks a new process to handle it. This is quite easy to write using lua posix or some more specialized libs (I will release one that wraps it up nicely someday).
• A single process server with multithreading. Lua does not do multithread but there are modules out there to do it. The most prominent one being Lua Lanes.

I have experienced with all three (in this order) and settled for multithreading using Lanes with a twist (it does not spawn a new thread for each connection it uses a pool of pre-existing ones) because it gave me, by far, the best performance.

In my next posts I will explain those three methods and how to implement them easily in Lua.

I have just released a new version of Tethys, you can find it here: Tethys SMTP server 2.0.3 .

This version changes the following things:

• Allows to specify a maximun number of thread spawned (config command “max_threads = X”)
• Allows to specify a timeout on socket operations to kill clients that keep connections open forever
• Self contained version available (contains all needed lua libs and lua itself)
• Allows to specify a maximun number of connections from a single IP
• Two process models to choose from: lua coroutines or process forking
• New “smtp” plugin type, allows to extend the smtp command-set

Have fun!

I have just released a new version of Tethys, you can find it here: Tethys SMTP server 2.0.2 .

This version changes the following things:

• Correct SMTP AUTH code when success (235 instead of 334)
• MySQL plugins now keep their connections open and are able to reopen them should the SQL server go away in between
• New config parameter “user_manager.failsafe_user”, if a deposit plugin fails to read the data that tells it which user to deliver to, it will deliver to this one. (So a mysql failure will not mean incomming mails are lost)
• tethys2.plugins.user_manager.MySQL plugin must now be preloaded in the “preload.plugins.receiver” section of the config file, if used
• FixMail plugin now can now replace receiving Date field by the current server date. It will keep the original date in a X-Tethys-Original-Date field. Activate it using the “filter.fixmail.replace.date = true” line in your config file
• A simple “Log” deposit that just prints the mail body to the log file, for easier debuging
• New “log.level” config directive (MUST be set, 0=no log, 1=errors, 2=info, 3=debug)
• Logs are sent to console if not running as a daemon
• Default config file now has a default user_manager (UnixAlias)

Have fun!

NetPIM is my second online service, as with the first one it was born from a specific need: I wanted to be able to access my contacts from anywhere and to keep them in sync. I also wanted to never forget a birthday, so I also needed a calendar, or at least an automatic reminder.

I knew some online calendar services (like google’s one) but there was no free LDAP server around that I felt confident to use, so I promptly installed OpenLDAP on my server and started learning about it. I soon realized I would need a web interface too to consult the contacts easily and to edit them. Merging that with an online calendar, the basic idea of NetPIM was born. Obviously I started doing it in Lua using mod_wombat (which will be included in apache as mod_lua for 2.4 release! yeah !). Some day I’ll post here a guide of how to do some web development with mod_wombat.

As usual I started pondering if I should open it to the world, so I offered it to my friends who started using it and liked it, so I thought, what the hell, let’s go! Over the time I added other features, like bookmarks (I even have a firefox extension to handle them, currently being beta tested by some friends) and notes taking (personal wiki).

So if you feel the need for the same features, go on, try it, I hope it is easy and fast to use, I have tried to minimize the number of clicks needed to do something with it, and I am not too dishonored by the tags system I think .. hope … pray

MailCatch is my first website in Lua. Its purpose is to allow any incomming emails on any accoutns to be available for a while. That is, it is a temporary email address system, or virtual inbox. It all started one year and a half ago when I got pissed off while trying to install a mail server for my newly aquired server(which now hosts NetCore and my services). I tried postfix, qmail, sendmail and more but each time I stumbled upon some annoying road blocks (like not being able to use mysql for virtual users). So being the Lua geek I am I decided to make my own smtp server (Tethys) instead of trying to work around the bloated config files other the “big ones”.

My goal was simple: fulfill my specific needs, which were at the time:

• maildir support
• virtual users using a mysql database for definition
• subdomains support
• easy configuration
• easily extensible by plugins
• probably some more I forgot now

After reading the SMTP RFCs,  I set out to find some nice Lua packages (LOOP OO programing lib, LuaSocket, …), made a few on my own (Telestos) and started having great fun! After a few days I had a basic implementation going and go onto the plugins part.

Then an idea hit me in the head (yes it hurts a bit), as I wanted to be able to make new virtual addresses easily to handle spam sent to me I thought “but why define addresses ? Just let them all in on a subdomain!”, and so the basic idea for MailCatch was born (which was totaly a non-new idea for the rest of the world it seems as therer are otehr such services, but I was not aware of them by then).

The MailCatch plugin for Tethys was started, it was to be simple and pretty fast. I did not want to burn my hard drive by writing spam to it, there is no use, usualy mail spam addresses are one time use only, so I made the plugin a delivery plugin. That means when Tethys has finished receiving a mail it passes it to a delivery plugin to do something with it, the default one writes it to a maildir spool, but this one just keeps it in memory for a little while and then forgets about it. No hard drive access!

Once I was satisfied I made a quick website in Lua (which I’ll explain in an other post) to display the mails in the plugin memory. This site later became MailCatch (which you all know .. obviously!) when I decided other people might find a use for it.

Welcome to my blog, as it seems everybody must have one those days

What will you find here?

• Some talks about Lua, a very nice and lovable programming language
• Some talks, explanations, details, .. about the various services and programs I maintain
• Some rantings about .. stuff
• Not too much boringness, I hope

Have fun !