new_from_buffer() VS new_from_file()

[uasan]

Hello.
The lua worker, read bytes from stream pipe, we know length buffer, but not know image type.

How best to do, create an image object, new_from_buffer(buff) or file.write(buff) and new_from_file(name)?

Thank.

[jcupitt]

I would read the stream into a huge Lua string, then call new_from_buffer. It'll guess the file type for you.

[uasan]

Ok, can know the type of the image, by the first bytes from the buffer, so as not to read the entire buffer, if the first bytes are not correct?

[jcupitt]

It varies with the image format, but the first 1000 bytes should be enough, and not take too long.

[uasan]

Thank!

Through systemd, we are very easy from Lua script, make pipe server, new process luajit for each connection.

vips-server.socket
https://www.freedesktop.org/software/systemd/man/systemd.socket.html

[Unit]
Description = Vips socket

[Socket]
Accept       = yes
KeepAlive    = yes
ListenStream = /run/vips.sock

[Install]
WantedBy = sockets.target

vips-server@ .service
https://www.freedesktop.org/software/systemd/man/systemd.service.html

[Unit]
Description = Vips server

[Service]
Type           = simple
NonBlocking    = yes
StandardInput  = socket
StandardOutput = socket
StandardError  = journal
ExecStart      = /usr/bin/luajit /var/www/vips-server.lua

[Install]
WantedBy = multi-user.target

vips-server.lua

while true do

	local buff = io.read()
	print('ECHO: ' .. buff)
	io.flush()

end

If this is interesting, then we can put an example.

[jcupitt]

Sure, it sounds fun. Open a pull request with your example and I'll add it.

[uasan]

Oк, later when I'm free I will make an example.

Maybe more interesting inotify server
https://www.freedesktop.org/software/systemd/man/systemd.path.html

[uasan]

I have one more question )

We receive traffic in chunks of 9 kb, we can process images also in chunks?

Traffic speed
10 Mb = 1 sec

Process image speed
10MB = 0.7 sec

If we parallels in the process of obtaining chunks and processing chunks, then
will finish the work in 1 second, now in successive work take 1.7 seconds.

Is it possible to process images with chunks?

[jcupitt]

Unfortunately not. This has been discussed quite a bit, but for now you need to read the entire image to a string, then start vips on it.

There is a branch that adds true streaming, but it's quite old now. Maybe we should look at this again.

[jcupitt]

The most recent discussion around this is here, fwiw:

lovell/sharp#179

[jcupitt]

You can still overlap read and process, you just need to buffer a couple of images in memory.

You can have one thread reading into buffer1, and another thread processing from buffer2. When an image has been completely read, check that the processing thread has finished on buffer2, then swap the buffers and start reading and processing again.

I've not tried threading in Lua, does it support this kind of thing?

[uasan]

I have not tried threads in Lau, but i will try.
This all the calculations are theoretical, only experiment in practice get real result, for calculate benefits.

I thought, there is an excellent and popular web server Nginx - 80% all sites, it has a thread pool and the ability to load dynamic modules:
https://www.nginx.com/blog/thread-pools-boost-performance-9x/
https://www.nginx.com/blog/compiling-dynamic-modules-nginx-plus/

It has a built-in very popular module for image processing by libgd (slow)
http://nginx.org/en/docs/http/ngx_http_image_filter_module.html

If you make Vips-Nginx module, then many web developers, do not need binding to programming languages anymore, most developers will use lazy image processing in the Vips-Nginx module threads pool.

If you're interested in here is a guide
http://nginx.org/en/docs/dev/development_guide.html

Thank, i will try threads in Lau.

[jcupitt]

Yes, a nginx module sounds fun. I don't have time though :(

[uasan]

Nginx module can be as HTTP middleware, that gets REST requests from browsers or web backend to image processing, then node-sharp, php-vips, and others, will not be much needed, Nginx is 80% servers.

I hope you will have lot free time :)