Error creating temp directory

[itowlson]

Reported in spinframework/spin#3139 (comment): when attempting to re-run an unsuccessful deploy:

fschutt@Mac m4p % spin deploy    
Error: Failed to load Spin app from "spin.toml"

Caused by:
    0: Failed to load component `m4p`
    1: Failed to create parent directory "/var/folders/5x/rpb8yr7x6890kpc5886gzv9r0000gn/T/.tmpDXjuHE/m4p/."
    2: No such file or directory (os error 2)

The user reported that they worked around this by rebuilding the Spin application, but that shouldn't be necessary.

[itowlson]

The error message arises during asset copying. The location is LocalLoader::copy_glob/copy_single_file, called from LocalLoader load_file and load_manifest. It happens when fs::create_dir_all (by default, tokio::fs::create_dir_all).

I am a bit bothered by that . suffix on the parent directory though. I am wondering if the Spin manifest contained a files entry where the destination path ended was ., and if caused problems where fs tried to create the . directory within the parent. But the user reported that doing a new spin build solved the problem. And we are supposed to check that destination paths are absolute anyway.

@fschutt In the application where you saw this error, did the m4p component have a files = ... line in spin.toml, and if so are you able to share it please? Thanks!

[fschutt]

Here is the spin.toml:

spin_manifest_version = 2

[application]
name = "m4p"
version = "0.1.7"
authors = ["Felix Schütt <felix.schuett@maps4print.com>"]
description = ""

[[trigger.http]]
route = "/..."
component = "m4p"

[component.m4p]
source = "target/wasm32-wasip2/release/m4p.wasm"
allowed_outbound_hosts = ["https://github.com", "https://fonts.gstatic.com", "https://raw.githubusercontent.com", "https://maps4print.com", "https://*.paypal.com"]
files = ["*", "."]

[component.m4p.build]
command = "cargo build --release --target wasm32-wasip2"
watch = ["src/**/*.rs", "Cargo.toml"]

I constantly got errors related to the file permissions, I just wanted to allow "access to all files", because I got errors like this:

Server Initialization Error: Failed to seed styles: https://github.com/maps4print/styles/raw/564ec38ba62c14113523748e623a5e29b56a17d5/berlin-10k/berlin-10k.pdf: /cache/2jpfP3np-berlin-10k.pdf: failed to find a pre-opened file descriptor through which "/cache/2jpfP3np-berlin-10k.pdf" could be opened

I had severe confusion about the file thing: I tried to debug and debug why it cannot open a file descriptor.

I can give you access to the m4p source code, so you can test for yourself - again, this is a more-or-less prod application.

Potentially it would also be better to give more control about readable / writable files separately, as an idea. I.e. I want some directories to only be readable, but never writable (like static assets).

Is the files thing not using regex? If it just said "this is a regex, which file paths are allowed", it would be mentally easier for the developer.

Also: I cannot package static files alongside the .wasm file. This is a big deal because now I have to include_bytes!() them into my .wasm file.

[itowlson]

Thanks @fschutt - this clears things up for me.

Regarding the files thing, it is a list of things to make available to the Wasm component, and each entry allows two forms:

• Glob form: "images/*.jpg" - this will make those files available to the Wasm component at that exact path. E.g. your Wasm component would have to refer to /images/foo.jpg. Note this is glob syntax not regex syntax.
• Directory form: { source = "images", destination = "/assets" } - this makes the entire directory tree available to the Wasm component at the destination path. E.g. images/foo.jpg in your source would be referred to as /assets/foo.jpg in Wasm.

Your current files entry is strings rather than source-destination pairs, so we are treating those as glob patterns. The * is I guess mounting all files in the root of your project. The . is, I think, causing a problem because it is being interpreted as "current directory" and we are trying to map your source directory to a . "file" which of course fails. I'll look at how we can provide a better error on this.

To allow all files, I would suggest:

files = [{ source = ".", destination = "/" }]

or

files = ["**/*"]  # not sure if you might need a "*.*` instead

Alternatively, you might choose a more specific subset of files e.g.

# Purely example, depending on your your application is laid out!
files = [{ source = "styles", destination = "/styles" }, { source = "scripts/dist", destination = "/scripts" }]

One of these options would hopefully solve your file bundling problems. If not, I'd be happy to work with you to figure out a good way to express what you need (and then feed that back into the documentation!).

You mentioned writeable files. In general, files are not writeable (because an application is stateless). The Spin CLI has an option to allow transient writes, but this is not available in all Spin runtimes. Specifically, Fermyon Cloud does not allow file writes (because different instances of a component may be instantiated on different hosts). To persist or cache blobs, we recommend key-value storage or SQLite. I will look at how we can make this clearer in the documentation.

[fschutt]

Yeah, this is all a bit confusing, specifically the starting / syntax. Better docs on this would be good + maybe an analysis for regexes that include each other?

Cargo.toml solves this by directly including commented-out documentation in the Cargo.toml when you do cargo new, so you could generate the "documentation" directly in the spin.toml:

# Welcome to Spin! This spin.toml will configure your application for deployment.
# See https://docs.fermyon.com/manifest/v2/docs for keys and values.
spin_manifest_version = 2

[application]
name = "m4p"
version = "auto" # auto = use the version from Cargo.toml
authors = [""] # fill in from git if possible
description = "Example description of the m4p application"
# some_commented_out_key = "example value"

[[trigger]]
type = "http" # note: could be later extended for DNS, Mail protocol (Mail Workers), etc.
route = "/..."
component = "m4p"

# Configure applications for deployment
[component.m4p]
source = "target/wasm32-wasip2/release/m4p.wasm"
# note: I would rename this to "network" (= network access configuration), 
# = shorter + semantic with "files" (file access configuration)
# note: also, this should allow regexes + shortnames for domains
# Which hosts can be called
allowed_outbound_hosts = ["https://github.com"]
# allow read-only access to specific directories
# warning: fermyon runs stateless, writes will fail
# syntax: see https://fermyon.cloud/manifest/v2#files
files = ["*"]

This way you can immediately warn users by outputting docs to the spin.toml.

I'd also allow for the outbound shortnames that don't include the scheme name:

# configure your HTTP component
[component.m4p]
type = "http" # to be extended later for Mail Workers, DNS workers, AI embedding workers, etc.
binary = "target/wasm32-wasip2/release/m4p.wasm" # "source" is confusing
pack = ["dist/*.css"] # which static files to pack into the fermyon package
network = ["github.com", "*.paypal.com", "*.sandbox.paypal.com"]
files = ["*.jpg", "static/"]
runcmd = ["m4p.wasm", "--some-flag", "$ENVVAR"]
mount = "up.sql"
unmount = "down.sql"

Note that there should also be a way to "pack" local files into the target dir

You mentioned writeable files. In general, files are not writeable (because an application is stateless).

Ah, yeah, that's maybe why it fails to save the PDF. Basically, the old server would take a config and then download and cache PDF files on startup. It would also create / recreate / fix the SQL schema, so that the server is "ready" to handle queries.

As a note, it would be good to have something like a #[mount] / #[unmount] handler, that is run only once by one application before it is deployed. That way, I can do things like create database tables, run SQL migrations, etc. but only ONCE, not on every HTTP request.

The stateless thing confused me, it makes sense, but it was very weird to adjust the thinking

[itowlson]

Cargo.toml solves this by directly including commented-out documentation

Yes, there's some discussion of more verbose templates at spinframework/spin#3127 - the challenge is a tension between "explain everything" / "overwhelm the reader" / "spend 10 minutes deleting boilerplate". Thanks for offering a perspective! We are also now generating schemas that can be used with e.g. Even Better TOML to provide hints and completions, but those are not fully ready for prime time yet. Again, your experience helps to see areas that we could helpfully clarify and expand - thank you!

As a note, it would be good to have something like a #[mount] / #[unmount] handler

It would! We have mulled this a few times (most recently spinframework/spin#2443) but do not have a solution for it yet, I'm afraid.

The stateless thing confused me, it makes sense, but it was very weird to adjust the thinking

It's a change of mental model for sure, and we are close enough to the project that we don't always see that. Thanks for sticking with it and providing your perspective and the pain points you're hitting - it's really helpful.

[bacongobbler]

FYI: new upstream location is fermyon/feedback#60