tcreech.com

FreeBSD OpenAFS Client Debugging

I’d really like to see FreeBSD’s OpenAFS client improve. It’s great that it has one that is mostly functional – and the packaging in FreeBSD’s ports is great – but it’s not in good enough shape to rely on for daily use. Here are my notes to myself on tracking down issues as I find them.

When I can, I’m trying to maintain an experimental net/openafs port with fixes for these issues on github.

“Small cache” problem

Symptom: writing a file that is large relative (near the same size or greater?) to the cache can hang one or more userspace processes. This is a problem in 1.6.15 as well as master.

What’s going on?

What are the afsd daemons doing when this happens? I’m trying to run with -daemon 1 to keep the number of processes down.

Figure out which processes are which of the daemons mentioned in the afsd man page using sudo procstat -k $(pgrep afsd).

sudo trace -F -n "fbt:libafs:*:*/pid == $PID_OF_INTERESTING_DAEMON/{ trace(probefunc); }"

Everything seems to be working ok and not stuck, but nothing is sending the data to the server. TruncateDaemon can’t empty anything out of the cache because it’s still the only copy. Who’s supposed to write stuff out before close if the cache is full?

EUREKA! This is https://lists.openafs.org/pipermail/openafs-devel/2002-July/008237.html, from 2002(!).

For Linux, this was fixed in 14dc5df, and then more properly in 4b2be95.

Status

I’ve submitted a patch to OpenAFS which fixes this in a crude manner.

EIO on some “ls -G” entries

Symptom: ls -G /afs/glue.umd.edu (or any other cell or non-fakestat directory) yields EIO on all but the last few entries. (And surely lots of other lookups are getting EIO too; ls is just the simplest way to see an issue.) This is a problem in master.

What’s going on?

Using git bisect, I tracked this down to 9b0d5f2.

Status

Fortunately, Michael Meffie has already discovered the issue, which was inadvertently caused by a related commit of his, and proposed a fix.

“disk I/O error” with sqlite3 database files in AFS (or, EINVAL on advisory locking)

Symptom: sqlite3 dbfile 'create table blah (int)' results in Error: disk I/O error if dbfile is in AFS. This seems obscure, but many applications (e.g., chromium) use sqlite3 databases all over the place.

What’s going on?

(Gleaned via dtrace)

The FreeBSD client’s afs_vop_advlock gets called to do an F_SETLCK with F_UNLCK as an argument in the struct flock argument. (This is done on behalf of kern_fcntl in the fcntl system call in the FreeBSD kernel.)

afs_vop_advlock is mostly a wrapper around the generic afs_lockctl function, which expects the fcntl operation (a_op == F_SETLCK) and the struct flock with flock->l_type == F_UNLCK to be passed to it to unlock. However, the FreeBSD kern_fcntl function incorrectly passes a_op == F_UNLCK. This makes no sense, so the afs_lockctl function returns that its arguments make no sense, and it returns EINVAL (22).

Unless I’m mistaken, this is a bug in FreeBSD’s sys/kern/kern_descrip.c that has existed at least since FreeBSD was forked from BSD 4.4 lite, in 1994! (The VOP_ADVLOCK man page refers to fcntl(2) for valid choices of a_op, and F_UNLCK is not one of them.)

Update: Thanks to dspinellis’ UNIX History repo, the questionable use of F_UNLCK has been traced back to the original BSD locking code from early 1991. (That’s older than Linux!)

Status

For now, the solution will be to change the FreeBSD client so that when its VOP_ADVLOCK function is called with a_op == F_SETFD == F_UNLCK (meaning the kernel really wanted an unlock, even though it’s asking for an invalid F_SETFD), change a_op <- F_SETLCK and leave l_type <- F_UNLCK. (This should be safe because it’s not sane to ask VOP_ADVLOCK for F_SETFD.)

This is a relatively easy change to make, and it allows locking applications to run. However, the locking still appears to be completely useless; for example, lockf will never block if the lock file is in AFS. I guess this is a different issue.