Commit Graph

13 Commits

Author SHA1 Message Date
Jakob Unterwurzacher
84e702126a fusefrontend: implement recursive diriv caching
The new contrib/maxlen.bash showed that we have exponential
runtime with respect to directory depth.

The new recursive diriv caching is a lot smarter as it caches
intermediate lookups. maxlen.bash now completes in a few seconds.

xfstests results same as
2d158e4c82/screenlog.0 :

  Failures: generic/035 generic/062 generic/080 generic/093 generic/099 generic/215 generic/285 generic/319 generic/426 generic/444 generic/467 generic/477 generic/523
  Failed 13 of 580 tests

benchmark.bash results are identical:

  $ ./benchmark.bash
  Testing gocryptfs at /tmp/benchmark.bash.BdQ: gocryptfs v2.0.1-17-g6b09bc0; go-fuse v2.1.1-0.20210611132105-24a1dfe6b4f8; 2021-06-25 go1.16.5 linux/amd64
  /tmp/benchmark.bash.BdQ.mnt is a mountpoint
  WRITE: 262144000 bytes (262 MB, 250 MiB) copied, 0,4821 s, 544 MB/s
  READ:  262144000 bytes (262 MB, 250 MiB) copied, 0,266061 s, 985 MB/s
  UNTAR: 8,280
  MD5:   4,564
  LS:    1,745
  RM:    2,244
2021-06-25 13:56:53 +02:00
Jakob Unterwurzacher
f73aee72f8 fusefrontend: print dirCache stats after unmount 2021-04-05 18:20:17 +02:00
Jakob Unterwurzacher
043f81dd01 fs: more idiomatic dirCache struct naming 2021-04-04 13:05:47 +02:00
Jakob Unterwurzacher
24d5d39300 fs: add initial dirfd caching
dirfd caching was temporarily removed when moving
to the v2api. Add it back to gain back some lost speed.
2021-04-03 13:08:28 +02:00
Jakob Unterwurzacher
f8ad2ac3e2 dircache: increase cache size & lifetime
Looking at the dircache debug output, we see
that a "git status" workload has a very bad
cache hit rate because the entries expire or
get evicted before they can be reused.

Increase both cache size and lifetime for
a 4x speedup:

Before: 75s
After:  17s

https://github.com/rfjakob/gocryptfs/issues/410
2020-05-17 21:37:36 +02:00
Jakob Unterwurzacher
f6088e5008 dircache: improve debug messages
Before:

Lookup "errno.html/1/2/3/4/5": miss
Store: "errno.html/1/2/3/4/5" fd=26 iv=21be6e083d60dcabfe7368264d5082b7
Lookup "errno.html": hit 25 6d68a16d217978915036a3bd55428ae7
Lookup "errno.html/1": hit 25 932a464c299b3430c5e55c924f98ac4d
Lookup "errno.html/1/2": hit 25 7d53348b1692d537f017bf86b3cf5feb
Lookup "errno.html/1/2/3": hit 25 2aef1c9d1ab2b55b163215053fefe703
Lookup "errno.html/1/2/3/4": hit 25 cb802be53721c46a46247c5e4e0f4ce6
Lookup "errno.html/1/2/3/4": hit 25 cb802be53721c46a46247c5e4e0f4ce6
Lookup "errno.html": hit 25 6d68a16d217978915036a3bd55428ae7

After:

Lookup "earlyoom/.git/refs"                     hit fd=10 dup=17 iv=6ae2cecd269a25e8d946aff6afe9b8b8
Lookup "earlyoom/.git/refs/remotes"             hit fd=19 dup=17 iv=f04c2d2a5bcc33ebdeaca664859c980d
Lookup "earlyoom/.git/refs/remotes/origin"      miss
Store  "earlyoom/.git/refs/remotes/origin"      fd=17 iv=834a64a1697c9f5705455ba6dbed22b5
Lookup "earlyoom"                               hit fd=7 dup=25 iv=2303a892d6e2357c483574a8070b7679
Lookup "earlyoom/.git"                          hit fd=11 dup=25 iv=d43ca4aff23720c57789c9f62f0aee00
Lookup "earlyoom/.git"                          hit fd=11 dup=25 iv=d43ca4aff23720c57789c9f62f0aee00
Lookup "earlyoom/.git/refs"                     hit fd=10 dup=25 iv=6ae2cecd269a25e8d946aff6afe9b8b8
Lookup "earlyoom/.git/refs/heads"               hit fd=13 dup=25 iv=f9245e7c066b9adc768a1a666da9fbc8
2020-05-17 21:26:56 +02:00
Sebastian Lackner
7e05e809b7 main: Run 'ensure fds' code early during the program startup.
The files are apparently processed in alphabetic order, so cli_args.go is
processed before main.go. In order to run before the go-fuse imports, put
the 'ensure fds' code in a separate package. Debug messages are omitted
to avoid additional imports (that might contain other code messing up our
file descriptors).
2019-01-05 16:12:16 +01:00
Sebastian Lackner
f17721c364 A few more spelling fixes. 2019-01-05 12:27:55 +01:00
Jakob Unterwurzacher
65eded4a22 tests: bump maxCacheFds to 3
As the dirCache now has 3 entries, the tests should accept
up to 3 extra fds without declaring an fd leak.
2019-01-04 23:50:01 +01:00
Jakob Unterwurzacher
c32066c5b0 fusefrontend: fix fd leak in dirCache
The missing break meant that we may find a second
hit in the cache, Dup() a second fd, and leak the first
one.

Thanks @slackner for finding this.
2019-01-04 23:35:48 +01:00
Jakob Unterwurzacher
3365cfc02b fusefrontend: disable dirCache stats printing
This was inadvertedly kept enabled after benchmarking.
2019-01-04 22:07:02 +01:00
Jakob Unterwurzacher
c0a7a14cde fusefrontend: upgrade to three-entry dirCache
3 entries should work well for up to three parallel users.
It works well for extractloop.bash (two parallel tar extracts).
2019-01-04 21:45:03 +01:00
Jakob Unterwurzacher
4f66d66755 fusefrontend: add dirCache 2019-01-03 15:31:13 +01:00