* [PATCH] [RESEND] fs: cachefiles: Add support for large files in filesystem caching @ 2012-06-11 5:38 Suresh Jayaraman 2012-06-18 23:04 ` Andrew Morton 0 siblings, 1 reply; 7+ messages in thread From: Suresh Jayaraman @ 2012-06-11 5:38 UTC (permalink / raw) To: David Howells; +Cc: Justin Lecher, Andrew Morton, LKML, linux-cachefs The below patch which was first posted here: https://lkml.org/lkml/2012/4/3/193 and tested by me and picked up by the maintainer here: https://lkml.org/lkml/2012/5/1/57 But it seems to not to have made to the linux-next or Linus tree. I'm resending what it seems like a lost patch. From: Justin Lecher <jlec@gentoo.org> Support the caching of large files. https://bugzilla.kernel.org/show_bug.cgi?id=31182 Signed-off-by: Justin Lecher <jlec@gentoo.org> Tested-by: Suresh Jayaraman <sjayaraman@suse.com> --- fs/cachefiles/rdwr.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/fs/cachefiles/rdwr.c b/fs/cachefiles/rdwr.c index 0e3c092..d136686 100644 --- a/fs/cachefiles/rdwr.c +++ b/fs/cachefiles/rdwr.c @@ -919,7 +919,7 @@ int cachefiles_write_page(struct fscache_storage *op, struct page *page) dget(object->backer); mntget(cache->mnt); file = dentry_open(object->backer, cache->mnt, O_RDWR, - cache->cache_cred); + (O_RDWR | O_LARGEFILE), cache->cache_cred); if (IS_ERR(file)) { ret = PTR_ERR(file); } else { ^ permalink raw reply related [flat|nested] 7+ messages in thread
* Re: [PATCH] [RESEND] fs: cachefiles: Add support for large files in filesystem caching 2012-06-11 5:38 [PATCH] [RESEND] fs: cachefiles: Add support for large files in filesystem caching Suresh Jayaraman @ 2012-06-18 23:04 ` Andrew Morton 2012-06-19 5:27 ` Suresh Jayaraman 0 siblings, 1 reply; 7+ messages in thread From: Andrew Morton @ 2012-06-18 23:04 UTC (permalink / raw) To: Suresh Jayaraman; +Cc: David Howells, Justin Lecher, LKML, linux-cachefs On Mon, 11 Jun 2012 11:08:21 +0530 Suresh Jayaraman <sjayaraman@suse.com> wrote: > The below patch which was first posted here: > > https://lkml.org/lkml/2012/4/3/193 > > and tested by me and picked up by the maintainer here: > > https://lkml.org/lkml/2012/5/1/57 > > But it seems to not to have made to the linux-next or Linus tree. > I'm resending what it seems like a lost patch. > > > From: Justin Lecher <jlec@gentoo.org> > > Support the caching of large files. > > https://bugzilla.kernel.org/show_bug.cgi?id=31182 > > Signed-off-by: Justin Lecher <jlec@gentoo.org> > Tested-by: Suresh Jayaraman <sjayaraman@suse.com> > --- > fs/cachefiles/rdwr.c | 2 +- > 1 files changed, 1 insertions(+), 1 deletions(-) > > diff --git a/fs/cachefiles/rdwr.c b/fs/cachefiles/rdwr.c > index 0e3c092..d136686 100644 > --- a/fs/cachefiles/rdwr.c > +++ b/fs/cachefiles/rdwr.c > @@ -919,7 +919,7 @@ int cachefiles_write_page(struct fscache_storage *op, struct page *page) > dget(object->backer); > mntget(cache->mnt); > file = dentry_open(object->backer, cache->mnt, O_RDWR, > - cache->cache_cred); > + (O_RDWR | O_LARGEFILE), cache->cache_cred); Doesn't compile. How extensively was this change tested? Please describe the testing which was performed? ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [PATCH] [RESEND] fs: cachefiles: Add support for large files in filesystem caching 2012-06-18 23:04 ` Andrew Morton @ 2012-06-19 5:27 ` Suresh Jayaraman 2012-06-19 7:52 ` Andrew Morton 0 siblings, 1 reply; 7+ messages in thread From: Suresh Jayaraman @ 2012-06-19 5:27 UTC (permalink / raw) To: Andrew Morton; +Cc: David Howells, Justin Lecher, LKML, linux-cachefs On 06/19/2012 04:34 AM, Andrew Morton wrote: >> From: Justin Lecher <jlec@gentoo.org> >> >> Support the caching of large files. >> >> https://bugzilla.kernel.org/show_bug.cgi?id=31182 >> >> Signed-off-by: Justin Lecher <jlec@gentoo.org> >> Tested-by: Suresh Jayaraman <sjayaraman@suse.com> >> --- >> fs/cachefiles/rdwr.c | 2 +- >> 1 files changed, 1 insertions(+), 1 deletions(-) >> >> diff --git a/fs/cachefiles/rdwr.c b/fs/cachefiles/rdwr.c >> index 0e3c092..d136686 100644 >> --- a/fs/cachefiles/rdwr.c >> +++ b/fs/cachefiles/rdwr.c >> @@ -919,7 +919,7 @@ int cachefiles_write_page(struct fscache_storage *op, struct page *page) >> dget(object->backer); >> mntget(cache->mnt); >> file = dentry_open(object->backer, cache->mnt, O_RDWR, >> - cache->cache_cred); >> + (O_RDWR | O_LARGEFILE), cache->cache_cred); > > Doesn't compile. Oops, my bad. While refreshing the original patch against the latest kernel, I seemed to have introduced an extra argument while attempting to limit the characters to 80. Extremely sorry about that. > How extensively was this change tested? Please describe the testing > which was performed? > The original patch was tested by mounting a NFS share with fscache option enabled, do a md5sum on a nfs file larger than 2GB and ensure that the file is getting cached by fscache (watch cache size growing) on x86_64. Please find below the fixed one. From: Justin Lecher <jlec@gentoo.org> Subject: [PATCH] fs: cachefiles: Add support for large files in filesystem caching Support the caching of large files. https://bugzilla.kernel.org/show_bug.cgi?id=31182 Tested by mounting a NFS share with fscache option enabled, do a md5sum on a nfs file larger than 2GB and ensure that the file is getting cached by fscache (watch cache size growing) on x86_64. Signed-off-by: Justin Lecher <jlec@gentoo.org> Signed-off-by: Suresh Jayaraman <sjayaraman@suse.com> Tested-by: Suresh Jayaraman <sjayaraman@suse.com> Cc: David Howells <dhowells@redhat.com> Acked-by: David Howells <dhowells@redhat.com> --- fs/cachefiles/rdwr.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/fs/cachefiles/rdwr.c b/fs/cachefiles/rdwr.c index 0e3c092..cc7ef95 100644 --- a/fs/cachefiles/rdwr.c +++ b/fs/cachefiles/rdwr.c @@ -918,7 +918,7 @@ int cachefiles_write_page(struct fscache_storage *op, struct page *page) * own time */ dget(object->backer); mntget(cache->mnt); - file = dentry_open(object->backer, cache->mnt, O_RDWR, + file = dentry_open(object->backer, cache->mnt, (O_RDWR | O_LARGEFILE), cache->cache_cred); if (IS_ERR(file)) { ret = PTR_ERR(file); ^ permalink raw reply related [flat|nested] 7+ messages in thread
* Re: [PATCH] [RESEND] fs: cachefiles: Add support for large files in filesystem caching 2012-06-19 5:27 ` Suresh Jayaraman @ 2012-06-19 7:52 ` Andrew Morton 2012-06-19 7:58 ` justin 2012-06-20 13:35 ` Suresh Jayaraman 0 siblings, 2 replies; 7+ messages in thread From: Andrew Morton @ 2012-06-19 7:52 UTC (permalink / raw) To: Suresh Jayaraman; +Cc: David Howells, Justin Lecher, LKML, linux-cachefs On Tue, 19 Jun 2012 10:57:27 +0530 Suresh Jayaraman <sjayaraman@suse.com> wrote: > > How extensively was this change tested? Please describe the testing > > which was performed? > > > > The original patch was tested by mounting a NFS share with fscache > option enabled, do a md5sum on a nfs file larger than 2GB and ensure > that the file is getting cached by fscache (watch cache size growing) > on x86_64. Well it will need a lot more coverage testing than that. truncate? expanding truncate? write, lseek, pwrite, pread(), mmap(MAP_SHARED), behavior at the new max file size (what is that?), etc. ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [PATCH] [RESEND] fs: cachefiles: Add support for large files in filesystem caching 2012-06-19 7:52 ` Andrew Morton @ 2012-06-19 7:58 ` justin 2012-06-20 13:35 ` Suresh Jayaraman 1 sibling, 0 replies; 7+ messages in thread From: justin @ 2012-06-19 7:58 UTC (permalink / raw) To: Andrew Morton; +Cc: Suresh Jayaraman, David Howells, LKML, linux-cachefs [-- Attachment #1: Type: text/plain, Size: 961 bytes --] On 19/06/12 09:52, Andrew Morton wrote: > On Tue, 19 Jun 2012 10:57:27 +0530 Suresh Jayaraman <sjayaraman@suse.com> wrote: > >>> How extensively was this change tested? Please describe the testing >>> which was performed? >>> >> >> The original patch was tested by mounting a NFS share with fscache >> option enabled, do a md5sum on a nfs file larger than 2GB and ensure >> that the file is getting cached by fscache (watch cache size growing) >> on x86_64. > > Well it will need a lot more coverage testing than that. truncate? > expanding truncate? write, lseek, pwrite, pread(), mmap(MAP_SHARED), > behavior at the new max file size (what is that?), etc. > Hello, I am using this patch since the 2.6.3* linux, while working with files >4 Gb on a daily bases. This isn't a testsuite, but it worked fine for me all the time. How would such testsuite you request look like? Anybody willing to write it? Thanks, Justin [-- Attachment #2: OpenPGP digital signature --] [-- Type: application/pgp-signature, Size: 302 bytes --] ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [PATCH] [RESEND] fs: cachefiles: Add support for large files in filesystem caching 2012-06-19 7:52 ` Andrew Morton 2012-06-19 7:58 ` justin @ 2012-06-20 13:35 ` Suresh Jayaraman 2012-06-20 19:45 ` Justin 1 sibling, 1 reply; 7+ messages in thread From: Suresh Jayaraman @ 2012-06-20 13:35 UTC (permalink / raw) To: Andrew Morton; +Cc: David Howells, Justin Lecher, LKML, linux-cachefs On 06/19/2012 01:22 PM, Andrew Morton wrote: > On Tue, 19 Jun 2012 10:57:27 +0530 Suresh Jayaraman <sjayaraman@suse.com> wrote: > >>> How extensively was this change tested? Please describe the testing >>> which was performed? >>> >> >> The original patch was tested by mounting a NFS share with fscache >> option enabled, do a md5sum on a nfs file larger than 2GB and ensure >> that the file is getting cached by fscache (watch cache size growing) >> on x86_64. > > Well it will need a lot more coverage testing than that. truncate? > expanding truncate? write, lseek, pwrite, pread(), mmap(MAP_SHARED), > behavior at the new max file size (what is that?), etc. > The changelog could be slightly misleading as the patch doesn't really add LFS support. It just passes O_LARGEFILE flag to dentry_open() to make VFS allow cachefiles to open larger files (instead of failing it with -EOVERFLOW). Cachefiles is just a cache that uses the VFS/VM interfaces to get another disk filesystem to do the requisite I/O on its behalf. So, I doubt whether more coverage testing is required for this change. Anyway, I'm attaching a quick and dirty test program that might help (not well tested though). I'm unable to to test it as I have modified my setup and have to redo it. Justin, do you have the setup and can you get this tested? /* * Few tests to ensure LFS (Large File Support) is working correctly. * * The requests (seek, truncate, write, mmap etc) that extend beyond the offset * maximum (off_t) should succeed. * * Compile it with -D_FILE_OFFSET_BITS=64 * * Run it like: * ./lfs-tests large_file * */ #include <stdio.h> #include <stdlib.h> #include <string.h> #include <fcntl.h> #include <unistd.h> #include <sys/types.h> #include <sys/stat.h> #include <sys/mman.h> #define BUF_SZ 32 #define err(msg) \ do { perror(msg); exit(EXIT_FAILURE); } while (0) void report_size(int fd, struct stat *st) { fstat(fd, st); printf("size of file: %ld\n", st->st_size); return; } int main(int argc, char *argv[]) { int fd; int rc = 0; unsigned long offset = 0; struct stat stbuf; char buf[BUF_SZ]; void *file_mem; if (argc != 2) { fprintf(stderr, "Usage: %s test_file\n", argv[0]); exit(EXIT_FAILURE); } fd = open(argv[1], O_RDWR | O_APPEND); if (fd == -1) err("open"); memset(buf, 0, sizeof(buf)); /* seek to the end of file */ offset = lseek(fd, 0, SEEK_END); if (offset < 0) err("lseek"); /* seek past the end of file */ offset = lseek(fd, BUF_SZ, SEEK_END); printf("%ld\n", offset); if (offset < 0) err("lseek"); /* truncate it to < 2GB and report size */ rc = ftruncate(fd, 2147483648 - BUF_SZ); if (rc != 0) err("ftruncate"); report_size(fd, &stbuf); /* truncate it to a size larger than 2GB */ rc = ftruncate(fd, 4294967360); if (rc != 0) err("ftruncate"); report_size(fd, &stbuf); /* seek to the end of file */ offset = lseek(fd, 0, SEEK_END); if (offset < 0) err("lseek"); memset(buf, 'a', sizeof(buf)); /* write past 4GB */ rc = write(fd, buf, sizeof(buf)); if (rc == -1) err("write"); report_size(fd, &stbuf); offset = lseek(fd, 0, SEEK_END); if (offset < 0) err("lseek"); /* pwrite and pread */ rc = pread(fd, buf, BUF_SZ, offset); if (rc == -1) err("pread"); rc = pwrite(fd, buf, sizeof(buf), offset); if (rc == -1) err("pwrite"); report_size(fd, &stbuf); /* page align offset */ offset = offset & ~(sysconf(_SC_PAGE_SIZE) - 1); file_mem = mmap(0, BUF_SZ, PROT_WRITE, MAP_SHARED, fd, offset); if (file_mem == MAP_FAILED) err("mmap"); munmap(file_mem, 64); close(fd); return EXIT_SUCCESS; } ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [PATCH] [RESEND] fs: cachefiles: Add support for large files in filesystem caching 2012-06-20 13:35 ` Suresh Jayaraman @ 2012-06-20 19:45 ` Justin 0 siblings, 0 replies; 7+ messages in thread From: Justin @ 2012-06-20 19:45 UTC (permalink / raw) To: Suresh Jayaraman; +Cc: Andrew Morton, David Howells, LKML, linux-cachefs [-- Attachment #1: Type: text/plain, Size: 4626 bytes --] On 20.06.2012 15:35, Suresh Jayaraman wrote: > On 06/19/2012 01:22 PM, Andrew Morton wrote: >> On Tue, 19 Jun 2012 10:57:27 +0530 Suresh Jayaraman <sjayaraman@suse.com> wrote: >> >>>> How extensively was this change tested? Please describe the testing >>>> which was performed? >>>> >>> >>> The original patch was tested by mounting a NFS share with fscache >>> option enabled, do a md5sum on a nfs file larger than 2GB and ensure >>> that the file is getting cached by fscache (watch cache size growing) >>> on x86_64. >> >> Well it will need a lot more coverage testing than that. truncate? >> expanding truncate? write, lseek, pwrite, pread(), mmap(MAP_SHARED), >> behavior at the new max file size (what is that?), etc. >> > > The changelog could be slightly misleading as the patch doesn't really > add LFS support. It just passes O_LARGEFILE flag to dentry_open() to make > VFS allow cachefiles to open larger files (instead of failing it with > -EOVERFLOW). Cachefiles is just a cache that uses the VFS/VM interfaces to > get another disk filesystem to do the requisite I/O on its behalf. > So, I doubt whether more coverage testing is required for this change. > > Anyway, I'm attaching a quick and dirty test program that might help > (not well tested though). I'm unable to to test it as I have modified > my setup and have to redo it. > > Justin, do you have the setup and can you get this tested? > > > > /* > * Few tests to ensure LFS (Large File Support) is working correctly. > * > * The requests (seek, truncate, write, mmap etc) that extend beyond the offset > * maximum (off_t) should succeed. > * > * Compile it with -D_FILE_OFFSET_BITS=64 > * > * Run it like: > * ./lfs-tests large_file > * > */ > #include <stdio.h> > #include <stdlib.h> > #include <string.h> > #include <fcntl.h> > #include <unistd.h> > #include <sys/types.h> > #include <sys/stat.h> > #include <sys/mman.h> > > #define BUF_SZ 32 > > #define err(msg) \ > do { perror(msg); exit(EXIT_FAILURE); } while (0) > > > void report_size(int fd, struct stat *st) > { > fstat(fd, st); > printf("size of file: %ld\n", st->st_size); > > return; > } > > int main(int argc, char *argv[]) > { > int fd; > int rc = 0; > unsigned long offset = 0; > struct stat stbuf; > char buf[BUF_SZ]; > void *file_mem; > > if (argc != 2) { > fprintf(stderr, "Usage: %s test_file\n", argv[0]); > exit(EXIT_FAILURE); > } > > fd = open(argv[1], O_RDWR | O_APPEND); > if (fd == -1) > err("open"); > > memset(buf, 0, sizeof(buf)); > > /* seek to the end of file */ > offset = lseek(fd, 0, SEEK_END); > if (offset < 0) > err("lseek"); > > /* seek past the end of file */ > offset = lseek(fd, BUF_SZ, SEEK_END); > printf("%ld\n", offset); > if (offset < 0) > err("lseek"); > > /* truncate it to < 2GB and report size */ > rc = ftruncate(fd, 2147483648 - BUF_SZ); > if (rc != 0) > err("ftruncate"); > > report_size(fd, &stbuf); > > /* truncate it to a size larger than 2GB */ > rc = ftruncate(fd, 4294967360); > if (rc != 0) > err("ftruncate"); > > report_size(fd, &stbuf); > > /* seek to the end of file */ > offset = lseek(fd, 0, SEEK_END); > if (offset < 0) > err("lseek"); > > memset(buf, 'a', sizeof(buf)); > > /* write past 4GB */ > rc = write(fd, buf, sizeof(buf)); > if (rc == -1) > err("write"); > > report_size(fd, &stbuf); > > offset = lseek(fd, 0, SEEK_END); > if (offset < 0) > err("lseek"); > > /* pwrite and pread */ > rc = pread(fd, buf, BUF_SZ, offset); > if (rc == -1) > err("pread"); > > rc = pwrite(fd, buf, sizeof(buf), offset); > if (rc == -1) > err("pwrite"); > > report_size(fd, &stbuf); > > /* page align offset */ > offset = offset & ~(sysconf(_SC_PAGE_SIZE) - 1); > > file_mem = mmap(0, BUF_SZ, PROT_WRITE, MAP_SHARED, fd, offset); > if (file_mem == MAP_FAILED) > err("mmap"); > > munmap(file_mem, 64); > > close(fd); > > return EXIT_SUCCESS; > } > Hi, so here are my test. Both system with fscache enabled. unpatched kernel (Opensuse 12.1 linux-3.1.10-1.9-default): $ ./large_file foo 8589934624 size of file: 2147483616 size of file: 4294967360 size of file: 4294967392 size of file: 4294967424 patched kernel (Gentoo Linux linux-3.4.3): $ ./large_file bar 8589934624 size of file: 2147483616 size of file: 4294967360 size of file: 4294967392 size of file: 4294967424 no crash, everything worked. Thanks justin [-- Attachment #2: OpenPGP digital signature --] [-- Type: application/pgp-signature, Size: 302 bytes --] ^ permalink raw reply [flat|nested] 7+ messages in thread
end of thread, other threads:[~2012-06-20 19:46 UTC | newest] Thread overview: 7+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2012-06-11 5:38 [PATCH] [RESEND] fs: cachefiles: Add support for large files in filesystem caching Suresh Jayaraman 2012-06-18 23:04 ` Andrew Morton 2012-06-19 5:27 ` Suresh Jayaraman 2012-06-19 7:52 ` Andrew Morton 2012-06-19 7:58 ` justin 2012-06-20 13:35 ` Suresh Jayaraman 2012-06-20 19:45 ` Justin
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.