{"id":5630,"date":"2013-05-31T22:34:38","date_gmt":"2013-06-01T02:34:38","guid":{"rendered":"http:\/\/g33kinfo.com\/info\/?p=5630"},"modified":"2013-05-31T22:34:38","modified_gmt":"2013-06-01T02:34:38","slug":"delete-millions-of-files-faster","status":"publish","type":"post","link":"https:\/\/g33kinfo.com\/info\/delete-millions-of-files-faster\/","title":{"rendered":"Delete millions of files faster"},"content":{"rendered":"

From http:\/\/linuxnote.net<\/a><\/p>\n

\n

Another Benchmark<\/h2>\n
\n

Several days ago,\u00a0Keith-Winstein<\/a>\u00a0replied at the\u00a0Quora Posts<\/a>\u00a0mentioned that my previous benchmark cannot be reproduced due to the time of all deletion operations lasting too long. To make it clear, those weird data might be that my computer was under heavy load in the past years that it may exist some fs errors during the previous benchmarks. Yet, I am not sure about it. Anyway, I got myself a relatively new rackable computer and did the benchmark again. This time I used\u00a0\/usr\/bin\/time<\/code>\u00a0that offers more detail results. Here is the new result,
\n
\n(The # of files is 1000000. Each of them has 0 size.)<\/p>\n

 <\/p>\n\n\n\n\n\n\n<\/colgroup>\n\n\n\n\n\n\n\n\n
Command<\/th>\nElapsed<\/th>\nSystem Time<\/th>\n%CPU<\/th>\ncs1<\/a>\u00a0(Vol\/Invol)<\/th>\n<\/tr>\n<\/thead>\n
rsync -a \u2013delete empty\/ a<\/td>\n10.60<\/td>\n1.31<\/td>\n95<\/td>\n106\/22<\/td>\n<\/tr>\n
find b\/ -type f -delete<\/td>\n28.51<\/td>\n14.46<\/td>\n52<\/td>\n14849\/11<\/td>\n<\/tr>\n
find c\/ -type f | xargs -L 100 rm2<\/a><\/td>\n41.69<\/td>\n20.60<\/td>\n54<\/td>\n37048\/15074<\/td>\n<\/tr>\n
find d\/ -type f | xargs -L 100 -P 100 rm2<\/a><\/td>\n34.32<\/td>\n27.82<\/td>\n89<\/td>\n929897\/21720<\/td>\n<\/tr>\n
rm -rf f<\/td>\n31.29<\/td>\n14.80<\/td>\n47<\/td>\n15134\/11<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n
\n

Original Output<\/h3>\n
\n
# method 1\n~\/test $ \/usr\/bin\/time -v  rsync -a --delete empty\/ a\/\n        Command being timed: \"rsync -a --delete empty\/ a\/\"\n        User time (seconds): 1.31\n        System time (seconds): 10.60\n        Percent of CPU this job got: 95%\n        Elapsed (wall clock) time (h:mm:ss or m:ss): 0:12.42\n        Average shared text size (kbytes): 0\n        Average unshared data size (kbytes): 0\n        Average stack size (kbytes): 0\n        Average total size (kbytes): 0\n        Maximum resident set size (kbytes): 0\n        Average resident set size (kbytes): 0\n        Major (requiring I\/O) page faults: 0\n        Minor (reclaiming a frame) page faults: 24378\n        Voluntary context switches: 106\n        Involuntary context switches: 22\n        Swaps: 0\n        File system inputs: 0\n        File system outputs: 0\n        Socket messages sent: 0\n        Socket messages received: 0\n        Signals delivered: 0\n        Page size (bytes): 4096\n        Exit status: 0\n\n# method 2\n        Command being timed: \"find b\/ -type f -delete\"\n        User time (seconds): 0.41\n        System time (seconds): 14.46\n        Percent of CPU this job got: 52%\n        Elapsed (wall clock) time (h:mm:ss or m:ss): 0:28.51\n        Average shared text size (kbytes): 0\n        Average unshared data size (kbytes): 0\n        Average stack size (kbytes): 0\n        Average total size (kbytes): 0\n        Maximum resident set size (kbytes): 0\n        Average resident set size (kbytes): 0\n        Major (requiring I\/O) page faults: 0\n        Minor (reclaiming a frame) page faults: 11749\n        Voluntary context switches: 14849\n        Involuntary context switches: 11\n        Swaps: 0\n        File system inputs: 0\n        File system outputs: 0\n        Socket messages sent: 0\n        Socket messages received: 0\n        Signals delivered: 0\n        Page size (bytes): 4096\n        Exit status: 0\n# method 3\nfind c\/ -type f | xargs -L 100 rm\n~\/test $ \/usr\/bin\/time -v .\/delete.sh\n        Command being timed: \".\/delete.sh\"\n        User time (seconds): 2.06\n        System time (seconds): 20.60\n        Percent of CPU this job got: 54%\n        Elapsed (wall clock) time (h:mm:ss or m:ss): 0:41.69\n        Average shared text size (kbytes): 0\n        Average unshared data size (kbytes): 0\n        Average stack size (kbytes): 0\n        Average total size (kbytes): 0\n        Maximum resident set size (kbytes): 0\n        Average resident set size (kbytes): 0\n        Major (requiring I\/O) page faults: 0\n        Minor (reclaiming a frame) page faults: 1764225\n        Voluntary context switches: 37048\n        Involuntary context switches: 15074\n        Swaps: 0\n        File system inputs: 0\n        File system outputs: 0\n        Socket messages sent: 0\n        Socket messages received: 0\n        Signals delivered: 0\n        Page size (bytes): 4096\n        Exit status: 0\n\n# method 4\nfind d\/ -type f | xargs -L 100 -P 100 rm\n~\/test $ \/usr\/bin\/time -v .\/delete.sh\n        Command being timed: \".\/delete.sh\"\n        User time (seconds): 2.86\n        System time (seconds): 27.82\n        Percent of CPU this job got: 89%\n        Elapsed (wall clock) time (h:mm:ss or m:ss): 0:34.32\n        Average shared text size (kbytes): 0\n        Average unshared data size (kbytes): 0\n        Average stack size (kbytes): 0\n        Average total size (kbytes): 0\n        Maximum resident set size (kbytes): 0\n        Average resident set size (kbytes): 0\n        Major (requiring I\/O) page faults: 0\n        Minor (reclaiming a frame) page faults: 1764278\n        Voluntary context switches: 929897\n        Involuntary context switches: 21720\n        Swaps: 0\n        File system inputs: 0\n        File system outputs: 0\n        Socket messages sent: 0\n        Socket messages received: 0\n        Signals delivered: 0\n        Page size (bytes): 4096\n        Exit status: 0\n\n# method 5\n~\/test $ \/usr\/bin\/time -v rm -rf f\n        Command being timed: \"rm -rf f\"\n        User time (seconds): 0.20\n        System time (seconds): 14.80\n        Percent of CPU this job got: 47%\n        Elapsed (wall clock) time (h:mm:ss or m:ss): 0:31.29\n        Average shared text size (kbytes): 0\n        Average unshared data size (kbytes): 0\n        Average stack size (kbytes): 0\n        Average total size (kbytes): 0\n        Maximum resident set size (kbytes): 0\n        Average resident set size (kbytes): 0\n        Major (requiring I\/O) page faults: 0\n        Minor (reclaiming a frame) page faults: 176\n        Voluntary context switches: 15134\n        Involuntary context switches: 11\n        Swaps: 0\n        File system inputs: 0\n        File system outputs: 0\n        Socket messages sent: 0\n        Socket messages received: 0\n        Signals delivered: 0\n        Page size (bytes): 4096\n        Exit status: 0<\/pre>\n<\/div>\n<\/div>\n<\/div>\n
\n
The Original Benchmark<\/h2>\n
\n
Yesterday, I saw a very interesting method for deleting huge number of files in a single directory. The method is provided by Zhenyu Lee at<\/p>\n
http:\/\/www.quora.com\/How-can-someone-rapidly-delete-400-000-files<\/a><\/p>\n
Instead of using\u00a0find<\/code>\u00a0and\u00a0xargs<\/code> Lee ingeniously takes the advantage of rsync that he uses\u00a0rsync \u2013delete<\/code>\u00a0to sync the target directory with an empty directory. Later, I did a comparison on various method that I’ve used. To my surprise, Lee’s method is much faster than others. The following is my benchmark,<\/p>\n
Environment:<\/p>\n