Introduction - If you have any usage issues, please Google them yourself
Text file deduplication tool, at work we always have some files have duplicate content in more and larger files when handling them more trouble, they wrote a small tool, the test found that in more than 30 million files after the trip, the efficiency will be very slow.