saad/FindDuplicateFiles
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
FindDuplicateFiles/README.md
saad 61aa5de173 Add functioning program
2026-08-29 04:31:57 -04:00

8.7 KiB
Raw Permalink Blame History

Duplicate File Finder (Java - MD5 Hashtable)

Finds duplicate files by content using MD5 hashing. Uses a Hashtable<String, List<String>> where MD5 hash -> list of filenames. Groups with 2+ files are reported as duplicates.

The core is a command-line program that can be used standalone, wrapped by a Swing GUI with tree view and selective deletion.

Last updated: 2026-08-29 - NPE fix for tree renderer + README refresh

Files

  • DuplicateFinder.java - CLI core + hashtable logic. Can run standalone. (Hashtable<String, List<String>> at DuplicateFinder.java:114, filterDuplicates at DuplicateFinder.java:145)
  • DuplicateFinderGUI.java - Swing GUI wrapper (tree view + delete) that reuses DuplicateFinder's hashtable logic. (CheckBoxTreeCellRenderer at DuplicateFinderGUI.java:347, CheckedNode at DuplicateFinderGUI.java:56, startScan at DuplicateFinderGUI.java:474)
  • build.bat / run.bat - Windows helpers
  • build.sh / run.sh - bash helpers (Linux/macOS/WSL/Git Bash)
  • README.md - this file

Requirements

  • Java 8+ (tested on Temurin 1.8.0_504). Compatible with Java 8, 11, 17, 21. No external deps.

Build

javac DuplicateFinder.java DuplicateFinderGUI.java

Or:

build.bat        :: Windows
./build.sh       :: bash (Linux/macOS/WSL/Git Bash) - chmod +x build.sh first

Verify:

java DuplicateFinder --help
java DuplicateFinderGUI
:: bash:
./run.sh --help
./run.sh         :: launches GUI

CLI Usage

java DuplicateFinder [options] <directory>
java DuplicateFinder --gui          # launch GUI
java DuplicateFinder --help

java DuplicateFinderGUI             # launch GUI directly

Options:

  • -r, --recursive recurse into subdirectories (default)
  • --no-recursive only top-level directory
  • -h, --help show help
  • --gui launch Swing GUI

Examples:

java DuplicateFinder C:\Users\Me\Documents
java DuplicateFinder --no-recursive C:\Temp
java DuplicateFinder --gui

Output example:

Scanning: C:\Users\Me\Documents
Recursive: true

=== Duplicate Group 1 ===
MD5: 3579c8da7f1e0ad94656e76c886e5125  (3 files)
  C:\...\a.txt  (13 bytes)
  C:\...\b.txt  (13 bytes)
  C:\...\sub\d.txt  (13 bytes)

Summary: 1 duplicate group(s), 3 files involved.
Scanned in 43 ms.

Exit codes: 0 success, 1 error (bad args / missing path).

GUI Usage

java DuplicateFinderGUI
:: or
java DuplicateFinder --gui
java DuplicateFinderGUI "C:\path\to\start\dir"

GUI Features

  • Directory text field + Browse button (JFileChooser - directories only)
  • Recursive checkbox
  • Find Duplicates / Cancel / Clear Results buttons
  • Progress bar + status label (SwingWorker background thread, doesn't freeze UI)
  • Tree view (JTree with checkboxes, DefaultTreeModel + CheckedNode):
    • Root = scan summary (invisible, rootVisible=false), children = duplicate groups, leaves = files
    • Group node: Group 1 - 3 files - 13 bytes each - MD5: abc... (bold, directory icon)
    • File node: filename [parent folder] (size bytes) with checkbox (file icon, red when checked)
    • Click checkbox or press Space on selected node to mark for deletion; group checkbox toggles all children (setNodeChecked at DuplicateFinderGUI.java:423)
    • Custom CheckBoxTreeCellRenderer shows JCheckBox + JLabel per row (DuplicateFinderGUI.java:347)
  • Tree actions bar:
    • Auto-Select (keep 1 per group) - checks all but first file in each group (safe default, DuplicateFinderGUI.java:538)
    • Clear Checks - unchecks all (DuplicateFinderGUI.java:561)
    • Delete Checked Files - deletes with confirmation (shows count + bytes, warns if all in group checked, DuplicateFinderGUI.java:651)
    • Open Location - opens containing folder via Desktop.getDesktop().open() / explorer
    • Expand All / Collapse All (DuplicateFinderGUI.java:571)
    • Label: N file(s) checked, X bytes to free (updateCheckedInfo at DuplicateFinderGUI.java:636)
  • Context popup (right-click): Auto-Select / Clear / Delete / Open Location / Copy Path
  • Menu bar: File/Browse/Exit, Edit/Auto-Select/Delete, View/Expand/Collapse, Help/About
  • Selection listener shows full path + size in status bar

Deletion Flow

  1. Scan -> tree populated, groups expanded (displayResultsTree at DuplicateFinderGUI.java:862, expandAll).
  2. Use Auto-Select or manually check unwanted duplicates (mouse click on row or Space).
  3. Review Delete Checked Files confirmation dialog:
    • Lists files to delete (first 15 + count), total bytes, warnings if entire group selected.
    • Requires YES confirmation (permanent File.delete() with setWritable(true) fallback).
  4. On success: tree nodes removed via treeModel.removeNodeFromParent(), empty/single groups pruned, summary updated, status shows Deleted N file(s).
  5. On failure (locked/read-only/not found): dialog lists failures, partial successes kept.

Safety: Auto-Select keeps 1 copy per group. Manual delete warns if you check all files in a group.

The GUI calls DuplicateFinder.buildHashTableWithProgress() with a ProgressCallback to update the progress bar without freezing the EDT.

How it works

  1. collectFiles(root, recursive) (DuplicateFinder.java:78) walks the file tree (recursive DFS).
  2. For each file, computeMD5(File) (DuplicateFinder.java:42) uses MessageDigest.getInstance("MD5") + 8KB buffered FileInputStream to compute hex hash.
  3. Hashtable population (DuplicateFinder.java:114): 
    Hashtable<String, List<String>> table = new Hashtable<>();
// md5 -> list
List<String> list = table.get(md5);
if (list == null) { list = new ArrayList<>(); table.put(md5, list); }
list.add(file.getAbsolutePath());
  4. filterDuplicates() (DuplicateFinder.java:145) keeps only entries where list.size() > 1.
  5. CLI prints sorted groups; GUI builds DefaultTreeModel with CheckedNode user objects and CheckBoxTreeCellRenderer (DuplicateFinderGUI.java:347). Groups sorted by hash, files sorted by path.

Empty files (MD5 d41d8cd98f00b204e9800998ecf8427e) are considered duplicates if multiple empties exist. Unreadable files are skipped with warning.

Testing

Create a test_data example:

mkdir test_data\sub
echo Hello World > test_data\a.txt
copy test_data\a.txt test_data\b.txt
copy test_data\a.txt test_data\sub\d.txt
echo Different > test_data\c.txt
copy /b nul test_data\empty1.txt
copy /b nul test_data\empty2.txt
  • a.txt, b.txt, sub\d.txt all "Hello World" -> one group of 3
  • empty1.txt, empty2.txt -> one group of 2 (if empties present)
  • bin1.dat, sub\bin2.dat (bytes 0-255) -> one group of 2 if created

Run:

java DuplicateFinder test_data           # expects 3 groups (or 1 if minimal set above)
java DuplicateFinder --no-recursive test_data  # expects fewer groups (sub excluded)
# GUI: scan test_data, Auto-Select, Delete -> leaves 1 per group, re-scan shows 1 group then 0

Manual NPE regression test (fixed 2026-08-29):

javac -d . DuplicateFinder.java DuplicateFinderGUI.java
java DuplicateFinderGUI
# Click Find Duplicates immediately -> before fix threw NPE at DuplicateFinderGUI.java:391
# After fix: shows "Scanning: ..." placeholder without crash, then tree populates

Changelog

  • 2026-08-29 (fix): DuplicateFinderGUI.java:391 NPE in CheckBoxTreeCellRenderer.getTreeCellRendererComponent() when data.md5==null for placeholder root nodes ("Scanning...", "No duplicates...", "Cancelled"). Added null guard for md5 (length check + empty). Also guarded DuplicateFinderGUI.java:670 warning shortMd5 substring. Verified via RendererTest + ScanTest harness; CLI unaffected.
  • 2026-08-29: Added tree view with checkboxes, selective delete, auto-select keep-one, open location, expand/collapse, context menu.
  • 2026-08-28: Initial CLI + basic GUI (JTextArea results).

Troubleshooting

  • NullPointerException at CheckBoxTreeCellRenderer.getTreeCellRendererComponent(DuplicateFinderGUI.java:391): fixed in latest version. Update DuplicateFinderGUI.java or re-clone. Recompile with javac.
  • javac not found: install JDK 8+ (Temurin 8 used for testing at C:\Program Files\Eclipse Adoptium\jdk-8.0.504.1-hotspot\bin\javac.exe) and add to PATH or use build.bat which auto-adds that path.
  • HeadlessException when running GUI on headless server: GUI requires display. Use CLI java DuplicateFinder <dir> instead or run with xvfb/X forwarding.
  • Files not deleted: checked for File.delete() returning false (locked/read-only). Close programs using the file, then retry. GUI retries with setWritable(true).

Notes

  • Uses Hashtable specifically (as required), not HashMap.
  • Large files are streamed, not loaded fully into memory.
  • CLI is read-only (listing only); GUI deletion is opt-in with confirmation and prunes non-duplicate groups.