Finds duplicate files by content using MD5 hashing. Uses a Hashtable<String, List<String>> where MD5 hash -> list of filenames. Groups with 2+ files are reported as duplicates.

The core is a command-line program that can be used standalone, wrapped by a Swing GUI with tree view and selective deletion.

Last updated: 2026-08-29 - NPE fix for tree renderer + README refresh

Files

DuplicateFinder.java - CLI core + hashtable logic. Can run standalone. (Hashtable<String, List<String>> at DuplicateFinder.java:114, filterDuplicates at DuplicateFinder.java:145)
DuplicateFinderGUI.java - Swing GUI wrapper (tree view + delete) that reuses DuplicateFinder's hashtable logic. (CheckBoxTreeCellRenderer at DuplicateFinderGUI.java:347, CheckedNode at DuplicateFinderGUI.java:56, startScan at DuplicateFinderGUI.java:474)
build.bat / run.bat - Windows helpers
build.sh / run.sh - bash helpers (Linux/macOS/WSL/Git Bash)
README.md - this file

Requirements

Java 8+ (tested on Temurin 1.8.0_504). Compatible with Java 8, 11, 17, 21. No external deps.

Build

javac DuplicateFinder.java DuplicateFinderGUI.java

Or:

build.bat        :: Windows
./build.sh       :: bash (Linux/macOS/WSL/Git Bash) - chmod +x build.sh first

Verify:

java DuplicateFinder --help
java DuplicateFinderGUI
:: bash:
./run.sh --help
./run.sh         :: launches GUI

CLI Usage

java DuplicateFinder [options] <directory>
java DuplicateFinder --gui          # launch GUI
java DuplicateFinder --help

java DuplicateFinderGUI             # launch GUI directly

Options:

-r, --recursive recurse into subdirectories (default)
--no-recursive only top-level directory
-h, --help show help
--gui launch Swing GUI

Examples:

java DuplicateFinder C:\Users\Me\Documents
java DuplicateFinder --no-recursive C:\Temp
java DuplicateFinder --gui

Output example:

Scanning: C:\Users\Me\Documents
Recursive: true

=== Duplicate Group 1 ===
MD5: 3579c8da7f1e0ad94656e76c886e5125  (3 files)
  C:\...\a.txt  (13 bytes)
  C:\...\b.txt  (13 bytes)
  C:\...\sub\d.txt  (13 bytes)

Summary: 1 duplicate group(s), 3 files involved.
Scanned in 43 ms.

Exit codes: 0 success, 1 error (bad args / missing path).

GUI Usage

java DuplicateFinderGUI
:: or
java DuplicateFinder --gui
java DuplicateFinderGUI "C:\path\to\start\dir"

GUI Features

Directory text field + Browse button (JFileChooser - directories only)
Recursive checkbox
Find Duplicates / Cancel / Clear Results buttons
Progress bar + status label (SwingWorker background thread, doesn't freeze UI)
Tree view (JTree with checkboxes, DefaultTreeModel + CheckedNode):
- Root = scan summary (invisible, rootVisible=false), children = duplicate groups, leaves = files
- Group node: Group 1 - 3 files - 13 bytes each - MD5: abc... (bold, directory icon)
- File node: filename [parent folder] (size bytes) with checkbox (file icon, red when checked)
- Click checkbox or press Space on selected node to mark for deletion; group checkbox toggles all children (setNodeChecked at DuplicateFinderGUI.java:423)
- Custom CheckBoxTreeCellRenderer shows JCheckBox + JLabel per row (DuplicateFinderGUI.java:347)
Tree actions bar:
- Auto-Select (keep 1 per group) - checks all but first file in each group (safe default, DuplicateFinderGUI.java:538)
- Clear Checks - unchecks all (DuplicateFinderGUI.java:561)
- Delete Checked Files - deletes with confirmation (shows count + bytes, warns if all in group checked, DuplicateFinderGUI.java:651)
- Open Location - opens containing folder via Desktop.getDesktop().open() / explorer
- Expand All / Collapse All (DuplicateFinderGUI.java:571)
- Label: N file(s) checked, X bytes to free (updateCheckedInfo at DuplicateFinderGUI.java:636)
Context popup (right-click): Auto-Select / Clear / Delete / Open Location / Copy Path
Menu bar: File/Browse/Exit, Edit/Auto-Select/Delete, View/Expand/Collapse, Help/About
Selection listener shows full path + size in status bar

Deletion Flow

Scan -> tree populated, groups expanded (displayResultsTree at DuplicateFinderGUI.java:862, expandAll).
Use Auto-Select or manually check unwanted duplicates (mouse click on row or Space).
Review Delete Checked Files confirmation dialog:
- Lists files to delete (first 15 + count), total bytes, warnings if entire group selected.
- Requires YES confirmation (permanent File.delete() with setWritable(true) fallback).
On success: tree nodes removed via treeModel.removeNodeFromParent(), empty/single groups pruned, summary updated, status shows Deleted N file(s).
On failure (locked/read-only/not found): dialog lists failures, partial successes kept.

Safety: Auto-Select keeps 1 copy per group. Manual delete warns if you check all files in a group.

The GUI calls DuplicateFinder.buildHashTableWithProgress() with a ProgressCallback to update the progress bar without freezing the EDT.

How it works

collectFiles(root, recursive) (DuplicateFinder.java:78) walks the file tree (recursive DFS).
For each file, computeMD5(File) (DuplicateFinder.java:42) uses MessageDigest.getInstance("MD5") + 8KB buffered FileInputStream to compute hex hash.

Hashtable population (DuplicateFinder.java:114):

Hashtable<String, List<String>> table = new Hashtable<>();
// md5 -> list
List<String> list = table.get(md5);
if (list == null) { list = new ArrayList<>(); table.put(md5, list); }
list.add(file.getAbsolutePath());

filterDuplicates() (DuplicateFinder.java:145) keeps only entries where list.size() > 1.
CLI prints sorted groups; GUI builds DefaultTreeModel with CheckedNode user objects and CheckBoxTreeCellRenderer (DuplicateFinderGUI.java:347). Groups sorted by hash, files sorted by path.

Empty files (MD5 d41d8cd98f00b204e9800998ecf8427e) are considered duplicates if multiple empties exist. Unreadable files are skipped with warning.

Testing

Create a test_data example:

mkdir test_data\sub
echo Hello World > test_data\a.txt
copy test_data\a.txt test_data\b.txt
copy test_data\a.txt test_data\sub\d.txt
echo Different > test_data\c.txt
copy /b nul test_data\empty1.txt
copy /b nul test_data\empty2.txt

a.txt, b.txt, sub\d.txt all "Hello World" -> one group of 3
empty1.txt, empty2.txt -> one group of 2 (if empties present)
bin1.dat, sub\bin2.dat (bytes 0-255) -> one group of 2 if created

Run:

java DuplicateFinder test_data           # expects 3 groups (or 1 if minimal set above)
java DuplicateFinder --no-recursive test_data  # expects fewer groups (sub excluded)
# GUI: scan test_data, Auto-Select, Delete -> leaves 1 per group, re-scan shows 1 group then 0

Manual NPE regression test (fixed 2026-08-29):

javac -d . DuplicateFinder.java DuplicateFinderGUI.java
java DuplicateFinderGUI
# Click Find Duplicates immediately -> before fix threw NPE at DuplicateFinderGUI.java:391
# After fix: shows "Scanning: ..." placeholder without crash, then tree populates

Changelog

2026-08-29 (fix): DuplicateFinderGUI.java:391 NPE in CheckBoxTreeCellRenderer.getTreeCellRendererComponent() when data.md5==null for placeholder root nodes ("Scanning...", "No duplicates...", "Cancelled"). Added null guard for md5 (length check + empty). Also guarded DuplicateFinderGUI.java:670 warning shortMd5 substring. Verified via RendererTest + ScanTest harness; CLI unaffected.
2026-08-29: Added tree view with checkboxes, selective delete, auto-select keep-one, open location, expand/collapse, context menu.
2026-08-28: Initial CLI + basic GUI (JTextArea results).

Troubleshooting

NullPointerException at CheckBoxTreeCellRenderer.getTreeCellRendererComponent(DuplicateFinderGUI.java:391): fixed in latest version. Update DuplicateFinderGUI.java or re-clone. Recompile with javac.
javac not found: install JDK 8+ (Temurin 8 used for testing at C:\Program Files\Eclipse Adoptium\jdk-8.0.504.1-hotspot\bin\javac.exe) and add to PATH or use build.bat which auto-adds that path.
HeadlessException when running GUI on headless server: GUI requires display. Use CLI java DuplicateFinder <dir> instead or run with xvfb/X forwarding.
Files not deleted: checked for File.delete() returning false (locked/read-only). Close programs using the file, then retry. GUI retries with setWritable(true).

Notes

Uses Hashtable specifically (as required), not HashMap.
Large files are streamed, not loaded fully into memory.
CLI is read-only (listing only); GUI deletion is opt-in with confirmation and prunes non-duplicate groups.