fix(mine): log warning when files exceed MAX_FILE_SIZE (#923)#924
Open
mvalentsev wants to merge 2 commits intoMemPalace:developfrom
Open
fix(mine): log warning when files exceed MAX_FILE_SIZE (#923)#924mvalentsev wants to merge 2 commits intoMemPalace:developfrom
mvalentsev wants to merge 2 commits intoMemPalace:developfrom
Conversation
igorls
added
bug
mvalentsev
requested review from
bensig,
igorls and
milla-jovovich
as code owners
Both miner.py and convo_miner.py silently skip files larger than the 10 MB limit with a bare continue. This is especially painful for conversation mining where long Claude/ChatGPT exports routinely exceed 10 MB and vanish with no trace. Print a SKIP warning per oversized file, matching the existing format in split_mega_files.py.
mvalentsev
force-pushed
the
fix/mine-log-oversized-skips
branch
from
75e47c9 to
2dcdf18
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
mempalace mineandmempalace mine --mode convossilently skip fileslarger than the 10 MB
MAX_FILE_SIZEwith a barecontinue. No log, nocounter, exit code 0. The output is indistinguishable from a directory
that legitimately had no mineable files.
This is especially painful for
--mode convos, where long Claude/ChatGPTexports routinely exceed 10 MB and silently vanish.
Prints a SKIP warning per oversized file, matching the format already used
in
split_mega_files.py:Changes
mempalace/miner.py: log SKIP warning inscan_project()when a fileexceeds
MAX_FILE_SIZEmempalace/convo_miner.py: same fix inscan_convos()Test plan
pytest tests/test_miner.py::test_scan_project_skips_oversized_filespytest tests/test_convo_miner_unit.py::TestScanConvos::test_scan_skips_oversized_filesruff check: cleanCloses #923