Uploaded image for project: 'eZ Publish / Platform'
  1. eZ Publish / Platform
  2. EZP-18508

Need cleanup of abandoned/broken cluster data

    Details

      Description

      The clusterpurge script and cronjob can delete various data stored on a cluster. What it can't do is to clear up broken cluster data:

      • ezdfsfile rows for which the file has been deleted
      • files for which the ezdfsfile row has been deleted
        Both are worthless to keep around, and should be deleted.

      How they appear is unknown and may have various causes, one possibility is PHP crashes during cache building, or timeouts on file operations. However, they seem to accumulate on some sites, and should be dealt with.

      The existing "cluster_maintenance" cronjob part is a suitable place for this. EDIT: A bin/php CLI script would be better.

        Activity

        Hide
        Gaetano Giunta added a comment -

        -1 for putting this in a cronjob: these things should not happen, and when they happen we should be able to track down the problem.

        Make instead a one-off cli script that can both check, report and repair any inconsistencies:

        • inexisting files (either in db or fs)
        • wrong size
        • wrong timestamp (is this supposed to be in sync?)

        The same script should also be able to check for inconsistency of the ezdb tables (there it can only check the reported size of file vs. its chunk rows)

        Show
        Gaetano Giunta added a comment - -1 for putting this in a cronjob: these things should not happen, and when they happen we should be able to track down the problem. Make instead a one-off cli script that can both check, report and repair any inconsistencies: inexisting files (either in db or fs) wrong size wrong timestamp (is this supposed to be in sync?) The same script should also be able to check for inconsistency of the ezdb tables (there it can only check the reported size of file vs. its chunk rows)
        Show
        Damien Pobel (eZ) added a comment - Implemented in ezpublish master https://github.com/ezsystems/ezpublish/commit/fc6cadc4092cfb44ad989704b770c05597819905
        Hide
        Damien Pobel (eZ) added a comment - - edited

        @Doc a new script has been added: bin/php/dfscleanup.php
        it allows to clean a DFS cluster install from stale files in DB that do not exist on NFS and the opposite too.
        Options:
        -S Check files on DFS share against files in the database
        -B Checks files in database against files on DFS share
        -D Delete nonexistent files

        It is recommended to first run it WITHOUT the -D swtich to see which files are going to be removed.

        Show
        Damien Pobel (eZ) added a comment - - edited @Doc a new script has been added: bin/php/dfscleanup.php it allows to clean a DFS cluster install from stale files in DB that do not exist on NFS and the opposite too. Options: -S Check files on DFS share against files in the database -B Checks files in database against files on DFS share -D Delete nonexistent files It is recommended to first run it WITHOUT the -D swtich to see which files are going to be removed.
        Hide
        Geir Arne Waaler added a comment -

        Doc has been added.

        Geir Arne Waaler
        eZ Documentation

        Show
        Geir Arne Waaler added a comment - Doc has been added. Geir Arne Waaler eZ Documentation
        Hide
        Geir Arne Waaler added a comment - - edited

        Documented in version 4.x, 4.3, 4.4, 4.5 and the new 4.6.

        http://doc.ez.no/eZ-Publish/Technical-manual/4.5

        Geir Arne Waaler
        eZ Documentation

        Show
        Geir Arne Waaler added a comment - - edited Documented in version 4.x, 4.3, 4.4, 4.5 and the new 4.6. http://doc.ez.no/eZ-Publish/Technical-manual/4.5 Geir Arne Waaler eZ Documentation
        Hide
        Geir Arne Waaler added a comment -

        Sorry, the link was for the eZ Publish admin. Try the link above now!

        Show
        Geir Arne Waaler added a comment - Sorry, the link was for the eZ Publish admin. Try the link above now!
        Hide
        Gaetano Giunta added a comment -

        ps: while at it, script should cehck also for differences between var and dfsvar...

        Show
        Gaetano Giunta added a comment - ps: while at it, script should cehck also for differences between var and dfsvar...

          People

          • Assignee:
            Damien Pobel (eZ)
            Reporter:
            (inactive) Gunnstein Lye
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved: