Uploaded image for project: 'eZ Publish / Platform'
  1. eZ Publish / Platform
  2. EZP-20813

eZDFS cleanup: Allow detection of NFS and DB availability to prevent accidental removal of files.

    Details

      Description

      The DFS cleanup script dfscleanup.php has options to check both the database and NFS share for existing files, and remove "stale" data in its counterpart.

      However, in case of failure of either, files/records will be (incorrectly) removed as they are not found.

      Steps to reproduce:
      • Run dfscleanup.php script
      • Disconnect the NFS mount point

      Files will be removed from the database.

      Suggested enhancement:
      • Either by default, or using a parameter/flag, check for general availability of the mount point (as well as database) if a file is not found.
      • On the NFS share, this could be done, for example, by checking for the existence of an existing file or directory (var dir?) when a file is not found.
      • If the NFS or DB are not available, the script could then terminate without further processing.

        Issue Links

          Activity

          Hide
          Bertrand Dunogier added a comment - - edited

          @QA: could you test the PR ? The change adds an exception at quite a low level, and there might be side effects.

          Sanity checks on DFS/MySQLi basically.

          Please make sure you send it back to me once the tests are done, it isn't merged at all.

          Show
          Bertrand Dunogier added a comment - - edited @QA: could you test the PR ? The change adds an exception at quite a low level, and there might be side effects. Sanity checks on DFS/MySQLi basically. Please make sure you send it back to me once the tests are done, it isn't merged at all.
          Hide
          Bertrand Dunogier added a comment - - edited

          On the tests I have made, all files that were listed were at least expired. dfscleanup.php has always removed expired files.

          Show
          Bertrand Dunogier added a comment - - edited -D alone isn't supposed to do anything: it can only be used either with -B or -S. I have committed to the PR a small change that improves the message you get in that case a bit ( https://github.com/ezsystems/ezpublish-legacy/commit/59da3df ) -S was indeed flawed, since I had left a debug statement... sorry about that. Removed now ( https://github.com/ezsystems/ezpublish-legacy/commit/e73a65d7a2695a2e8b820039323703b75b16aca5 ). This fgets( STDIN ) was used to pause the script so that I could simulate the DB or NFS breaking down. On the tests I have made, all files that were listed were at least expired. dfscleanup.php has always removed expired files.
          Hide
          Bertrand Dunogier added a comment -

          Ok. I guess we can only test without and with the patch, to check that it fixes what it is supposed to fix.

          Basically, if either nfs or the db become unavailable, it must stop running without altering any file.

          Show
          Bertrand Dunogier added a comment - Ok. I guess we can only test without and with the patch, to check that it fixes what it is supposed to fix. Basically, if either nfs or the db become unavailable, it must stop running without altering any file.
          Hide
          Joao Pingo (Inactive) added a comment - - edited

          @Bertrand
          I've finished the test with your 3 patch's and all worked as supposed to
          Without the nfs present i get "DFS mount seems to be gone, aborting" and no files or database records are removed
          Without the db i get "An unexpected error has occurred. Please contact the webmaster" and no files or database records are removed
          The above mentioned issues are present

          Show
          Joao Pingo (Inactive) added a comment - - edited @Bertrand I've finished the test with your 3 patch's and all worked as supposed to Without the nfs present i get "DFS mount seems to be gone, aborting" and no files or database records are removed Without the db i get "An unexpected error has occurred. Please contact the webmaster" and no files or database records are removed The above mentioned issues are present
          Hide
          Bertrand Dunogier added a comment -

          Hmmmm, I'm not sure this is what you are supposed to get when the DB falls down. I'll check what is to be expected.

          Regarding the issues above, they're indeed valid, but totally unrelated.

          Show
          Bertrand Dunogier added a comment - Hmmmm, I'm not sure this is what you are supposed to get when the DB falls down. I'll check what is to be expected. Regarding the issues above, they're indeed valid, but totally unrelated.
          Hide
          Bertrand Dunogier added a comment -

          This is what I get if I kill the DB in the middle of the foreach loop that starts at bin/php/dfscleanup.php:107:

          $ php bin/php/dfscleanup.php -S
          Performing cleanup on directory <var/ezdemo_site>.
          Checking files registered in the database...
           
          Database error, aborting.
          dfs/mysqli DB error: MySQL server has gone away
          SQL Query: SELECT name, mtime FROM ezdfsfile WHERE name_hash=MD5('var/ezdemo_site/storage/images/media/images/mount-annapurna/536-1-eng-GB/mount-Annapurna_gallery.jpg')

          The Unexpected error implies that an exception wasn't caught, and this shouldn't happen here. Could you re-run the test that fails this way but with the -d option, so that we see what actually fails ?

          Thank you.

          Show
          Bertrand Dunogier added a comment - This is what I get if I kill the DB in the middle of the foreach loop that starts at bin/php/dfscleanup.php:107 : $ php bin/php/dfscleanup.php -S Performing cleanup on directory <var/ezdemo_site>. Checking files registered in the database...   Database error, aborting. dfs/mysqli DB error: MySQL server has gone away SQL Query: SELECT name, mtime FROM ezdfsfile WHERE name_hash=MD5('var/ezdemo_site/storage/images/media/images/mount-annapurna/536-1-eng-GB/mount-Annapurna_gallery.jpg') The Unexpected error implies that an exception wasn't caught, and this shouldn't happen here. Could you re-run the test that fails this way but with the -d option, so that we see what actually fails ? Thank you.
          Hide
          Joao Pingo (Inactive) added a comment - - edited

          If the database stops when the script is running i'm getting the same result as you...
          My older result is present if the database is stoped before the script is run's, with -d i get

          An unexpected error has occurred. Please contact the webmaster.
          Error 2003: Can't connect to MySQL server on '192.168.2.220' (111) in /var/www/apache2php53/ezp5/ezpublish_testsystem/.run/kernel/private/classes/clusterfilehandlers/dfsbackends/mysqli.php on line 158
          

          Show
          Joao Pingo (Inactive) added a comment - - edited If the database stops when the script is running i'm getting the same result as you... My older result is present if the database is stoped before the script is run's, with -d i get An unexpected error has occurred. Please contact the webmaster. Error 2003: Can't connect to MySQL server on '192.168.2.220' (111) in /var/www/apache2php53/ezp5/ezpublish_testsystem/.run/kernel/private/classes/clusterfilehandlers/dfsbackends/mysqli.php on line 158
          Hide
          Joaquim Cavalleri (Inactive) added a comment -

          @Joao Pingo Could your unexpected error result of the database that died not being the DFS database but the main DB instead? Did you ran the test with two separate DB instances and killed DFS only?

          Show
          Joaquim Cavalleri (Inactive) added a comment - @ Joao Pingo Could your unexpected error result of the database that died not being the DFS database but the main DB instead? Did you ran the test with two separate DB instances and killed DFS only?
          Hide
          Bertrand Dunogier added a comment -

          Note that the error happens in dfshandlers/mysqli.php, it can't be an error from the eZ DB.

          Show
          Bertrand Dunogier added a comment - Note that the error happens in dfshandlers/mysqli.php , it can't be an error from the eZ DB.
          Show
          Yannick Roger (Inactive) added a comment - Fixed in master: https://github.com/ezsystems/ezpublish-legacy/commit/766ee0022fc93e3499aa44ebf329ff7a79cb0bad
          Hide
          Joao Pingo (Inactive) added a comment -

          QA Approved

          Show
          Joao Pingo (Inactive) added a comment - QA Approved

            People

            • Assignee:
              Unassigned
              Reporter:
              Joao Inacio (Inactive)
            • Votes:
              0 Vote for this issue
              Watchers:
              11 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Time Tracking

                Estimated:
                Original Estimate - Not Specified
                Not Specified
                Remaining:
                Remaining Estimate - 0 minutes
                0m
                Logged:
                Time Spent - 1 week, 3 days, 4 hours, 15 minutes
                1w 3d 4h 15m