Uploaded image for project: 'eZ Publish / Platform'
  1. eZ Publish / Platform
  2. EZP-17960

Fix the 32 000 files limit that exists on binary files

    Details

      Description

      Unlike images, binary files are stored in a subfolder per content type independently of their location in the content tree. With many files, the 32000 files limit that exists on most common filesystems can easily be reached, preventing upload of new files.

      This can be fixed by splitting binary files in subdirectories, depending on their hashes, like it is done for ViewCache:

      var/plain_site/storage/original/application/ff12e1b4895bb2a92e3290e17316fe63.rar
      =>
      var/plain_site/storage/original/application/f/f/1/ff12e1b4895bb2a92e3290e17316fe63.rar

        Issue Links

          Activity

          Hide
          Bertrand Dunogier added a comment -

          ezshardbinaryfiles.php POC of a script that "shards" binary files as suggested in the issue desc.

          Show
          Bertrand Dunogier added a comment - ezshardbinaryfiles.php POC of a script that "shards" binary files as suggested in the issue desc.
          Hide
          Bertrand Dunogier added a comment -

          Note that the ezshardbinaryfiles.php attachment above is by no mean complete, and has just been uploaded in case it could be used as a draft or helper for the actual enhancement.

          Show
          Bertrand Dunogier added a comment - Note that the ezshardbinaryfiles.php attachment above is by no mean complete, and has just been uploaded in case it could be used as a draft or helper for the actual enhancement.
          Hide
          Georg Franz added a comment -

          Hi,

          the 32000 file limit only belongs to directories.

          So you can't have more than 32000 directories in one. But fortunatly you can have more than 32000 files in one dir which are not directories.

          Best wishes,
          Georg.

          Show
          Georg Franz added a comment - Hi, the 32000 file limit only belongs to directories. So you can't have more than 32000 directories in one. But fortunatly you can have more than 32000 files in one dir which are not directories. Best wishes, Georg.
          Hide
          Gaetano Giunta added a comment -

          note: using ext4 you can get to 64k

          Show
          Gaetano Giunta added a comment - note: using ext4 you can get to 64k

            People

            • Assignee:
              unknown
              Reporter:
              Bertrand Dunogier
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated: