Details

      Description

      Option for returning unlimited number of search hits needs to be removed.

      While this worked with relational database (Legacy Search Engine), Solr and Elasticsearch were never made to provide for this. Using big number to fake unlimited (1073741824 for Legacy Search Engine) routinely causes OOM exceptions with Solr and Elasticsearch.

      Solr and Elasticsearch were made to work with returning relatively small number of hits, providing paging/scrolling for the case when full set is required.

      Null as an alias for unlimited in Query objects should be removed, but also other service methods that rely on search should be updated.
      Limit should always be expected to be set to a definite number.

        Issue Links

          Activity

          Petar Spanja (Inactive) created issue -
          André Rømcke made changes -
          Field Original Value New Value
          Rank Ranked higher
          Petar Spanja (Inactive) made changes -
          Summary Fix size (limit) parameter for unlimited Elasticsearch: Fix size (limit) parameter for unlimited
          Petar Spanja (Inactive) made changes -
          Workflow eZ Engineering Kanban SubTasks Workflow [ 64670 ] eZ Engineering Scrumban Workflow [ 64830 ]
          Parent EZP-23329 [ 42305 ]
          Issue Type Sub-task [ 5 ] Bug [ 1 ]
          Petar Spanja (Inactive) made changes -
          Link This issue relates to EZP-23329 [ EZP-23329 ]
          Hide
          Petar Spanja (Inactive) added a comment -

          It seems unlimited size is not possible in Elasticsearch, so we might want to add default limit value to the API Query object.

          Show
          Petar Spanja (Inactive) added a comment - It seems unlimited size is not possible in Elasticsearch, so we might want to add default limit value to the API Query object.
          André Rømcke made changes -
          Issue Type Bug [ 1 ] Improvement [ 4 ]
          André Rømcke made changes -
          Assignee Petar Spanja [ petar.spanja@ez.no ]
          André Rømcke made changes -
          Status InputQ [ 10001 ] Backlog [ 10000 ]
          André Rømcke made changes -
          Epic Link EZP-22944 [ 32101 ]
          Petar Spanja (Inactive) logged work - 12/May/15 2:00 AM
          • Time Spent:
            1 hour, 30 minutes
             

            fixing

          Hide
          Petar Spanja (Inactive) added a comment -

          Also valid for Solr.

          Elasticsearch and Solr were not built for returning complete set of search hits. Current limit in service is set to 1073741824.
          That works with relational database, but with Solr (particularly multicore) and Elasticsearch causes OOM exceptions.

          Show
          Petar Spanja (Inactive) added a comment - Also valid for Solr. Elasticsearch and Solr were not built for returning complete set of search hits. Current limit in service is set to 1073741824. That works with relational database, but with Solr (particularly multicore) and Elasticsearch causes OOM exceptions.
          Petar Spanja (Inactive) made changes -
          Summary Elasticsearch: Fix size (limit) parameter for unlimited Search: Fix size (limit) parameter for unlimited
          Petar Spanja (Inactive) made changes -
          Status Backlog [ 10000 ] InputQ [ 10001 ]
          Petar Spanja (Inactive) made changes -
          Status InputQ [ 10001 ] Development [ 3 ]
          Petar Spanja (Inactive) made changes -
          Summary Search: Fix size (limit) parameter for unlimited Remove unlimited search hits option
          Petar Spanja (Inactive) made changes -
          Description ATM {{size}} for unspecified limit is hardcoded to {{1000}}, as high number seem to cause memory problems.

          It needs to be investigated if unlimited value can be set, or we can set the default limit on the {{Query}} value object.
          Option for returning unlimited number of search hits needs to be removed.

          While this worked with relational database (Legacy Search Engine), Solr and Elasticsearch were never made to provide for this. Using big number to fake unlimited (1073741824 for Legacy Search Engine) routinely causes OOM exceptions with Solr and Elasticsearch.

          Solr and Elasticsearch were made to work with returning relatively small number of hits, providing paging/scrolling for the case when full set is required.

          Null as an alias for unlimited in Query objects should be removed, but also other service methods that rely on search should be updated.
          Limit should always be expected to be set to a definite number.
          Show
          Petar Spanja (Inactive) added a comment - Pull request: https://github.com/ezsystems/ezpublish-kernel/pull/1257
          Petar Spanja (Inactive) logged work - 13/May/15 5:56 PM
          • Time Spent:
            1 hour, 45 minutes
             

            fixing

          Show
          Petar Spanja (Inactive) added a comment - Merged to ezpublish-kernel/master in https://github.com/ezsystems/ezpublish-kernel/commit/8e6e38e1589ee75085dba38e9b5cc450d4224fb6
          Petar Spanja (Inactive) made changes -
          Status Development [ 3 ] Documentation Review done [ 10011 ]
          Affects Version/s 5.4.2 [ 13979 ]
          Fix Version/s Pollux [ 13981 ]
          Rui Silva (Inactive) made changes -
          Status Documentation Review done [ 10011 ] QA [ 10008 ]
          Assignee Petar Spanja [ petar.spanja@ez.no ] Rui Silva [ rui.silva@ez.no ]
          Hide
          Rui Silva (Inactive) added a comment -

          Tested and approved by QA for master.

          Show
          Rui Silva (Inactive) added a comment - Tested and approved by QA for master.
          Rui Silva (Inactive) made changes -
          Assignee Rui Silva [ rui.silva@ez.no ]
          Status QA [ 10008 ] Closed [ 6 ]
          Resolution Fixed [ 1 ]
          André Rømcke made changes -
          Workflow eZ Engineering Scrumban Workflow [ 64830 ] EZ* Development Workflow [ 85385 ]
          Petar Spanja (Inactive) made changes -
          Resolution Fixed [ 1 ]
          Status Closed [ 6 ] Reopened [ 4 ]
          Petar Spanja (Inactive) made changes -
          Remaining Estimate 0 minutes [ 0 ]
          Time Spent 1 hour, 45 minutes [ 6300 ]
          Worklog Id 55248 [ 55248 ]
          Petar Spanja (Inactive) made changes -
          Status Reopened [ 4 ] Closed [ 6 ]
          Resolution Fixed [ 1 ]
          Petar Spanja (Inactive) made changes -
          Resolution Fixed [ 1 ]
          Status Closed [ 6 ] Reopened [ 4 ]
          Petar Spanja (Inactive) made changes -
          Time Spent 1 hour, 45 minutes [ 6300 ] 3 hours, 15 minutes [ 11700 ]
          Worklog Id 55257 [ 55257 ]
          Petar Spanja (Inactive) made changes -
          Status Reopened [ 4 ] Closed [ 6 ]
          Resolution Fixed [ 1 ]
          Petar Spanja (Inactive) made changes -
          Link This issue relates to EZP-24720 [ EZP-24720 ]
          Alex Schuster made changes -
          Workflow EZ* Development Workflow [ 85385 ] EZEE Development Workflow [ 124207 ]
          Transition Time In Source Status Execution Times Last Executer Last Execution Date
          InputQ InputQ Backlog Backlog
          40d 4h 46m 1 André Rømcke 08/Nov/14 6:35 PM
          Backlog Backlog InputQ InputQ
          184d 19h 51m 1 Petar Spanja (Inactive) 12/May/15 3:26 PM
          InputQ InputQ Development Development
          4s 1 Petar Spanja (Inactive) 12/May/15 3:26 PM
          Development Development Documentation Review done Documentation Review done
          1d 17h 14m 1 Petar Spanja (Inactive) 14/May/15 8:41 AM
          Documentation Review done Documentation Review done QA QA
          25m 22s 1 rui.silva@ez.no 14/May/15 9:07 AM
          QA QA Closed Closed
          1d 5h 55m 1 rui.silva@ez.no 15/May/15 3:02 PM
          Closed Closed Reopened Reopened
          31d 1h 3m 2 Petar Spanja (Inactive) 15/Jun/15 4:06 PM
          Reopened Reopened Closed Closed
          1s 2 Petar Spanja (Inactive) 15/Jun/15 4:06 PM

            People

            • Assignee:
              Unassigned
              Reporter:
              Petar Spanja (Inactive)
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Time Tracking

                Estimated:
                Original Estimate - Not Specified
                Not Specified
                Remaining:
                Remaining Estimate - 0 minutes
                0m
                Logged:
                Time Spent - 3 hours, 15 minutes
                3h 15m