Uploaded image for project: 'eZ Publish / Platform'
  1. eZ Publish / Platform
  2. EZP-24329

Implement Solr search engine prototype using multicore capability to index Content per language

    Details

    • Sprint:
      Pollux Platform S12, Pollux Platform S13

      Description

      At the moment Solr Search Engine indexes Content field data in multivalued fields. This means values from different translations are stored in the same document, in multivalued fields. Because of this no sorting support on field data (Field sort clause) is implemented. Also, having all languages indexed in the same document means relevancy calculation won't be correct.

      Using Solr's multicore/sharding capability to index Content field data per language would enable implementation of Field sort clause while keeping correct document/term statistics per language.

      Search would have to target all/specific cores/shards, depending on the set of languages that need to be searched. Grouping should be used to resolve matches of the same Content (but different language) from the different cores/shards.

      Scope:

      Implement multicore/sharded prototype of the Solr Search Engine, supporting:

      • targeting specific languages (aka field filters parameter)
      • Field sort clause

        Issue Links

          Activity

          Petar Spanja (Inactive) created issue -
          Petar Spanja (Inactive) made changes -
          Field Original Value New Value
          Link This issue relates to EZP-24300 [ EZP-24300 ]
          Petar Spanja (Inactive) made changes -
          Labels candidate-for-next-sprint search solr search solr y2015s12
          Petar Spanja (Inactive) made changes -
          Original Estimate 4 days [ 115200 ]
          Remaining Estimate 4 days [ 115200 ]
          André Rømcke made changes -
          Description At the moment Solr Search Engine indexes Content field data in multivalued fields. This means values from different translations are stored in the same document, in multivalued fields. Because of this no sorting support on field data (Field sort clause) is implemented. Also, having all languages indexed in the same document means relevancy calculation won't be correct.

          Using Solr's multicore/sharding capability to index Content field data per language would enable implementation of Field sort clause while keeping correct document/term statistics per language.

          Search would have to target all/specific cores/shards, depending on the set of languages that need to be searched. Grouping should be used to resolve matches of the same Content (but different language) from the different cores/shards.

          Scope:

          Implement multicore/sharded prototype of the Solr Search Engine, supporting:

          * targeting specific languages (aka field filters parameter)
          * Field sort clause
          At the moment Solr Search Engine indexes Content field data in multivalued fields. This means values from different translations are stored in the same document, in multivalued fields. Because of this no sorting support on field data (Field sort clause) is implemented. Also, having all languages indexed in the same document means relevancy calculation won't be correct.

          Using Solr's multicore/sharding capability to index Content field data per language would enable implementation of Field sort clause while keeping correct document/term statistics per language.

          Search would have to target all/specific cores/shards, depending on the set of languages that need to be searched. Grouping should be used to resolve matches of the same Content (but different language) from the different cores/shards.

          Scope:

          Implement multicore/sharded prototype of the Solr Search Engine, supporting:

          * targeting specific languages (aka field filters parameter)
          * Field sort clause

          _Keep in mind followup is to split out index for Location Search, so getting this to work for Location search on current indexing strategy is not a priority_
          André Rømcke made changes -
          Description At the moment Solr Search Engine indexes Content field data in multivalued fields. This means values from different translations are stored in the same document, in multivalued fields. Because of this no sorting support on field data (Field sort clause) is implemented. Also, having all languages indexed in the same document means relevancy calculation won't be correct.

          Using Solr's multicore/sharding capability to index Content field data per language would enable implementation of Field sort clause while keeping correct document/term statistics per language.

          Search would have to target all/specific cores/shards, depending on the set of languages that need to be searched. Grouping should be used to resolve matches of the same Content (but different language) from the different cores/shards.

          Scope:

          Implement multicore/sharded prototype of the Solr Search Engine, supporting:

          * targeting specific languages (aka field filters parameter)
          * Field sort clause

          _Keep in mind followup is to split out index for Location Search, so getting this to work for Location search on current indexing strategy is not a priority_
          At the moment Solr Search Engine indexes Content field data in multivalued fields. This means values from different translations are stored in the same document, in multivalued fields. Because of this no sorting support on field data (Field sort clause) is implemented. Also, having all languages indexed in the same document means relevancy calculation won't be correct.

          Using Solr's multicore/sharding capability to index Content field data per language would enable implementation of Field sort clause while keeping correct document/term statistics per language.

          Search would have to target all/specific cores/shards, depending on the set of languages that need to be searched. Grouping should be used to resolve matches of the same Content (but different language) from the different cores/shards.

          Scope:

          Implement multicore/sharded prototype of the Solr Search Engine, supporting:

          * targeting specific languages (aka field filters parameter)
          * Field sort clause

          Hide
          André Rømcke added a comment -

          Keep in mind followup is to split out index for Location Search, so getting this to work for Location search on current indexing strategy is not a priority

          Show
          André Rømcke added a comment - Keep in mind followup is to split out index for Location Search, so getting this to work for Location search on current indexing strategy is not a priority
          André Rømcke made changes -
          Status Open [ 1 ] Confirmed [ 10037 ]
          André Rømcke made changes -
          Status Confirmed [ 10037 ] Backlog [ 10000 ]
          André Rømcke made changes -
          Sprint Pollux Platform S12 [ 70 ]
          Sarah Haïm-Lubczanski (Inactive) made changes -
          Labels search solr y2015s12 search solr
          André Rømcke made changes -
          Assignee Petar Spanja [ petar.spanja@ez.no ]
          Petar Spanja (Inactive) made changes -
          Status Backlog [ 10000 ] Development [ 3 ]
          Petar Spanja (Inactive) logged work - 07/May/15 5:40 PM
          • Time Spent:
            6 hours
             

            implement

          Petar Spanja (Inactive) logged work - 08/May/15 2:00 AM
          • Time Spent:
            6 hours, 30 minutes
             

            implement

          Petar Spanja (Inactive) logged work - 11/May/15 2:00 AM
          • Time Spent:
            6 hours, 15 minutes
             

            implement

          Petar Spanja (Inactive) logged work - 12/May/15 2:00 AM
          • Time Spent:
            2 hours, 45 minutes
             

            implement

          Petar Spanja (Inactive) logged work - 13/May/15 2:00 AM
          • Time Spent:
            4 hours, 30 minutes
             

            implement

          Show
          Petar Spanja (Inactive) added a comment - Pull request: https://github.com/ezsystems/ezpublish-kernel/pull/1256
          Petar Spanja (Inactive) made changes -
          Link This issue testing discovered EZP-24365 [ EZP-24365 ]
          Petar Spanja (Inactive) made changes -
          Status Development [ 3 ] Development Review [ 10006 ]
          Petar Spanja (Inactive) made changes -
          Original Estimate 4 days [ 115200 ] 0 minutes [ 0 ]
          Petar Spanja (Inactive) logged work - 19/May/15 2:00 AM
          • Time Spent:
            2 hours
             

            implement

          Petar Spanja (Inactive) made changes -
          Link This issue testing discovered EZP-24375 [ EZP-24375 ]
          Petar Spanja (Inactive) made changes -
          Link This issue testing discovered EZP-24376 [ EZP-24376 ]
          Petar Spanja (Inactive) made changes -
          Link This issue testing discovered EZP-24377 [ EZP-24377 ]
          Petar Spanja (Inactive) made changes -
          Original Estimate 0 minutes [ 0 ] 4 days [ 115200 ]
          Remaining Estimate 4 days [ 115200 ] 0 minutes [ 0 ]
          André Rømcke made changes -
          Rank Ranked higher
          André Rømcke made changes -
          Sprint Pollux Platform S12 [ 70 ] Pollux Platform S12, Pollux Platform S13 [ 70, 72 ]
          André Rømcke made changes -
          Rank Ranked higher
          Petar Spanja (Inactive) logged work - 21/May/15 2:00 AM
          • Time Spent:
            1 hour, 30 minutes
             

            sync AR, PB, KN

          Petar Spanja (Inactive) logged work - 21/May/15 2:00 AM
          • Time Spent:
            20 minutes
             

            implement

          Show
          Petar Spanja (Inactive) added a comment - Merged in https://github.com/ezsystems/ezpublish-kernel/commit/8e50f0a0dab002dfce573dfd304087af6cdcf82c .
          Petar Spanja (Inactive) made changes -
          Status Development Review [ 10006 ] Development Review done [ 10028 ]
          Petar Spanja (Inactive) made changes -
          Status Development Review done [ 10028 ] Documentation Review done [ 10011 ]
          Rui Silva (Inactive) made changes -
          Status Documentation Review done [ 10011 ] QA [ 10008 ]
          Assignee Petar Spanja [ petar.spanja@ez.no ] Rui Silva [ rui.silva@ez.no ]
          Paulo Nunes (Inactive) made changes -
          Flagged Impediment [ 10000 ]
          Hide
          Rui Silva (Inactive) added a comment -

          Documentation is needed to install and setup Solr, to work with search and index, and also regarding multicore setup right after.
          I created EZP-24412 for this.

          Show
          Rui Silva (Inactive) added a comment - Documentation is needed to install and setup Solr, to work with search and index, and also regarding multicore setup right after. I created EZP-24412 for this.
          Rui Silva (Inactive) made changes -
          Status QA [ 10008 ] InputQ [ 10001 ]
          Assignee Rui Silva [ rui.silva@ez.no ]
          André Rømcke made changes -
          Workflow eZ Engineering Scrumban Workflow [ 67392 ] EZ* Development Workflow [ 69795 ]
          André Rømcke made changes -
          Rank Ranked higher
          Petar Spanja (Inactive) made changes -
          Time Spent 6 hours [ 21600 ]
          Worklog Id 55216 [ 55216 ]
          Petar Spanja (Inactive) made changes -
          Time Spent 6 hours [ 21600 ] 1 day, 4 hours, 30 minutes [ 45000 ]
          Worklog Id 55217 [ 55217 ]
          Petar Spanja (Inactive) made changes -
          Time Spent 1 day, 4 hours, 30 minutes [ 45000 ] 2 days, 2 hours, 45 minutes [ 67500 ]
          Worklog Id 55218 [ 55218 ]
          Petar Spanja (Inactive) made changes -
          Time Spent 2 days, 2 hours, 45 minutes [ 67500 ] 2 days, 5 hours, 30 minutes [ 77400 ]
          Worklog Id 55219 [ 55219 ]
          Petar Spanja (Inactive) made changes -
          Time Spent 2 days, 5 hours, 30 minutes [ 77400 ] 3 days, 2 hours [ 93600 ]
          Worklog Id 55220 [ 55220 ]
          Petar Spanja (Inactive) made changes -
          Time Spent 3 days, 2 hours [ 93600 ] 3 days, 4 hours [ 100800 ]
          Worklog Id 55221 [ 55221 ]
          Petar Spanja (Inactive) made changes -
          Time Spent 3 days, 4 hours [ 100800 ] 3 days, 5 hours, 30 minutes [ 106200 ]
          Worklog Id 55222 [ 55222 ]
          Petar Spanja (Inactive) made changes -
          Time Spent 3 days, 5 hours, 30 minutes [ 106200 ] 3 days, 5 hours, 50 minutes [ 107400 ]
          Worklog Id 55223 [ 55223 ]
          Hide
          André Rømcke added a comment -

          Closing as fixed, doc issue moved to backlog: EZP-24412

          Show
          André Rømcke added a comment - Closing as fixed, doc issue moved to backlog: EZP-24412
          André Rømcke made changes -
          Status InputQ [ 10001 ] Closed [ 6 ]
          Resolution Fixed [ 1 ]
          Alex Schuster made changes -
          Workflow EZ* Development Workflow [ 69795 ] EZEE Development Workflow [ 124780 ]
          Transition Time In Source Status Execution Times Last Executer Last Execution Date
          Open Open Confirmed Confirmed
          1d 6h 59m 1 André Rømcke 06/May/15 5:27 PM
          Confirmed Confirmed Backlog Backlog
          2s 1 André Rømcke 06/May/15 5:27 PM
          Backlog Backlog Development Development
          16h 39m 1 Petar Spanja (Inactive) 07/May/15 10:06 AM
          Development Development Development Review Development Review
          7d 12h 37m 1 Petar Spanja (Inactive) 14/May/15 10:43 PM
          Development Review Development Review Development Review done Development Review done
          6d 15h 26m 1 Petar Spanja (Inactive) 21/May/15 2:10 PM
          Development Review done Development Review done Documentation Review done Documentation Review done
          5s 1 Petar Spanja (Inactive) 21/May/15 2:10 PM
          Documentation Review done Documentation Review done QA QA
          51m 58s 1 rui.silva@ez.no 21/May/15 3:02 PM
          QA QA InputQ InputQ
          3d 20h 49m 1 rui.silva@ez.no 25/May/15 11:51 AM
          InputQ InputQ Closed Closed
          23d 2h 51m 1 André Rømcke 17/Jun/15 2:43 PM

            People

            • Assignee:
              Unassigned
              Reporter:
              Petar Spanja (Inactive)
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Time Tracking

                Estimated:
                Original Estimate - 4 days
                4d
                Remaining:
                Remaining Estimate - 0 minutes
                0m
                Logged:
                Time Spent - 3 days, 5 hours, 50 minutes Time Not Required
                3d 5h 50m

                  Agile