Uploaded image for project: 'eZ Publish / Platform'
  1. eZ Publish / Platform
  2. EZP-24294

Search Not working correctly after decoupling from Persistence

    Details

      Description

      Since Search functionality was decoupled from Persistence, that it has stopped working correctly. This comes as a regression of the functionality tested at EZP-23403, which worked before.
      Steps to reproduce (ids are for reference only):

      1. Install an independent Solr setup according to the steps on the attached file solrConf.v3.md;

      2. Access legacy admin and go to Content Structure;

      3. Create the following content structure inside "Home":

      TREE        | LOCATION ID |
      FolderA     | 83          |
      » FolderB   | 83/84       | 
      »» FolderB1 | 83/84/86    |
      FolderC     | 85          |
      

      3. Go now to Users;

      4. Create the following user group structure inside user groups root:

      TREE             | CONTENT ID | LOCATION ID |
      User Group A     | 79         | 87          |
      » User Group B   | 79/80      | 87/88       |
      »» User Group B1 | 79/80/81   | 87/88/89    |
      User Group C     | 82         | 90          |
      

      5. Index all created content on Solr;

      TESTING CONTENT:

      6. Using public API or whatever method, move content at location 84 (FolderB) into location 85 (FolderC);

      7. Use Search for finding contents based on Subtree criterion (using as criterion the subtree of location root 83, FolderA, now empty inside).
      Only content of location id 83 itself, FolderA, should be found now, but it won't. FolderB and FolderB1 will still be returned as search results as well.

      8. Use search for finding contents based on Subtree criterion (using as criterion the subtree of location root 85, FolderC, now with the other contents nested in it).
      Both contents of location ids 84 and 86 (FolderB and FolderB1) should be returned, but they won't.

      TESTING USER GROUPS:

      9. Using public API or whatever method, move user group at location 80 (User Group B) into location 82 (User Group C);

      10. Use Search for finding contents based on Subtree criterion (using as criterion the subtree of location root 87, User Group A, now empty inside).
      Only User Group of location id 87 itself, User Group A, should be found now, but it won't. User Group B and User Group B1 will still be returned as search results as well.

      11. Use search for finding contents based on Subtree criterion (using as criterion the subtree of location root 90, User Group C, now with the other User Groups nested in it).
      Both User Groups of location ids 88 and 89 (User Group B and User Group B1) should be returned, but they won't.

      Use the attached bundle for testing, if needed.
      Usage examples:

      Access:
      http://<your_virtual_host>/ezp23403move/location/84/85
      moves content at location 84 into location 85.

      Access:
      http://<your_virtual_host>/ezp23403find/location/83
      returns contents found on subtree of location id 83

        Issue Links

          Activity

          Hide
          Petar Spanja (Inactive) added a comment -

          @[~rui.silva@ez.no]

          I could not reproduce this issue.
          The PR with added test cases: https://github.com/ezsystems/ezpublish-kernel/pull/1407

          Show
          Petar Spanja (Inactive) added a comment - @ [~rui.silva@ez.no] I could not reproduce this issue. The PR with added test cases: https://github.com/ezsystems/ezpublish-kernel/pull/1407
          Hide
          Rui Silva (Inactive) added a comment -

          Petar Spanja, after testing this again with the procedure described here (same as when first QA'ed), the results are still the same:
          the location of the moved contents doesn't get retrieved in search with updated location (using subtree criterion).

          After chatting with you on skype, it appears your tests (from the PR from your previous comment) execute this ok, because of your usage of the:
          $this->refreshSearch($repository);
          function from within your php file that extends eZ\Publish\API\Repository\Tests\BaseTest.php, whenever you move locations;

          Thus, it seems that in order to update the location of moved contents for Solr to be aware of, that calls to this function are required for the location change to somehow take effect.
          Until the possibility of using calls to this from within a bundle (and perhaps some documentation about this), this cannot be tested correctly, as the reported issue will still occur with regular public API testing.

          Show
          Rui Silva (Inactive) added a comment - Petar Spanja , after testing this again with the procedure described here (same as when first QA'ed), the results are still the same: the location of the moved contents doesn't get retrieved in search with updated location (using subtree criterion). After chatting with you on skype, it appears your tests (from the PR from your previous comment) execute this ok, because of your usage of the: $this->refreshSearch($repository); function from within your php file that extends eZ\Publish\API\Repository\Tests\BaseTest.php, whenever you move locations; Thus, it seems that in order to update the location of moved contents for Solr to be aware of, that calls to this function are required for the location change to somehow take effect. Until the possibility of using calls to this from within a bundle (and perhaps some documentation about this), this cannot be tested correctly, as the reported issue will still occur with regular public API testing.
          Hide
          André Rømcke added a comment - - edited

          [~rui.silva@ez.no] Could you test with Solr configured with autoSoftCommit.maxTime = 0?

          Cause putting this in tests is ugly, so I'll rather just document this and we can change solr integration tests to do this as well.

          Something like:

            <autoSoftCommit>
                 <maxTime>0</maxTime>
               </autoSoftCommit>
          

          https://wiki.apache.org/solr/SolrConfigXml

          Show
          André Rømcke added a comment - - edited [~rui.silva@ez.no] Could you test with Solr configured with autoSoftCommit.maxTime = 0? Cause putting this in tests is ugly, so I'll rather just document this and we can change solr integration tests to do this as well. Something like: <autoSoftCommit> <maxTime>0</maxTime> </autoSoftCommit> https://wiki.apache.org/solr/SolrConfigXml
          Hide
          Petar Spanja (Inactive) added a comment -

          André Rømcke

          I will configure auto commit instead of exposing the method, not sure if it would be meaningful to have it in the API.

          Show
          Petar Spanja (Inactive) added a comment - André Rømcke I will configure auto commit instead of exposing the method, not sure if it would be meaningful to have it in the API.
          Hide
          Rui Silva (Inactive) added a comment -

          Petar Spanja, you're right, it doesn't make much sense to have to use that with the Public API, one would have to be aware there is the need for commit everytime some content changed / was created. This functionality should perhaps be transparent to the user of the Public API. André Rømcke I'm going to test it with the commit config in the meanwhile.

          Show
          Rui Silva (Inactive) added a comment - Petar Spanja , you're right, it doesn't make much sense to have to use that with the Public API, one would have to be aware there is the need for commit everytime some content changed / was created. This functionality should perhaps be transparent to the user of the Public API. André Rømcke I'm going to test it with the commit config in the meanwhile.
          Hide
          Rui Silva (Inactive) added a comment -

          André Rømcke, Petar Spanja, having set:

          <autoSoftCommit>
              <maxTime>0</maxTime>
          </autoSoftCommit>
          

          on the solrconfig.xml of my solr core, the behaviour on both 5.4 and master holds the same: it seems the retrieval of search results, after having moved a subtree, does not update locations as described on the issue..
          Are you aware of some other config or something else that needs to be done for this setting to take effect perhaps?

          Show
          Rui Silva (Inactive) added a comment - André Rømcke , Petar Spanja , having set: <autoSoftCommit> <maxTime>0</maxTime> </autoSoftCommit> on the solrconfig.xml of my solr core, the behaviour on both 5.4 and master holds the same: it seems the retrieval of search results, after having moved a subtree, does not update locations as described on the issue.. Are you aware of some other config or something else that needs to be done for this setting to take effect perhaps?
          Hide
          Petar Spanja (Inactive) added a comment -

          It seems this is not possible to achieve with configuration, the results will be available in nearly real time, which will still be too slow in some cases and will cause some tests to fail.
          The other approach would be to add commit parameters when submitting documents as was the case before.
          That should be used only for the test purposes, so I don't think it would really be an improvement over the current state.

          [~rui.silva@ez.no] can you try with getting the search handler from container and calling commit() from there as was suggested?

          Show
          Petar Spanja (Inactive) added a comment - It seems this is not possible to achieve with configuration, the results will be available in nearly real time, which will still be too slow in some cases and will cause some tests to fail. The other approach would be to add commit parameters when submitting documents as was the case before. That should be used only for the test purposes, so I don't think it would really be an improvement over the current state. [~rui.silva@ez.no] can you try with getting the search handler from container and calling commit() from there as was suggested?
          Hide
          Rui Silva (Inactive) added a comment - - edited

          I've included Petar's test made available at:

          https://github.com/ezsystems/ezpublish-kernel/pull/1407/files
          

          into a bundle, and created a command to run it.
          The bundle is attached onto this issue labelled as "PetarBundle".
          To run it, start from a regular ezpublish installation with ezdemo and no demo content, import the bundle onto your installation, set up Solr with current settings and launch it.
          Execute the following command from ezpublish installation root:

          php ezpublish/console ezpublish:petartest
          

          It will output:

          PASS 1: should find 1, found 1.
          PASS 2: Found 'Members' user group.
          PASS 3: should find 0, found 0.
          FAIL 4: should find only 1, found another number.
          FAIL 5: didn't find Administrators under Members.
          FAIL 6: should find only 1, found another number.
          FAIL 7: didn't find Administrators under Editors.
          PASS 8: should find 0, found 0.
          

          whereas it should output all tests passed.
          The tests output here are basically 'mirrors' to the output of Petar's PHPunit assertion tests.
          Those can be easily identified from the code itself.

          Show
          Rui Silva (Inactive) added a comment - - edited I've included Petar's test made available at: https://github.com/ezsystems/ezpublish-kernel/pull/1407/files into a bundle, and created a command to run it. The bundle is attached onto this issue labelled as "PetarBundle". To run it, start from a regular ezpublish installation with ezdemo and no demo content, import the bundle onto your installation, set up Solr with current settings and launch it. Execute the following command from ezpublish installation root: php ezpublish/console ezpublish:petartest It will output: PASS 1: should find 1, found 1. PASS 2: Found 'Members' user group. PASS 3: should find 0, found 0. FAIL 4: should find only 1, found another number. FAIL 5: didn't find Administrators under Members. FAIL 6: should find only 1, found another number. FAIL 7: didn't find Administrators under Editors. PASS 8: should find 0, found 0. whereas it should output all tests passed. The tests output here are basically 'mirrors' to the output of Petar's PHPunit assertion tests. Those can be easily identified from the code itself.
          Show
          Petar Spanja (Inactive) added a comment - Pull request: https://github.com/ezsystems/ezpublish-kernel/pull/1439
          Show
          Petar Spanja (Inactive) added a comment - Merged to ezpublish-kernel/master: https://github.com/ezsystems/ezpublish-kernel/commit/5535eea0747562b17d1b74a743299290a95f84e1
          Hide
          Rui Silva (Inactive) added a comment -

          Updated as attachment an updated version of the testing bundle (labelled PetarBundle.tar.gz.v2), since there were new commits.
          Applied latest fix.
          All the tests in the bundle seem to be passing now on a 5.4 (to verify a little further though)

          Show
          Rui Silva (Inactive) added a comment - Updated as attachment an updated version of the testing bundle (labelled PetarBundle.tar.gz.v2), since there were new commits. Applied latest fix. All the tests in the bundle seem to be passing now on a 5.4 (to verify a little further though)
          Hide
          Rui Silva (Inactive) added a comment -

          The first tests this had been tested with at first by QA (from the issue EZP-23403) are now passing too, after this latest change, on a 5.4!

          Show
          Rui Silva (Inactive) added a comment - The first tests this had been tested with at first by QA (from the issue EZP-23403 ) are now passing too, after this latest change, on a 5.4!
          Hide
          Rui Silva (Inactive) added a comment -

          Tested and approved by QA for 5.4 and master.

          Show
          Rui Silva (Inactive) added a comment - Tested and approved by QA for 5.4 and master.

            People

            • Assignee:
              Unassigned
              Reporter:
              Rui Silva (Inactive)
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Time Tracking

                Estimated:
                Original Estimate - Not Specified
                Not Specified
                Remaining:
                Remaining Estimate - 0 minutes
                0m
                Logged:
                Time Spent - 5 hours, 40 minutes
                5h 40m